KR102651577B1 - Crowd worker performance evaluation device, system and method including feedback on evaluation - Google Patents

Abstract

The present invention is a performance evaluation device for crowd workers that includes evaluation feedback to perform a more objective performance evaluation by identifying and correcting the unfair evaluation of the evaluator evaluating the performance of the crowd worker and feeding back the unfair evaluation to the evaluator; It relates to systems and methods. A performance evaluation device collects performance evaluations for crowdsourcing performed by multiple crowdworkers by multiple evaluators. The performance evaluation device calculates evaluation tendency similarity for multiple crowd workers and specific crowd workers for each multiple evaluator. The performance evaluation device determines whether there is unfair evaluation of multiple evaluators based on similarity in evaluation tendencies. The performance evaluation device feeds back the possibility of unfair evaluation to the evaluator terminal of the evaluator whose evaluation was determined to be unfair. The performance evaluation device counts the number of violations for each evaluator according to the number of unfair evaluations by the evaluator. The performance evaluation device calculates a weight for each evaluator based on the number of violations for each evaluator. And the performance evaluation device corrects the performance evaluation by applying weight to the performance evaluation.

Description

Crowd worker performance evaluation device, system and method including feedback on evaluation}

The present invention relates to the performance evaluation of crowd workers, and more specifically, to identify and correct the unfair evaluation of the evaluator who evaluates the performance of the crowd worker, and to provide feedback on the unfair evaluation to the evaluator to provide a more objective performance evaluation. It relates to a performance evaluation device, system, and method of crowd workers including feedback on the evaluation performed.

Artificial intelligence (AI) aims to implement intelligent artifacts, and these artifacts are called intelligent agents. Intelligent agents use sensors to perceive the surrounding environment and use actuators to take appropriate actions.

The form of an intelligent agent may be software, a computing system including hardware, or a robot. Intelligent agents can use computer keyboards, microphones, and cameras as sensors, and motors as actuators. Intelligent agents in software form sometimes use display devices and speakers as actuators.

Recently, despite the development of intelligent agents, the development of technology to preemptively provide information by considering the user's intention is slow. The biggest cause of the absence of artificial intelligence based on preemptive information provision that takes the user's intention into account is the absence of a data set for the design and training of artificial intelligence.

Meanwhile, crowd sourcing is a compound word of crowd and outsourcing, which means involving the public in some process of corporate activities. When a company uses crowdsourcing to produce a new product or improve a service, the company can receive feedback from the public about the product or service.

Performance evaluation of work performed by crowd workers performing crowdsourcing is also being conducted. Compared to the work performance of general office workers, crowdsourcing by crowdworkers has the characteristics of a non-face-to-face, short-cycle, and one-off type of work.

Because of this, there are limitations in applying the traditional face-to-face performance evaluation centered on the workplace to evaluate the performance of crowd workers.

Registered Patent Publication No. 10-2214006 (registered on February 3, 2021)

Therefore, the purpose of the present invention is to identify and correct the unfair evaluation of the evaluator evaluating the performance of the crowd worker, and provide feedback on the evaluation to perform a more objective performance evaluation by feeding back the unfair evaluation to the evaluator. To provide evaluation devices, systems, and methods.

In order to achieve the above object, the present invention includes the steps of: a performance evaluation device collecting performance evaluations for crowdsourcing performed by a plurality of crowd workers by a plurality of evaluators; Calculating, by the performance evaluation device, similarity in evaluation tendencies for the plurality of crowd workers and a specific crowd worker for each of the plurality of evaluators; determining, by the performance evaluation device, whether the evaluation of the plurality of evaluators is unfair based on the evaluation tendency similarity; Feeding back, by the performance evaluation device, the possibility of unfair evaluation to the evaluator terminal of the evaluator whose evaluation was determined to be unfair; Counting, by the performance evaluation device, the number of violations for each evaluator according to the number of unfair evaluations by the evaluator; calculating, by the performance evaluation device, a weight for each evaluator according to the number of violations for each evaluator; and a step of the performance evaluation device correcting the performance evaluation by applying the weight to the performance evaluation.

The evaluation tendency similarity is calculated using the equation below.

[Equation]

Specific evaluation score (i): The evaluation score of the crowd worker (i) given by the user

Average self-evaluation: Average of evaluation scores of multiple crowd workers given by a specific evaluator

In the step of determining whether the evaluation is unfair, the performance evaluation device may determine that the evaluation is unfair if the calculated evaluation tendency similarity is greater than or equal to a threshold value.

In the step of calculating the weight, the performance evaluation device may calculate a lower weight as the number of violations increases.

The performance evaluation device can calculate the weight as a value between 0 and 1.

The performance evaluation method of a crowd worker according to the present invention includes the steps of correcting the existing performance evaluation by the performance evaluation device applying the weight to the existing performance evaluation by the plurality of evaluators, which is performed after the correction step; It may further include.

The present invention also includes a communication unit that communicates with a plurality of evaluator terminals; And collecting performance evaluations of crowdsourcing performed by a plurality of crowd workers by a plurality of evaluators from a plurality of evaluator terminals through the communication unit, and evaluating the evaluation tendencies of the plurality of crowd workers and specific crowd workers for each of the plurality of evaluators. Calculate the degree of similarity, determine whether the evaluation is unfair for the plurality of evaluators based on the similarity in evaluation tendencies, feed back the possibility of unfair evaluation to the evaluator terminal of the evaluator determined to be an unfair evaluation as a result of the judgment, and evaluate the evaluator according to the number of unfair evaluations by the evaluator. A control unit that counts the number of violations for each evaluator, calculates a weight for each evaluator according to the number of violations for each evaluator, and corrects the performance evaluation by applying the weight to the performance evaluation. Provides a performance evaluation device of a crowd worker including a. .

The crowd worker performance evaluation apparatus according to the present invention further includes a storage unit for storing performance evaluations by the plurality of evaluators.

The control unit may correct the stored performance evaluation by applying the weight to the performance evaluation stored in the storage unit.

Additionally, the present invention includes a plurality of evaluator terminals that perform performance evaluation on crowdsourcing performed by a plurality of crowd workers; And collect performance evaluations for crowdsourcing performed by a plurality of crowdworkers by a plurality of evaluators from the plurality of evaluator terminals, and calculate evaluation tendency similarity for the plurality of crowdworkers and a specific crowdworker for each of the plurality of evaluators. The evaluation tendency similarity is used to determine whether the evaluation is unfair for the plurality of evaluators, and the possibility of unfair evaluation is fed back to the evaluator terminal of the evaluator determined to be an unfair evaluation as a result of the judgment, and the number of violations for each evaluator is calculated according to the number of unfair evaluations by the evaluator. A performance evaluation device that counts, calculates a weight for each evaluator according to the number of violations for each evaluator, and corrects the performance evaluation by applying the weight to the performance evaluation. It provides a performance evaluation system of a crowd worker including a.

According to the present invention, it is possible to perform a more objective performance evaluation by identifying and correcting unfair evaluations by evaluators evaluating the performance of crowd workers. This can improve the accuracy and reliability of performance evaluation of crowd workers.

By correcting previous incorrect performance evaluation results performed by unfair evaluators through the performance evaluation of crowd workers according to the present invention, it is possible to prevent crowd workers from being influenced by performance evaluation errors on their ability to participate in other crowdsourcing projects. there is. In other words, it is expected that crowd workers will be able to secure opportunities to participate in crowdsourcing projects such as data labeling through more objective performance evaluation.

The performance evaluation of crowd workers according to the present invention has the advantage of securing verified crowd workers by using objective performance evaluation from the crowd sourcer's perspective.

Since the present invention determines unfair evaluation by calculating evaluation tendency similarity using only the performance evaluation scores for each evaluator, it is possible to induce a fair performance evaluation by providing feedback on the possibility of unfair evaluation even during the evaluation.

In addition, the crowd worker's performance evaluation system according to the present invention provides fair evaluation by feedbacking the possibility of unfair evaluation of the evaluator in case an unfair evaluation is performed on a specific evaluator (crowd worker) without the evaluator being aware during the evaluation. can lead to .

1 is a block diagram showing a performance evaluation system of a crowd worker including feedback on evaluation according to an embodiment of the present invention.
Figure 2 is a block diagram showing the performance evaluation device of Figure 1.
Figure 3 is a flowchart of a crowd worker performance evaluation method including feedback on evaluation according to an embodiment of the present invention.
Figures 4 to 9 are exemplary diagrams according to the performance evaluation method of the crowd worker in Figure 3.

It should be noted that in the following description, only the parts necessary to understand the embodiments of the present invention will be described, and descriptions of other parts will be omitted without departing from the gist of the present invention.

The terms or words used in the specification and claims described below should not be construed as limited to their usual or dictionary meanings, and the inventor should use the concept of terminology appropriately to explain his/her invention in the best way. It must be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined clearly. Therefore, the embodiments described in this specification and the configurations shown in the drawings are only preferred embodiments of the present invention, and do not represent the entire technical idea of the present invention, and therefore, various equivalents can be substituted for them at the time of filing the present application. It should be understood that there may be variations.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the attached drawings.

1 is a block diagram showing a performance evaluation system of a crowd worker including feedback on evaluation according to an embodiment of the present invention.

Referring to FIG. 1, the performance evaluation system 100 of a crowd worker according to this embodiment is a system that evaluates the performance of crowd sourcing performed by a crowd worker, and includes a plurality of evaluator terminals 10 and a performance evaluation device 20. ) includes. Here, crowdsourcing can include data labeling.

The evaluator terminal 10 is a communication terminal used by an evaluator who evaluates the performance of crowdsourcing performed by a crowd worker. The evaluator evaluates the performance of the crowd worker through the evaluator terminal 10 and transmits the evaluation results to the performance evaluation device 20. The evaluator terminal 10 can score the evaluator's performance evaluation results and transmit them to the performance evaluation device 20. This evaluator terminal 10 is a terminal capable of communicating with the performance evaluation device 20 and may include, for example, a smartphone, laptop, handheld PC, tablet PC, wearable device, etc., but is not limited thereto.

The evaluator terminal 10 may receive feedback about the possibility of unfair evaluation from the performance evaluation device 20 that is evaluating the performance of the crowd worker. In other words, during an evaluator's evaluation of the performance of a crowd worker, an unfair evaluation may be performed on a specific evaluator (crowd worker) without the evaluator being aware of it. The performance evaluation device 20 can guide the performance of a fair evaluation by feeding back the possibility of such unfair evaluation to the relevant evaluator. For example, an evaluator who receives feedback on the possibility of unfair evaluation of a specific evaluator can modify the performance evaluation for that specific evaluator. The evaluator can transmit the revised evaluation results to the performance evaluation device 20 through the evaluator terminal 10. The performance evaluation device 20 may re-perform the evaluation based on the revised performance evaluation.

The communication method between the evaluator terminal 10 and the performance evaluation device 20 includes a wired or wireless network. Wired networks may include LANs (Local Area Networks), WANs (Wide Area Networks), MANs (Metropolitan Area Networks), and ISDNs (Integrated Service Digital Networks), while wireless networks may include wireless LANs, CDMA, Bluetooth, and satellite communications. It may include, but is not limited to this.

Additionally, the communication method may include short-distance communication or long-distance communication. Here, short-range communication may include Bluetooth, RFID (radio frequency identification), infrared communication (IrDA, infrared data association), UWB (ultra-wideband), ZigBee, and Wi-Fi (wireless fidelity) technology. It is not limited to. Long-distance communications include code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), and single carrier frequency division multiple access (SC-FDMA) technologies. It can be done, and it is not limited to this.

Additionally, the performance evaluation device 20 performs a performance evaluation based on performance evaluations of a plurality of crowd workers collected from a plurality of evaluators, and performs a more objective performance evaluation by identifying and correcting the evaluator's unfair evaluation. In addition, if an unfair evaluation is performed on a specific evaluation target (crowd worker) without the evaluator being aware of it during the evaluation, the performance evaluation device 20 will induce a fair evaluation by feeding back the possibility of unfair evaluation of the evaluation target. You can.

This performance evaluation device 20 collects performance evaluations for crowdsourcing performed by a plurality of crowd workers by a plurality of evaluators from a plurality of evaluator terminals 10. The performance evaluation device 20 calculates evaluation tendency similarity for a plurality of crowd workers and a specific crowd worker for each evaluator. The performance evaluation device 20 determines whether or not there is unfair evaluation of multiple evaluators based on similarity in evaluation tendencies. The performance evaluation device 20 feeds back the possibility of unfair evaluation to the evaluator terminal 10 of the evaluator whose evaluation was determined to be unfair. The performance evaluation device 20 counts the number of violations for each evaluator according to the number of unfair evaluations by the evaluator. The performance evaluation device 20 calculates a weight for each evaluator according to the number of violations for each evaluator. And the performance evaluation device 20 corrects the performance evaluation by applying weight to the performance evaluation.

In addition, when the performance evaluation device 20 according to the present embodiment stores existing performance evaluations by a plurality of evaluators who calculated weights, the existing performance evaluations can be corrected by applying the calculated weights to the existing performance evaluations. there is.

In this way, the performance evaluation device 20 according to this embodiment uses evaluation tendency similarity as a fairness score for the evaluation when assigning evaluation scores to each evaluator in order to classify, identify, and correct unfair evaluations in performance evaluations applied to crowd workers. It is used as the main item of the evaluation factor. As a result, the performance evaluation device 20 according to this embodiment can perform a more objective performance evaluation by identifying and correcting unfair evaluations by evaluators evaluating the performance of crowd workers.

In addition, when an unfair evaluation is performed on a specific evaluator (crowd worker) without the evaluator being aware of it during the evaluation, the performance evaluation device 20 according to this embodiment prevents the possibility of unfair evaluation of the evaluator from being evaluated by the evaluator. Feedback is provided to the evaluator terminal (20). Because of this, the performance evaluation device 20 can guide the evaluator who has confirmed the possibility of unfair evaluation to perform a fair evaluation. In particular, the performance evaluation device 20 can induce a fair evaluation by the evaluator of a specific crowd worker for which the possibility of unfair evaluation has been pointed out.

The performance evaluation device 20 according to this embodiment will be described with reference to FIGS. 1 and 2 as follows. Here, FIG. 2 is a block diagram showing the performance evaluation device 20 of FIG. 1.

The performance evaluation device 20 according to this embodiment includes a communication unit 21, a storage unit 23, and a control unit 25.

The communication unit 21 communicates with a plurality of evaluator terminals 10. The communication unit 21 collects performance evaluations of a plurality of crowd workers performed by a plurality of evaluators through the evaluator terminal 10. Under the control of the control unit 25, the communication unit 21 feeds back the possibility of unfair evaluation to the evaluator terminal 10 of the evaluator judged to be an unfair evaluation.

The storage unit 23 stores a program necessary for controlling the operation of the performance evaluation device 20 and data generated during execution of the program. The storage unit 23 stores an execution program for performing performance evaluation. The storage unit 23 stores the original performance evaluation collected from the evaluator terminal 10 and stores the corrected performance evaluation reflecting the weight calculated based on the original performance evaluation. The storage unit 23 may store the corrected performance evaluation together with the previously stored performance evaluation, or may store the corrected performance evaluation by reflecting the calculated weight of the previously stored performance evaluation.

And the control unit 25 includes a microprocessor that performs overall control of the performance evaluation device 20. The control unit 25 can evaluate the performance of crowdsourcing performed by the crowd worker as follows.

The control unit 25 collects performance evaluations for crowdsourcing performed by a plurality of crowd workers by a plurality of evaluators from the plurality of evaluator terminals 10 through the communication unit 21. The control unit 25 calculates evaluation tendency similarity for a plurality of crowd workers and a specific crowd worker for each plurality of evaluators. The control unit 25 determines whether the evaluation of multiple evaluators is unfair based on the similarity in evaluation tendencies. The control unit 25 feeds back the possibility of unfair evaluation to the evaluator terminal 10 of the evaluator whose evaluation is determined to be unfair. The control unit 25 counts the number of violations for each evaluator according to the number of unfair evaluations by the evaluator. The control unit 25 calculates a weight for each evaluator according to the number of violations for each evaluator. Then, the control unit 25 corrects the performance evaluation by applying weight to the performance evaluation.

Here, the control unit 25 calculates the evaluation tendency similarity using Equation 1 below.

[Equation 1]

Specific evaluation score (i): The evaluation score of the crowd worker (i) given by the user

Average self-evaluation: Average of evaluation scores of multiple crowd workers given by a specific evaluator

In this way, the control unit 25 uses the evaluation tendency similarity to classify the possibility of unfair evaluation that may occur. In other words, the most desirable case is when the evaluation tendency similarity score converges to 0. Therefore, the closer the evaluation tendency similarity score is to 0, the more fair the performance evaluation by multiple evaluators can be judged to be. Conversely, the larger the evaluation tendency similarity score, the more unfairly the performance evaluation by the evaluator can be judged.

The control unit 25 may determine an unfair evaluation if the calculated evaluation tendency similarity is greater than or equal to a threshold value. The threshold is a standard value for determining whether the evaluator violates the fairness of the evaluation, and can be determined by considering the type and characteristics of the crowdsourcing performed by the crowd worker, the intention of the manager performing the performance evaluation, etc.

The control unit 25 can calculate the weight as follows. That is, the control unit 25 can calculate a lower weight as the number of violations increases. The control unit 25 can set the weight to decrease linearly, non-linearly, or stepwise according to the increase in the number of violations. The control unit 25 can calculate the weight as a value between 0 and 1.

The control unit 25 corrects the performance evaluation by applying the calculated weight to the performance evaluation. The control unit 25 may calculate the final evaluation result for a plurality of crowd workers by adding up and averaging the corrected performance evaluation scores.

In addition, when the control unit 25 stores an existing performance evaluation by the evaluator who calculated the calculated weight, the control unit 25 can correct the existing performance evaluation by applying the calculated weight to the existing performance evaluation. In this way, by feeding back the weight calculated to the performance evaluation of existing evaluation subjects by an unfair evaluator, the performance evaluation given to the existing incorrect crowd worker can be corrected. As a result, by correcting previous incorrect performance evaluation results performed by unfair evaluators, it is possible to reduce the influence of performance evaluation errors on the possibility of crowdworkers participating in other crowdsourcing projects. In other words, crowd workers can avoid being disadvantaged due to incorrect performance evaluation in securing opportunities to participate in crowdsourcing projects such as data labeling through more objective performance evaluation.

The performance evaluation method of the crowd worker by the performance evaluation system 100 according to this embodiment will be described with reference to FIGS. 1 to 3 as follows. Here, FIG. 3 is a flowchart of a crowd worker performance evaluation method including feedback on evaluation according to an embodiment of the present invention.

First, in step S10, the performance evaluation device 20 collects performance evaluations for crowdsourcing performed by a plurality of crowd workers by a plurality of evaluators. That is, a plurality of evaluators perform performance evaluation on crowdsourcing performed by a plurality of crowd workers through the evaluator terminal 10. The performance evaluation device 20 collects performance evaluation results performed from a plurality of evaluator terminals 10.

Next, in step S20, the performance evaluation device 20 calculates the evaluation tendency similarity for a plurality of crowd workers and a specific crowd worker for each evaluator. Here, the evaluation tendency similarity is calculated using Equation 1.

Next, in step S30, the performance evaluation device 20 determines whether the evaluation of multiple evaluators is unfair based on the similarity in evaluation tendencies.

If there is an evaluator whose evaluation is determined to be unfair as a result of the judgment in step S30, the performance evaluation device 20 feeds back the possibility of unfair evaluation to the evaluator terminal 10 of the evaluator in step S40.

Meanwhile, an evaluator who receives feedback on the possibility of unfair evaluation of a specific evaluator can modify the performance evaluation for that specific evaluator. The performance evaluation device 20 may collect the revised performance evaluation in step S10 and perform subsequent steps again.

Of course, the evaluator can maintain the performance evaluation despite receiving feedback about the possibility of an unfair evaluation.

Next, in step S50, the performance evaluation device 20 counts the number of violations for each evaluator according to the number of unfair evaluations by the evaluator.

Subsequently, in step S60, the performance evaluation device 20 calculates a weight for each evaluator according to the number of violations for each evaluator.

And in step S70, the performance evaluation device 20 corrects the performance evaluation by applying weight to the performance evaluation. That is, the performance evaluation device 20 can calculate the final evaluation result for a plurality of crowd workers by adding up and averaging the corrected performance evaluation scores.

After step S70, the performance evaluation device 20 may correct the stored performance evaluation by applying weight to the stored performance evaluation by a plurality of evaluators in step S80.

As such, in this embodiment, in performance evaluation, by correcting or correcting the performance evaluation using weights, it is possible to identify and correct the evaluator's unfair evaluation and perform a more objective performance evaluation. In other words, by comparing and identifying situations where abnormal scores (unfairness) are imposed on a specific evaluator among multiple evaluators, the weight of the scores of unfair evaluators judged to have a high possibility of unfair evaluation is adjusted when adding up the performance evaluation scores for the evaluators. The impact of unfair evaluation can be reduced or completely eliminated. For example, if the weight is 0, the influence of unfair evaluation can be completely excluded.

And in this embodiment, in performance evaluation, the possibility of unfair evaluation can be fed back to the evaluator who is judged to have performed an unfair evaluation, thereby encouraging the evaluator to perform a fair evaluation.

The performance evaluation method of crowd workers according to this embodiment will be described with reference to specific examples shown in FIGS. 4 to 9 as follows. Here, Figures 4 to 9 are exemplary diagrams according to the performance evaluation method of the crowd worker in Figure 3.

I would like to explain this as an example where 4 evaluators (1 to 4) performed a performance evaluation on 3 crowd workers (A, B, C) who were the evaluation subjects.

The evaluator assigned a performance evaluation score out of 100. The performance evaluations collected by the performance evaluation device from evaluators 1 to 4 are shown in Figure 4. At this time, the evaluation results are not shared between evaluators 1 to 4. For example, evaluator 1, who is currently conducting an evaluation, cannot know the performance evaluation scores of evaluators 2, 3, and 4.

Next, the evaluation tendency similarity calculated using Equation 1 based on the performance evaluation score in Figure 4 is shown in Figure 5.

Evaluator 1's evaluation tendency similarity to Crowd Worker A is calculated as shown in Equation 1-1 below.

[Equation 1-1]

Evaluator 1 assigned a performance evaluation score of 60 to Crowd Worker A. It can be seen that Evaluator 1's performance evaluation score for Crowd Worker A has a large difference from the performance evaluation scores of other evaluators, and also a large difference from the performance evaluation scores he gave to other evaluators (Crowd Workers B and C). .

Next, as shown in FIG. 6, the performance evaluation device determines whether the evaluators have made unfair evaluations based on the set threshold, and marks the unfair evaluation as a violation. The performance evaluation device determines that the evaluation is unfair if the calculated evaluation tendency similarity is above the threshold. To facilitate understanding, in this example, the threshold was set to 100 to determine whether there was unfair evaluation. Violation status was indicated as '1' if there was a violation, and '0' if there was no violation. Referring to the violation, it can be determined that Evaluator 1 conducted an unfair evaluation of Crowd Worker A.

The performance evaluation device determines that Evaluator 1 has unfairly evaluated Crowd Worker A in this performance evaluation (evaluation round 1), and can feed back the possibility of unfair evaluation to Evaluator 1's evaluator terminal.

As shown in FIG. 7, the performance evaluation device stores this performance evaluation as history data between evaluator 1 and crowd worker A. The historical data includes information on performance evaluation scores and violations performed in the past for Crowd Worker A of Evaluator 1.

Evaluator 1 has a history of conducting performance evaluations for Crowd Worker A a total of 4 times. Therefore, it can be judged that the performance evaluation device is highly likely to have conducted an unfair evaluation of Crowd Worker A in this evaluation, as there were three unfair evaluations out of a total of four. The performance evaluation device can improve evaluation fairness by feeding back to Evaluator 1 the possibility of unfair evaluation of Crowd Worker A, allowing the evaluator himself to be aware of the possibility of emotionally unfair evaluation of Crowd Worker A without being aware of it.

At this time, the performance evaluation device can provide feedback to the evaluator about the possibility of unfair evaluation as follows. The performance evaluation device can feed back the possibility of unfair evaluation to the relevant evaluator whenever a violation due to unfair evaluation is confirmed. Alternatively, if the performance evaluation device confirms a certain number of violations due to unfair evaluation of a specific evaluator, it can feed back the possibility of unfair evaluation to the relevant evaluator.

Next, the weight for each number of violations can be set as shown in FIG. 8. By setting a lower weight in performance evaluation as the number of violations increases, the influence of the performance evaluation score of the evaluator who performed an unfair evaluation can be reduced. For example, in this evaluation of Crowd Worker A by Evaluator 1, the number of violations is 3 in total, including this evaluation, so the performance evaluation device calculates the weight as 0.1.

The performance evaluation results of evaluator 1 calculating the weights for crowd workers A, B, and C are shown in Figure 9.

Referring to Figure 9, Evaluator 1's evaluation tendency similarity to Crowd Worker A is 156.4, and since it belongs to the number of violations 3 times, a weight of 0.1 is applied. Evaluator 1's performance evaluation of crowd workers B and 3 is judged to be a normal performance evaluation and a weight of 1 is applied. Evaluator 1's corrected performance evaluation score for crowd worker A is 6 points (60×0.1=6).

The final performance evaluation score of Crowd Worker A is as follows when comparing the case calculated as before and the case where the weight calculated as in this embodiment is applied. Here, the final performance evaluation score of Crowd Worker A is calculated based on the evaluation score in Figure 5, and the weight of evaluators 2-4 is set to 1.

In the existing case, the performance evaluation scores given to crowd worker A are added up and averaged to calculate the final evaluation score. In other words, the final evaluation score given to Crowd Worker A according to the existing method is (60+95+93+98)/4=86.5, which is the average value of peer evaluation.

In this embodiment, the corrected performance evaluation scores given to crowd worker A are added up and averaged to calculate the final evaluation score. That is, the final evaluation score given to crowd worker A according to this embodiment is (60×0.1+95+93+98)/3.1=94.19.

The final evaluation score calculated based on the performance evaluation scores of evaluators 2 to 4 who performed a normal performance evaluation is (95+93+98)/3=95.33.

Therefore, there is not much difference between the final evaluation score of 94.19 calculated based on the performance evaluation scores of evaluators 1 to 4 according to this embodiment and the final evaluation score of 95.33 calculated based on the performance evaluation scores of evaluators 2 to 4. Able to know.

In this way, by comparing and identifying situations where abnormal scores (unfairness) are imposed on specific evaluators among evaluators, the performance evaluation scores of unfair evaluators judged to have a high possibility of unfair evaluation are given less weight when adding up the performance evaluation scores for the evaluation subjects. The influence can be reduced or eliminated. In other words, if the weight is 0, the influence on unfair evaluation can be completely excluded.

Meanwhile, in the method of evaluating the performance of crowd workers according to this embodiment, the determination of whether the evaluation is unfair may be performed by evaluator 1 after confirming the performance evaluation of the three crowd workers, or before confirming the performance evaluation. .

If a step to determine whether an unfair evaluation is performed before confirming the performance evaluation, for example, the performance evaluation device may feed back the possibility of an unfair evaluation to evaluator 1. Evaluator 1 can re-perform the performance evaluation of Crowd Worker A based on the possibility of unfair evaluation received from the feedback and then confirm the performance evaluation.

Meanwhile, the embodiments disclosed in the specification and drawings are merely provided as specific examples to aid understanding, and are not intended to limit the scope of the present invention. It is obvious to those skilled in the art that in addition to the embodiments disclosed herein, other modifications based on the technical idea of the present invention can be implemented.

10: Evaluator terminal
20: Performance evaluation device
21: Department of Communications
23: storage unit
25: control unit
100: Performance evaluation system

Claims (9)

A performance evaluation device collecting performance evaluations for crowdsourcing performed by a plurality of crowd workers by a plurality of evaluators;
Calculating, by the performance evaluation device, similarity in evaluation tendencies for the plurality of crowd workers and a specific crowd worker for each of the plurality of evaluators;
determining, by the performance evaluation device, whether the evaluation of the plurality of evaluators is unfair based on the evaluation tendency similarity;
Feeding back, by the performance evaluation device, the possibility of unfair evaluation to the evaluator terminal of the evaluator whose evaluation was determined to be unfair;
Counting, by the performance evaluation device, the number of violations for each evaluator according to the number of unfair evaluations by the evaluator;
calculating, by the performance evaluation device, a weight for each evaluator according to the number of violations for each evaluator;
correcting the performance evaluation by the performance evaluation device applying a weight for each evaluator to the performance evaluation; and
Comprising: the performance evaluation device correcting the existing performance evaluation by applying a weight for each evaluator to the existing performance evaluation by the plurality of evaluators,
The evaluation tendency similarity is calculated using the equation below,
In the step of determining whether the evaluation is unfair and the step of providing feedback,
The performance evaluation device feeds back the possibility of unfair evaluation to the evaluator terminal whenever a violation due to unfair evaluation is confirmed, or if violations due to unfair evaluation for a specific evaluator are confirmed more than a certain number of times, it reports the possibility of unfair evaluation to the evaluator terminal. give feedback,
In the step of calculating the weight for each evaluator,
The performance evaluation device calculates a lower weight for each evaluator as the number of violations increases, but calculates it to a value between 0 and 1,
In the step of determining whether the evaluation is unfair,
The performance evaluation device determines that it is an unfair evaluation if the calculated evaluation tendency similarity is more than a threshold, and the threshold is a standard value for determining whether the evaluator has violated the fairness of the evaluation, and the type, characteristics, and performance of crowdsourcing performed by the crowd worker. A performance evaluation method for crowd workers, characterized in that the decision is made taking into account the intention of the manager performing the evaluation.
[Equation]

Specific evaluation score (i): The evaluation score of the crowd worker (i) given by the user
Average self-evaluation: Average of evaluation scores of multiple crowd workers given by a specific evaluator delete delete delete delete delete A communication unit that communicates with a plurality of evaluator terminals;
a storage unit that stores performance evaluations by the plurality of evaluators; and
Through the communication unit, performance evaluations for crowdsourcing performed by a plurality of crowdworkers by a plurality of evaluators are collected from a plurality of evaluator terminals, and evaluation tendency similarities between the plurality of crowdworkers and a specific crowdworker are calculated for each of the plurality of evaluators. Calculate, determine whether the evaluation is unfair for the plurality of evaluators based on the evaluation tendency similarity, feed back the possibility of unfair evaluation to the evaluator terminal of the evaluator determined to be an unfair evaluation as a result of the judgment, and each evaluator according to the number of unfair evaluations by the evaluator. Count the number of violations, calculate a weight for each evaluator according to the number of violations for each evaluator, correct the performance evaluation by applying the weight for each evaluator to the performance evaluation, and apply the weight for each evaluator to the performance evaluation stored in the storage unit. A control unit that corrects the stored performance evaluation by applying a;
The control unit calculates the evaluation tendency similarity using the following equation,
When providing feedback on the possibility of unfair evaluation, the control unit feeds back the possibility of unfair evaluation to the evaluator terminal whenever a violation due to unfair evaluation is confirmed, or when violations due to unfair evaluation for a specific evaluator are confirmed more than a certain number of times, Feedback on the possibility of unfair evaluation to the evaluator terminal,
When calculating the weight for each evaluator, the control unit calculates the weight for each evaluator lower as the number of violations increases, but calculates the weight for each evaluator to a value between 0 and 1,
When determining whether the evaluation is unfair, the control unit determines that it is an unfair evaluation if the calculated evaluation tendency similarity is greater than or equal to a threshold, and the threshold is a standard value for determining whether the evaluator violates the fairness of the evaluation, which is performed by the crowd worker. A crowdworker performance evaluation device characterized by determining the type and characteristics of crowdsourcing and the intention of the manager performing the performance evaluation.
[Equation]

Specific evaluation score (i): The evaluation score of the crowd worker (i) given by the user
Average self-evaluation: Average of evaluation scores of multiple crowd workers given by a specific evaluator delete A plurality of evaluator terminals that perform performance evaluation on crowdsourcing performed by a plurality of crowd workers; and
Stores performance evaluations by the plurality of evaluators, collects performance evaluations for crowdsourcing performed by a plurality of crowd workers by a plurality of evaluators from the plurality of evaluator terminals, and collects performance evaluations of the crowdsourcing performed by a plurality of crowdworkers by the plurality of evaluators from the plurality of evaluator terminals. Calculate the similarity of evaluation tendencies for workers and specific crowd workers, determine whether the evaluation is unfair for the plurality of evaluators based on the similarity of evaluation tendencies, and feed back the possibility of unfair evaluation to the evaluator terminal of the evaluator judged to be unfair as a result of the judgment. , counts the number of violations for each evaluator according to the number of unfair evaluations by the evaluator, calculates a weight for each evaluator according to the number of violations for each evaluator, corrects the performance evaluation by applying the weight for each evaluator to the performance evaluation, and stores the stored performance. It includes a performance evaluation device that corrects the stored performance evaluation by applying a weight for each evaluator to the evaluation,
The performance evaluation device calculates the evaluation tendency similarity using the following equation,
When feedbacking the possibility of unfair evaluation, the performance evaluation device feeds back the possibility of unfair evaluation to the evaluator terminal every time a violation due to unfair evaluation is confirmed, or confirms that violations due to unfair evaluation for a specific evaluator have occurred more than a certain number of times. When this happens, the possibility of unfair evaluation is fed back to the evaluator terminal.
When calculating the weight for each evaluator, the performance evaluation device calculates the weight for each evaluator lower as the number of violations increases, but calculates the weight for each evaluator to a value between 0 and 1,
When determining whether the evaluation is unfair, the performance evaluation device determines that it is an unfair evaluation if the calculated evaluation tendency similarity is more than a threshold, and the threshold is a standard value for determining whether the evaluator violates the fairness of the evaluation, and the crowd worker A crowdworker performance evaluation system characterized by determining the type of crowdsourcing being performed, its characteristics, and the decision of the manager performing the performance evaluation.
[Equation]

Specific evaluation score (i): The evaluation score of the crowd worker (i) given by the user
Average self-evaluation: Average of evaluation scores of multiple crowd workers given by a specific evaluator

Images