KR100976054B1 - A feedback method to secure the consistency in appraisals which consist of the combination quantitative indices and qualitative indices - Google Patents

Abstract

PURPOSE: A feedback method for securing the consistency of the evaluation and the evaluation of indexes is provided to evaluate the deviations of each evaluator by feedback of the fact and evaluating the evaluation factor. CONSTITUTION: An information processing device inputs an index of evaluation values. An estimation rank is calculated with the inputted index of evaluation values(S21). Average and standard deviation about each evaluation is calculated(S25). A tendency average according to each class and inclination standard deviation is set. The consistency average according to each class and consistency standard deviation is set(S27).

Description

A feedback method to secure the consistency in appraisals which consist of the combination quantitative indices and qualitative indices}

The present invention relates to a feedback method applied to an evaluation implemented by an information processing apparatus, and more particularly, to an evaluation feedback method for minimizing deviation by each evaluator and ensuring consistency of evaluation in order to improve the reliability of the evaluation result.

Applicant can systematically manage risks by reflecting business insolvency risks and environmental risks on target technology in order to utilize technology evaluation results for credit, credit guarantee, investment, etc. The technology evaluation method was filed on December 22, 2005, and a patent registration (registration number: 10-0713546) was issued on April 24, 2007.

However, in the method of calculating the grade by a combination of metric and non-measurement indicators including the technical evaluation method, if the evaluator makes a bias or inconsistent evaluation of the non-measurement evaluation index, the reliability of the evaluation result There is a problem that is difficult to secure.

The present invention was devised to solve the above problems, and an object thereof is to provide a feedback method and an information processing process for minimizing the deviation of each evaluator and ensuring the consistency of the evaluation in order to improve the reliability of the evaluation result.

In order to achieve the above objects, one aspect of the feedback method for securing the evaluation consistency according to the present invention is (a) the information processing apparatus is to calculate the evaluation rating first by the evaluation method combined the input metering and non-metering indicators step; (b) calculating, by the information processing apparatus, the difference between the mode value and the individual grade value for each evaluation case and calculating the average of each evaluation case and the standard deviation of each evaluation case; (c) For each evaluation grade calculated by the information processing device, the average and standard deviation of the average for each evaluation case are calculated and set as the propensity mean for each grade and the standard deviation for the propensity by grade, respectively. Calculating standard deviations and setting the standard deviations as the consistency mean for each grade and the standard deviation for each consistency; And (d) calculating, by the information processing apparatus, a propensity value and a consistency value for the evaluator's evaluation case.

가 나타날 것으로 기대되는 확률

을 구하고, 상기 확률

에 의해 평가자 반응

를 위한 표준잔차

를 산출하는 단계;를 포함하여 구성된다.In order to achieve the above objects, another aspect of the feedback method for ensuring evaluation consistency according to the present invention comprises the steps of: (a) the information processing apparatus comprising the evaluation data of a plurality of evaluators for a plurality of items in the matrix data; (b) the information processing apparatus comprising binary matrix data indicating " 1 " and " 0 " (c) calculating, by the information processing apparatus, an initial item propensity measurement value and an initial evaluator propensity measurement value based on the binary matrix data; (d) calculating, by the information processing apparatus, item expansion factors and evaluator expansion factors in consideration of the degree of dispersion of the data; (e) the information processing apparatus multiplying and normalizing the initial item propensity measurement value and the expansion factor corresponding to the initial evaluator propensity measurement, respectively, to calculate a standardized final item propensity measurement value and a standardized final evaluator propensity measurement; And (f) the information processing apparatus responds to the evaluator response by the Rasch model based on the standardized final item propensity measure and the standardized final appraiser tendency measure.

Is expected to appear

Find the probability

Evaluator Response by

Standard residual for

Comprising; is configured to include.

According to the present invention, in the evaluation method in which the quantitative and non-measurement indicators are combined, the propensity value and the consistency value of the evaluator's evaluation case are calculated, and if the calculated value is out of a predetermined range, the feedback is provided to the evaluator to minimize the deviation for each evaluator. It is possible to ensure consistency of evaluation, and ultimately to improve the reliability of technology evaluation results.

In addition, in the case of using the Rasch model, not only the propensity and consistency of the evaluator, but also the propensity and consistency of the items can be obtained, and statistical verification of whether the overall evaluation is consistent through model verification can be obtained.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. The present invention can be applied to all evaluation methods in which the metered and non-measured indicators are combined and evaluated, and specific examples will be described based on the technical evaluation method of the applicant's patent registration (registration number: 10-0713546).

<Example using the difference between an individual grade value and a mode>

FIG. 1 illustrates the overall operation of a technology evaluation method to which a feedback method is applied to ensure consistency of evaluation in which the metered and non-metered indicators are combined according to the present invention. Each step of the invention according to FIG. 1 is performed by a combination of a processor (not shown) and associated software provided in an information processing apparatus (for example, a server connected on the Internet).

First, in the process of inputting the technical evaluation table, necessary values are input for 45 evaluation indexes by the technical evaluation method of patent registration (registration number: 10-0713546) (S10). Subsequently, in the feedback process, the results are collected and the propensity value and the consistency value of the evaluator are output on the screen (S20), and when the propensity value or the consistency value for the evaluator's evaluation is out of a predetermined level, the information processing apparatus is evaluated by the evaluator. Re-check request (S30). On the other hand, if re-confirmation is not required, the technical project evaluation grade for the case is output (S40).

Hereinafter, the feedback process S20 will be described in more detail.

First, the technology business evaluation grade is first calculated by the technology evaluation method of patent registration (registration number: 10-0713546) for each evaluation case based on 45 evaluation indicators input in step S10 (S21). Detailed process of step S21 is already known, so a detailed description thereof will be omitted. The technology business evaluation grade consists of 10 grades ranging from AAA to D grade.

The input values of the 45 evaluation indicators are shown in Table 1, and the sector of the evaluation case is "electrical / electronic", and the technical project evaluation grade of the evaluation case is "BBB". In general, the configuration of each technology business evaluation grade ("BBB" grade in this embodiment) is as shown in FIG.

Evaluation index Individual grade Non-quantitative
Individual grade value 1.1 Management Experience Level A 1.2 Technology Management Strategy B 4 1.3 Technical knowledge level of managers D 1.4 Management Skills Knowledge A 5 2.1 Technical personnel management B 2.2 Crisis Response Capacity A 5 2.3 Management will and business A 5 3.1 Educational background and career of management B 3.2 Capital Involvement C 3.3 Relationship with management and teamwork A 5 4.1 Organization dedicated to technology development A 4.2 Technical (Design) Manpower D 5.1 Technology Development and Awards C 5.2 Status of Intellectual Property Rights, etc. E 5.3 R & D Investment Ratio C 6.1 Technology Differentiation B 4 6.2 Difficulty Level A 5 6.3 Location in the life cycle of the technology A 5 7.1 Completeness of Technology D 2 7.2 Independence of Technology B 4 7.3 Conformity with Business Strategy A 5 8.1 Ripple Effects B 4 8.2 Application and Scalability B 4 9.1 Market Structure C 3 9.2 Competition in the same industry B 4 9.3 Market Access A 5 10.1 Market Size B 4 10.2 Market Growth B 4 10.3 Restrictions / Incentives including laws and regulations C 3 11.1 Awareness A 5 11.2 Market Share C 3 11.3 Comparative Advantages with Alternatives B 4 12.1 Ease of Securing Production Facilities A 5 12.2 Easy to secure production personnel A 5 12.3 Ease of Procurement of Materials and Parts B 4 13.1 Capital Raising Capacity B 4 13.2 Value-added ability C 13.3 Appropriateness of Investment Scale B 3 14.1 Feasibility of Sales Plan A 5 14.2 Diversity and stability of distributor A 5 14.3 Recruit sales and marketing personnel B 4 15.1 Sales Growth C 15.2 Net Profit Margin Before Income Tax Expenses C 16.1 Recoverability compared to investment C 3 16.2 Return on Investment C

Where A = 5, B = 4, C = 3, D = 2, E = 1

Next, the mode of each non-measurement index of the corresponding grade is calculated by classifying it by the technology project evaluation grade based on the calculated data on past evaluation cases.

The mode calculation method is based on equation (1). In Equation 1, MODE is a formula for deriving the mode value in statistics, and represents the most frequent individual rating value in the evaluation index.

Thereafter, as shown in Table 2, the difference between the mode value and the individual grade value for each non-metric indicator is calculated (S23).

turn Evaluation index Mode of BBB class without electric and electronic
Y _(i) Individual grade value
X _{(i, j)} Difference Value (Individual Rating Value-Mode)
W _{(i, j)} = X _{(i, j)} -Y _(i) One 1-2 Technology Management Strategy 4 4 0 2 1-4 Managerial Skills 4 5 One 3 2-2 Crisis Response Ability 4 5 One 4 2-3 Management will and business 4 5 One 5 3-3 Relationship with managers and teamwork 5 5 0 6 6-1 Differentiation of Technology 4 4 0 7 6-2 Difficulty Level 3 5 2 8 6-3 Lifecycle Position of Technology 3 5 2 9 7-1 Completeness of Technology 5 2 -3 10 7-2 Independence of Technology 4 4 0 11 7-3 Conformity with Business Strategy 4 5 One 12 8-1 Technology Ripple Effects 4 4 0 13 8-2 Technology Application and Scalability 3 4 One 14 9-1 Market Structure 4 3 -One 15 9-2 Competition in the same industry 4 4 0 16 9-3 Market Accessibility 4 5 One 17 10-1 Market Size 4 4 0 18 10-2 Market Growth 4 4 0 19 10-3 Restrictions / Incentives 3 3 0 20 11-1 Awareness 4 5 One 21 11-2 Market Share 3 3 0 22 11-3 Comparative Advantages with Substitutes 4 4 0 23 12-1 Easy to secure production facilities 4 5 One 24 12-2 Easy to secure production manpower 4 5 One 25 12-3 Ease of Procurement of Materials and Parts 4 4 0 26 13-1 Capital Raising Capacity 4 4 0 27 13-3 Appropriateness of Investment 3 3 0 28 14-1 Feasibility of Sales Plan 4 5 One 29 14-2 Diversity and Stability 4 5 One 30 14-3 Acquiring Marketing Manpower 4 4 0 31 16-1 Return on Investment 4 3 -One Mean of the differences (g _(j) ) 0.323 Standard Deviation of Difference (h _(j) ) 0.945

Next, for each of the calculated technical project evaluation grades (for example, BBB), the average and standard deviation of the difference between the mode value and the individual grade value for each evaluation case are calculated as in Equation 2 (S25).

Thereafter, for each technology project evaluation grade, as shown in Equation 3, the average and standard deviation of the average for each evaluation case are calculated and placed as the standard deviation of the propensity average and the propensity standard per grade, respectively (S27).

In addition, for each technical project evaluation grade, the average and standard deviation of the standard deviation for each evaluation case are calculated for the entire evaluation case, as shown in Equation 4, and the standard deviation is given as the consistency average for each grade and the standard deviation for each grade.

Therefore, reference values for the grade propensity and consistency (grade propensity average, grade propensity standard deviation, grade coherence mean, and grade coherence standard deviation) are set as illustrated in Table 3.

The propensity and consistency of the evaluator can be calculated by comparing the individual rating values of the appraisers with the mode-specific mode and the reference values for the propensity and consistency by class. That is, as shown in FIG. 2, the propensity and consistency of the evaluator can be measured by normalizing the sample mean to 0 and the sample dispersion to 1.

The propensity and consistency of the evaluator's evaluation case is calculated as in Equation 5 (S29).

The propensity and consistency for the exemplified assessments (individual assessments belonging to the BBB class of the electrical and electronics industry) is calculated as follows.

Pp (evaluator propensity value) and Pc (evaluator consistency value) are derived as standardized values, and the reference value can be determined according to the application direction of the feedback process. These reference values can be applied based on the ± 2σ level, which is usually 95.45% in the normal distribution.

If the value of the evaluator's propensity is in the positive direction, it means that it shows a generous tendency compared with the evaluation results of the corresponding grade, and if it shows the negative direction, it means the strict inclination.

If the evaluator consistency value indicates a positive direction, it means that the evaluation index is randomly performed. If the evaluator consistency value indicates a negative direction, it means that the same grade is continuously given to the evaluation index.

Positive and negative values within the ± 2σ level suggested by the present invention mean an allowable error, and the information processing apparatus requests the evaluator to reconfirm the error when the ± 2σ level is exceeded. Send a signal. The reference value can be changed continuously according to the data analysis result and the policy direction.

The information processing apparatus can display on the plane graph with the calculated propensity for the evaluator's evaluation case as the x-axis and the coherence value as the y-axis. Referring to FIG. 3, a value (2.043) in which an evaluator propensity value exceeds 2 sigma (σ) is derived for the illustrated evaluation case, and a "first propensity is generous" screen is output and fed back to the evaluator.

On the other hand, the feedback process is not only classified by the evaluation grade, but may be classified by the industry and evaluation grade to which the evaluation target company belongs. Therefore, the baseline values for the propensity and consistency of the evaluator are not the baseline values for the propensity and consistency for each grade, but for the propensity and consistency for each industry / grade. Reference values for propensity and consistency can be used.

<Example using Rasch model>

The Rasch model estimates the evaluator's response to an item by a statistical model and measures the evaluator's response correcting the difference between the items. The most commonly used statistical model in the Rasch model is the logit model. The Rasch model, which is a statistical model using probabilistic concepts, is used to correct the difference of items, and then the evaluator's disposition and consistency are obtained. In this embodiment, not only the propensity and consistency of the evaluator can be obtained, but also the propensity and consistency of the items can be obtained, and a statistical measure of whether the overall evaluation is consistent through model verification can be obtained.

First, the Rasch model, which is an analysis model, is briefly described as follows.

는 반응값으로, 최대빈도 등급 이상의 등급이면 1이고, 그렇지 않으면 0이다. 1일 확률이 크면 높은 등급을 받을 확률이 크며, 0일 확률이 크면 낮은 등급을 받을 확률이 크다.here,

Is the response value, 1 for grades above the maximum frequency grade, and 0 otherwise. The greater the probability of 1, the higher the probability of getting a higher grade, and the greater the probability of 0, the higher the chance of getting a lower grade.

는 평가자 성향으로,

가 크면 높은 점수를 주는 성향을 보이는 평가자로 관대한 성향을 의미하며,

가 작으면 낮은 점수를 주는 성향으로 엄격함을 의미한다.

Is an evaluator propensity,

A high value means a tendency to give high score and tenderness.

A small value means strictness with a low score propensity.

는 문항 성향으로,

가 크면 낮은 점수를 받는 문항을 의미하며,

가 작으면 높은 점수를 받는 문항을 의미한다.

Is an item inclination,

A large value means a low scored item.

If is small, it means the item that gets high score.

Hereinafter, with reference to FIG. 4 will be described in detail the feedback process for ensuring the consistency of technology evaluation using the Rasch model. Each step of the present invention by FIG. 4 is performed by a combination of a processor (not shown) and associated software provided in an information processing apparatus (for example, a server connected on the Internet).

① Composition of data

In order to estimate parameters using the Rasch model, it is necessary to construct matrix data of responses expressed by the evaluator for each item as shown in Table 4 as data representing the evaluator's response to each item (S51).

When there are evaluators' evaluation grades for the target evaluation items, the maximum frequency rating is first summarized for each evaluation item.

In the case of giving a score of more than the maximum frequency rating for each item of the evaluator, the Rasch model can be applied by arranging it in binary data form as shown in Table 5, which is indicated by "1", otherwise "0" (S53).

In this case, items or evaluators that are all "1" or all "0" are excluded from the calculation.

② Measurement of the initial item disposition and initial evaluator disposition measurement (S55)

번째 문항에 대해서 문항점수(

)를 전체 평가자의 수(N)로 나누어 준 것이 그 문항의 "1"로 채점될 확률(

)이다.

를 Rasch 모형에 의한 식을 사용하는 대신에 간편하게 '문항점수(

)/표본 수(N)'에 의하여 계산한다. 문항 성향을 구하기 위해서는 "0"일 확률이 필요한데, 1-

에 의해서 계산될 수 있다. 각 문항에서 성향을 구하기 위하여 "0"이 될 승산비(Odds ratio)를 구하고, 그 값을 자연로그(ln)로 취하여 "0"인 로짓(Logit)을 계산한다. 마지막으로 초기 문항 성향 측정치(

)를 각 문항의 "0"인 로짓으로부터 평균 로짓을 감산해 줌으로써 산출해낼 수 있다. 여기서 문항 측정치가 크면 클수록 평균이하의 낮은 등급으로 채점되는 확률이 큼을 의미한다. The propensity of each item is estimated based on the responses of the evaluators in each item. Basically, item propensity is a low grade, so the evaluator uses the probability of evaluating "0". The more "0" items the appraiser rated, the lower the grade was.

The item score for the first item

) Divided by the total number of evaluators (N) is the probability that the item will be scored as "1"

)to be.

Instead of using an equation by the Rasch model,

) / Number of samples (N) '. To find the item disposition, we need a probability of "0".

Can be calculated by In order to find the propensity for each item, the odds ratio to be "0" is obtained, and the value is taken as the natural log (ln), and the logit of "0" is calculated. Finally, measure initial item disposition (

) Can be calculated by subtracting the average logit from the logit of "0" in each item. The larger the item measure, the higher the probability of scoring below the average.

)를 산출하는 과정은 수학식 7과 수학식 8에 의해 표현되고, 표 6에 예시적인 값이 표시된다. Initial item disposition measure (

) Is represented by equations (7) and (8), and exemplary values are shown in Table 6.

In Table 6, the item score is the sum of the number of ratings scored by the evaluator for each item at "1", and the probability of being "1" is (the sum of the number of evaluators with rating score "1") 9), the probability of "0" is obtained by 1- (probability of "1"), and the logit for "0" is obtained by taking the natural logarithm of the odds ratio of "0" to "0". The initial item propensity is calculated by subtracting the average logit from the logit for "0" of each item, which means that the larger the initial item propensity, the higher the probability of scoring below the average.

번째 평가자에 대해서 "1"일 확률(

)을 계산하는데, 이는 평가자가 각 문항에 대해서 "1"로 채점한 문항의 합(

)을 전체 문항들의 수(L)로 나누어줌으로써 구할 수 있다. 평가자에 대한 "1"인 로짓(

)을 구하기 위하여 평가자가 "0" 대신에 "1"을 할 확률의 승산비를 구하고, 이 값에 자연로그를 취하여 "1"인 로짓(

)을 계산한다. 초기 평가자 성향 측정치(

)는 각 평가자의 "1"인 로짓으로부터 그것의 평균 로짓을 감산해 줌으로써 산출해 낼 수 있다. 여기서 평가자 성향이 크면 클수록(즉, 양의 방향으로 갈수록) 높은 등급으로 채점하는 성향으로 관대함을 작으면 작을수록(즉, 음의 방향으로 갈수록) 엄격함을 의미한다. Measuring the evaluator's disposition is to determine how much the evaluator scores above the maximum frequency rating. Therefore, the probability that the evaluator "1" is used to calculate a measure indicating the evaluator's propensity. To determine the probability that an evaluator will score "1" on items

Probability of "1" for the first evaluator (

), Which is the sum of the items the evaluator scored as "1" for each item (

) Can be found by dividing the total number of items by L. Logit, which is "1" for the evaluator (

) To find the odds ratio of the probability that the evaluator will do "1" instead of "0", and take the natural logarithm of this value and logit ("1").

). Initial evaluator propensity measurements (

) Can be calculated by subtracting its average logit from the logit that is "1" of each evaluator. The larger the evaluator's propensity (i.e., the positive direction), the higher the tendency to score higher grades; the smaller the generosity, the smaller (i.e., the negative direction), the more stringent.

)를 산출하는 과정은 수학식 9과 수학식 10에 의해 표현되고, 표 7에 예시적인 값이 표시된다. Initial evaluator propensity measurements (

) Is represented by Equations 9 and 10, and exemplary values are shown in Table 7 below.

In Table 7, the evaluator score is the sum of the number of items for which the evaluator scored the grade score as "1", and the probability of "1" is (the sum of the number of items with "1") / (total question 9) Is obtained by 1- (probability of being "1"), and the logit for "1" is obtained by taking a natural logarithm of the odds ratio of "1" to "0", The initial evaluation unit tendency is calculated by subtracting the average logit from the logit for each evaluator's correct answer, which means that the larger the initial evaluation unit tendency is, the more generous the evaluation unit tends to be.

③ Expansion factor calculation (S57)

It is necessary to calibrate the measurement in consideration of the dispersion of the data. Item measures should consider the variance of evaluator disposition, and evaluator measures should adjust the initial measures to account for the dispersal of item disposition. In this case, an expansion factor (Y) for adjusting the item measurement in consideration of the variance of the evaluators is calculated by Equation 11.

Y=

Y =

On the other hand, the expansion factor (X) for adjusting the evaluator measurement in consideration of the difficulty distribution of the items is calculated by Equation 12.

X=

X =

Table 8 shows example values for item expansion factors and evaluator expansion factors.

Expansion factor Y (item expansion factor) 2.10 X (expansion factor) 2.01

④ Calculation of final item propensity measurement value and final evaluator propensity measurement value (S59)

)에 확장요인(Y)을 곱해줌으로써 교정된 최종 문항 성향의 측정치(

)가 산출된다. 표준화된 최종 문항 성향 측정치(Z)는 교정된 최종 문항 성향의 측정치(

)를 문항 측정치의 표준오차 SE(

)로 나누어 산출하는데, 이때 각 문항 측정치의 표준오차 SE(

)는 수학식 13에 의하여 계산된다.Measurement of initial item propensity for each item (

) Is a measure of the final item propensity corrected by multiplying the expansion factor (Y) (

) Is calculated. The standardized final item propensity measure (Z) is a measure of the corrected final item propensity (

) Is the standard error SE of the item measurement.

Dividing by), where the standard error SE of each measurement

Is calculated by (13).

Table 9 shows exemplary values of the final item propensity measurement.

Calculate final item propensity measurement Item Number score Initial Item Measurement Expansion factor (Y) Last question measurement Measurement Standard Error (SE) Z value One 8 -1.64 2.1 -3.44 2.23 -1.54 2 8 -1.64 2.1 -3.44 2.23 -1.54 3 7 -0.82 2.1 -1.72 1.68 -1.02 4 6 -0.26 2.1 -0.55 1.48 -0.37 5 6 -0.26 2.1 -0.55 1.48 -0.37 6 4 0.66 2.1 1.39 1.41 0.99 7 6 -0.26 2.1 -0.55 1.48 -0.37 8 2 1.69 2.1 3.55 1.68 2.11 9 One 2.52 2.1 5.29 2.23 2.37

According to Table 9, items with a high probability of "0" are items with Z greater than 2, items 8 and 9, and items with a low probability of "1" are items with Z less than -2. Is not.

)에 확장요인(X)을 곱해줌으로써 교정된 최종 평가단위 성향 측정치(

)가 산출된다. 표준화된 최종 평가자 성향 측정치(Z)는 교정된 최종 평가자 성향의 측정치(

)를 평가자 측정치의 표준오차 SE(

)로 나누어 산출하는데, 이때 각 평가자를 위한 측정치의 표준오차 SE(

)는 수학식 14에 의하여 계산된다.Final evaluator disposition measures can also be calculated by the same method as item measures. Initial evaluator measurements for each evaluator (

) Multiplied by the expansion factor (X)

) Is calculated. The standardized final evaluator disposition measure (Z) is a measure of the corrected final evaluator disposition (

) Is the standard error SE of the evaluator's measurements.

Dividing by), where the standard error SE (

Is calculated by equation (14).

Table 10 shows exemplary values of the final evaluator propensity measurement.

Calculate final unit of measure propensity Evaluation unit number score Initial Evaluation Unit
Measure Expansion factor (X) Final measurement Measurement Standard Error (SE) Z value One 8 -2.48 2.01 -4.98 2.13 -2.33 2 8 -2.48 2.01 -4.98 2.13 -2.33 3 7 -0.18 2.01 -0.36 1.61 -0.22 4 7 -0.18 2.01 -0.36 1.61 -0.22 5 6 0.29 2.01 0.58 1.42 0.41 6 5 0.85 2.01 1.71 1.35 1.27 7 5 0.85 2.01 1.71 1.35 1.27 8 One 1.67 2.01 3.36 2.13 1.58 9 One 1.67 2.01 3.36 2.13 1.58 10 except

According to Table 10, an evaluator with a high probability of "1" has a Z value greater than 2, and no evaluator has a value. An evaluator with a high probability of "0" has a Z value less than -2. This is the evaluator.

⑤ Calculation of the residuals (S61)

가 나타날 것으로 기대되는 확률을 수학식 15의

에 의해 계산할 수 있다.Unit response by Rasch model

The probability that is expected to appear is

Can be calculated by

의 예시적인 값이 표시된다. Table 11 shows the expected probabilities

Exemplary values of are shown.

One 2 3 4 5 6 7 8 9 One 1.00 1.00 0.99 0.98 0.98 0.88 0.98 0.45 0.13 2 1.00 1.00 0.99 0.98 0.98 0.88 0.98 0.45 0.13 3 0.99 0.99 0.97 0.91 0.91 0.58 0.91 0.14 0.03 4 0.99 0.99 0.97 0.91 0.91 0.58 0.91 0.14 0.03 5 0.98 0.98 0.91 0.76 0.76 0.31 0.76 0.05 0.01 6 0.96 0.96 0.80 0.55 0.55 0.15 0.55 0.02 0.00 7 0.96 0.96 0.80 0.55 0.55 0.15 0.55 0.02 0.00 8 0.18 0.18 0.04 0.01 0.01 0.00 0.01 0.00 0.00 9 0.18 0.18 0.04 0.01 0.01 0.00 0.01 0.00 0.00

를 위한 표준잔차

를 수학식 16에 의해 추정할 수 있다. Unit response

Standard residual for

Can be estimated by equation (16).

의 예시적인 값이 표시된다. The standard residual calculated in this way has a normal distribution with an average of 0 and a variance of 1. Table 12 shows the standard residuals

Exemplary values of are shown.

According to Table 12, if the absolute value of the residual is greater than 2, the evaluator 3, the 8 evaluator and the 9 evaluator correspond.

를 이용하여 문항과 평가자의 일관성 측도를 계산할 수 있다(S63).The standard residuals thus obtained

Using can calculate the consistency measure of the item and the evaluator (S63).

First, the equation for calculating the measure of consistency of an item is the same as Equation 17. If V is greater than 2, it is determined that the consistency of the item is inappropriate.

 Table 13 shows exemplary values of the consistency measure of the item.

Item Number Question_V | Question_t |> 2 One 0.62 2 0.62 3 0.10 4 1.46 5 0.33 6 0.32 7 0.33 8 0.35 9 4.53 v

In Table 13, the item whose absolute value of the item's consistency measure (Item_V) is 2 or more is item 9, and item 9 is out of the measure of consistency and needs to be reviewed.

Next, the equation for calculating the measure of consistency of the evaluator is shown in Equation 18. If V is greater than 2, it is determined that the evaluator is inconsistent.

Table 14 shows exemplary values of the measure of consistency of evaluators.

Evaluator number Evaluator_V | Question_t |> 2 One 0.20 2 0.20 3 5.83 v 4 0.16 5 0.20 6 0.43 7 0.43 8 0.62 9 0.62

In Table 14, because the item whose absolute value of the evaluator's consistency measure (evaluator_V) is 2 or more is evaluator 3, evaluator 3 needs to review the evaluator from the consistency measure. In this case, the information processing apparatus may send a signal requesting reconfirmation to the evaluator 3 (S65).

After standardizing the standardized final item propensity measurement at the step S59 and the item consistency measure at the step S63, the propensity and consistency for the non-metric indicators can be analyzed by displaying them on a two-dimensional chart as shown in FIG. 5.

Referring to FIG. 5, the items having the absolute value (propensity) of the item_Z value of 2 or more are questions 8 and 9, and the items having the absolute value (consistency) of the item_V value of 2 or more are item 9. There is.

In addition, the standardized final measure of evaluator tendency in the step S59 and the measure of consistency of the evaluator in the step S63 are normalized and then displayed on a two-dimensional diagram as shown in FIG. Can be.

According to FIG. 5, it can be seen that items having an absolute value (propensity) of the evaluator_Z value of 2 or more are evaluator 1 and an evaluator 3, and an item having an absolute value (consistency) of the evaluator_V value of 2 or more is evaluator 3 have.

In addition, statistical verification of the entire evaluation system may be performed through a goodness of fit as shown in Equation 19.

이면 통계적으로 검토하였을 때 평가 시스템의 일관성이 유지되고 있다고 판단하며,

일때는 시스템의 일관성이 유지되고 있지 않아 개선이 필요하다는 결론을 내린다.if

If it is statistically reviewed, it is determined that the evaluation system is consistent.

Sometimes concludes that the system is inconsistent and needs improvement.

Table 15 shows exemplary values for calculating model fit.

One 2 3 4 5 6 7 8 9 One 0.001 0.001 0.006 0.020 0.020 0.139 0.020 1.208 0.145 2 0.001 0.001 0.006 0.020 0.020 0.139 0.020 1.208 0.145 3 0.006 0.006 0.032 9.545 0.105 0.723 0.105 0.159 35.945 4 0.006 0.006 0.032 0.105 0.105 0.723 0.105 0.159 0.028 5 0.018 0.018 0.100 0.324 0.324 0.447 0.324 0.051 0.009 6 0.046 0.046 0.256 0.830 0.830 0.174 1.204 0.020 0.004 7 0.046 0.046 0.256 0.830 1.204 0.174 0.830 0.020 0.004 8 0.215 4.646 0.038 0.012 0.012 0.002 0.012 0.000 0.000 9 4.646 0.215 0.038 0.012 0.012 0.002 0.012 0.000 0.000 Question sum 4.984 4.984 0.766 11.698 2.633 2.523 2.633 2.826 36.279 total 69.327

로 2보다 작기 때문에 평가 시스템이 전체적으로 일관성을 유지한다고 할 수 있다.According to Table 15,

Since it is less than 2, it can be said that the evaluation system is overall consistent.

Table 16 exemplarily shows the averages of the consistency and disposition analysis by characteristics of individual evaluators.

rater Evaluator number tendency consistency One -2.34 0.20 2 -2.34 0.20 3 -0.22 5.83 4 -0.22 0.16 5 0.41 0.20 6 1.27 0.43 7 1.27 0.43 8 1.57 0.62 9 1.57 0.62 Evaluator average 0.96

Meanwhile, the embodiments of the present invention described above may be recorded in a medium used in a general purpose computer including a personal computer. The medium may be a magnetic recording medium (e.g., ROM, floppy disk, hard disk, etc.), an optical reading medium (e.g., CD-ROM, DVD, etc.), an electrical recording medium (e.g., flash memory, memory stick, etc.) And record carriers such as carrier waves (eg, transmission over the Internet).

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

FIG. 1 illustrates the overall operation of an evaluation method to which a first embodiment of a feedback method for ensuring consistency of evaluation combined with a metered and non-metered index according to the present invention is applied.

2 illustrates the conversion of a normal distribution to a standard normal distribution.

FIG. 3 exemplarily shows the results fed back to the evaluator on a flat graph having the propensity value for the evaluation case as the x-axis and the coherence value as the y-axis.

4 is a flowchart illustrating a second embodiment of a feedback method for ensuring consistency of evaluation according to the present invention.

5 exemplarily shows the propensity and consistency of the item on a two-dimensional diagram.

6 exemplarily shows the propensity and consistency of the evaluator on a two-dimensional diagram.

FIG. 7 exemplarily illustrates a configuration of evaluation cases having a technology business evaluation rating of “BBB” in the electric and electronic industry.

Claims (16)

(a) receiving, by the information processing apparatus, a value of the evaluation index in which the metering and non-measurement indexes are combined, and calculating the evaluation grade first by the value of the input evaluation index; (b) calculating, by the information processing apparatus, the difference between the mode value and the individual grade value for each evaluation case and calculating the average of each evaluation case and the standard deviation of each evaluation case; (c) For each evaluation grade calculated by the information processing device, the average and standard deviation of the average for each evaluation case are calculated and set as the propensity mean for each grade and the standard deviation for the propensity by grade, respectively. Calculating standard deviations and setting the standard deviations as the consistency mean for each grade and the standard deviation for each consistency; (d) The information processing apparatus calculates the propensity value and the consistency value for the evaluator's evaluation.

Calculating by; (e) displaying, by the information processing apparatus, on the planar graph with the propensity value of the calculated evaluator's evaluation case as the x-axis and the coherence value as the y-axis; And (f) if the propensity value or consistency value of the evaluator's evaluation is out of a predetermined level, requesting the evaluator to reconfirm the evaluator. delete delete The method of claim 1, wherein step (b) Feedback method for ensuring consistency of evaluation, which is performed only for non-metric evaluation indicators among evaluation indicators of each evaluation case. The method of claim 1, wherein step (b) to (d) Feedback method to ensure consistency of evaluation, which is performed by each industry, evaluation grade, and rater. A computer-readable recording medium having recorded thereon a program in which the method of any one of claims 1, 4 and 5 is implemented. delete delete delete delete delete delete delete delete delete delete

Images