CN104091298B - Implementation method of mutual evaluation system - Google Patents

Abstract

The invention provides an implementation method of a mutual evaluation system. The method comprises the steps that the mutual evaluation system receives all homework submitted by students and reference versions, the evaluation standards and the homework samples corresponding to the homework at all the time and submitted by teachers; for the homework at each time, the corresponding reference version, the evaluation standard and the homework sample are sent to all the students by the mutual evaluation system, all the homework submitted by the students is distributed to two or more students for conducting evaluation randomly, and the students feed back evaluation results to the mutual evaluation system; the mutual evaluation system synthesizes the evaluation results of the homework of multiple times of the students and processes the data submitted by the students to obtain a final score result. The mutual evaluation system achieves division and intelligent computing of score results through the mode, the correctness and the reliability of a finally-obtained score result are improved, automatic and intelligent processing to the data submitted by the students is achieved, and the teaching efficiency is effectively improved.

Description

A kind of implementation method mutually commenting system

Technical field

The invention belongs to teaching technique field, it is related to data statisticss technology, be a kind of implementation method mutually commenting system.

Background technology

Teacher, in teaching process, needs students' work is carried out correcting scoring, to understand teaching efficiency, controls teaching Quality.

The mode corrected students' papers is roughly divided into Three models: teacher comment (expert's assessment), computer judge (artificial intelligence), Student corrects mutually (life life is mutually commented).

In Three models, expert's assessment is most reliable appraisal procedure, but due to of a high price, is not suitable for extensive (as university's public course, extensive online teaching) is applied in teaching.

Artificial intelligence is only used for the limited objective problem assessment of answer, to subjective problem, such as mathematics at present Proof, programming, writing etc. are difficult to provide rational evaluation.

It is that student evaluates to oneself and other students' works under conditions of with reference to standards of grading that raw life is mutually commented, this machine System is applied on moocs platform cousera, for mitigating teachers ' teaching burden, allows student more participate in teaching In, improve teaching efficiency.Mutually comment including below scheme:

1. operation is submitted to.Student before the off period by operation in Submitted online.

2. evaluate.Teacher announces Key for Reference and standards of grading, is that every student provides that a certain amount of (3-5 part) is pending to transform into Industry.Student submitted to before the off period and corrects result.

3. pair correcting the student that result is discontented with can submit to the person of correcting to reappraise.Arbitrated by teacher if necessary.

4. appraise.System, according to multiple marking's result, determines operation score by certain rule.

The raw of prior art mutually comments the electronic system great majority of use only data to be received and transmitted, and life life is mutually commented Substantially still student carries out subjective assessment to student.In order to improve teaching automatization, improve teaching efficiency, how according to raw Mutually comment result rationally to determine operation score, and be not only transmitting and receiving data, be mutually to comment system to need the matter of utmost importance solving. The mode that research is mentioned is had to take middle position counting method, method of averaging and remove maximum, average after minima at present Deng.The achievement that said method is given, only using the appraisal result of scoring person as judgement basis, nevertheless suffers from scoring person's ability and master See the impact of wish, the reliability of achievement there is a problem of certain.

Content of the invention

The problem to be solved in the present invention is: the life life adopting in existing teaching automated system mutually comments system, its realization Mode is subject to scoring person's subjective impact big, and accuracy, reliability are not high enough it is impossible to effective realize improving teaching automatization and teaching The purpose of efficiency.

The technical scheme is that a kind of implementation method mutually commenting system, in a course, teacher arranges many to student Subjob, mutually comment system receive student submit to All Jobs, and teacher submit to the Key for Reference of each subjob of correspondence, comment Minute mark is accurate and operation sample, described operation sample by teacher by the job design arranged the operation comprising many places mistake； For each subjob, mutually comment system that corresponding Key for Reference, standards of grading and operation sample are issued each student, and will be every The operation that part student submits to is given more than 2 students at random and is evaluated, and evaluation result is fed back to system of mutually commenting by student；

After End-of-Course, mutually comment system synthesis student to multiple job eveluation result, process student by following items and submit to Data:

1) the th subjob that student txh submits to is made up of several point scorings, and dfh is the numbering of point scoring, this score The score value weight of point is m (th, dfh), and student pxh is scored to point scoring dfh as scoring student, df (pxh, txh, Th, dfh) appraisal result for student pxh, 0 represents not score, 1 expression score, and -1 represents and do not score, such as score then score value by m (th, dfh) is calculated；

2) evaluating ability of calculating student pxh:

Diagnostic sensitivity is defined to scoring student as follows:

In the All Jobs sample that student pxh evaluates, comprise n mistake, that is, have n point scoring evaluation of teacher must not be Point, student pxh is evaluated as not score to i point scoring in described n point scoring, then the diagnostic sensitivity of student pxh is fixed Justice is:

$se=\frac{i}{n}$

Specificity is defined to scoring student as follows:

In the All Jobs sample that student pxh evaluates, m point scoring evaluation of teacher is had to be score, student pxh is to described m The scoring of j point scoring in individual point scoring is score, then the Student Diagnosis specificity for pxh for the student number is defined as

$sp=\frac{j}{m}$

3) the operation score of calculating student txh:

3.1) for the point scoring that evaluation is consistent:

Submit in each point scoring of operation in student txh, the point scoring that all scoring students are unanimously chosen as score determines For score, the point scoring being unanimously chosen as not score is defined as not score；

3.2) error generation rate of estimation students' work:

Student's txh submission is obtained evaluating consistent point scoring, the number being wherein chosen as score is t₁, must not be chosen as The number dividing is t₂, then the error generation rate of student txh be:

$er=\frac{t_2}{t_1+t_2}$

3.3) for evaluating inconsistent point scoring, the operation not score that nonuniformity is evaluated is calculated according to Bayesian model Probability:

If a total of n scoring student independently scores to the same point scoring of student txh, corresponding diagnostic sensitivity It is respectively se with specificity₁,se₂,...,se_n,sp₁,sp₂,...,sp_nIf wherein front k Students ' Evaluation is divided into 0, rear n-k Individual Students ' Evaluation is divided into 1, if the corresponding event of scoring is a₁,a₂,...,a_kAnda₁,a₂,...,a_kRepresent k Raw point scoring is scored for 0,After expression, n-k student scores as 1 to point scoring, operation submitter txh's Error rate is er,

Remember branch not score event be b, according to condition probability formula, not scoring probability For:

$\begin{array}{c}p(b/a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})=\frac{p({\mathrm{ba}}_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})}{p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})}\\ =\frac{p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/b)p\left(b\right)}{p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})}\end{array}$

Score between due to each student separate, obtain:

$p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/b)=\underset{p=1}{\overset{k}{\π}}p(a_p/b)\underset{q=k+1}{\overset{n}{\π}}p(\stackrel{\&overbar;}{a_q}/b)=\underset{p=1}{\overset{k}{\π}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\π}}(1-{\mathrm{se}}_q)$

According to total probability formula, have:

$\begin{array}{c}p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})\\ =p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/b)p\left(b\right)+p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/\stackrel{\&overbar;}{b})p\left(\stackrel{\&overbar;}{b}\right)\\ =\underset{p=1}{\overset{k}{\π}}p(a_p/b)\underset{q=k+1}{\overset{n}{\π}}p(\stackrel{\&overbar;}{a_q}/b)p\left(b\right)+\underset{p=1}{\overset{k}{\π}}p(a_p/\stackrel{\&overbar;}{b})\underset{q=k+1}{\overset{n}{\π}}p(\stackrel{\&overbar;}{a_q}/\stackrel{\&overbar;}{b})p\left(\stackrel{\&overbar;}{b}\right)\\ =\underset{p=1}{\overset{k}{\π}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\π}}(1-{\mathrm{se}}_q)er+\underset{p=1}{\overset{k}{\π}}(1-{\mathrm{sp}}_p)\underset{q=k+1}{\overset{n}{\π}}{\mathrm{sp}}_p(1-er)\end{array}$

Therefore:

$p(b/a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})=\frac{\underset{p=1}{\overset{k}{\π}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\π}}(1-{\mathrm{se}}_q)er}{\underset{p=1}{\overset{k}{\π}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\π}}(1-{\mathrm{se}}_q)er+\underset{p=1}{\overset{k}{\π}}(1-{\mathrm{sp}}_p)\underset{q=k+1}{\overset{n}{\π}}{\mathrm{sp}}_q(1-er)}$

3.4) score according to the inconsistent point scoring of determine the probability evaluation:

Given threshold α, ifThen this point scoring not score, if Then this point scoring score, ifThen mutually comment system commenting this point scoring Valency task is sent to teacher, and α corresponds to teacher and processes intensity, and α is more high, and the probability being sent to teacher's process is higher.

It is preferred that, to threshold alpha, arrange 80% < α < 100%.

Further, mutually comment system according to the scoring evaluation result of student and the final appraisal result meter being evaluated operation Calculate the scoring effective percentage of scoring student, marking system submits the final appraisal result of operation and commenting of this student to each student Divide effective percentage to be respectively provided with weight, be calculated the operation scores of each student by this two parts.

The present invention, based on Bayesian Diagnosis theory, constructs estimation operation actual score mathematical algorithm, and to participation The distinguishing ability of the student of scoring gives rational evaluation.Raw mutually the commenting using at present is the simply unidirectional evaluation of one kind, that is, Estimator evaluates the operation accurateness of evaluated person.And in the present invention, no longer simply come by the evaluation result of scoring student Calculate score, but consider the ability of scoring student, constituted by the diagnostic sensitivity and specificity quantifying Evaluate the weight had in the evaluation of a operation of student, the system of mutually commenting of the present invention achieves to commenting in this way Divide division and the intelligence computation of result, improve the correctness of appraisal result finally giving and reliability, realize automatic intelligent To student submit to data process, effectively increase teaching efficiency.In addition, the present invention also proposes the effective percentage that scores, realize double To evaluation, that is, estimator evaluates the operation accurateness of evaluated person, and the evaluation result of estimator is also evaluated, at this simultaneously Bright mutually comment under system so that random stronger student's appraisal result suffered restraints originally, constantly restrain to correctness.

Specific embodiment

When the inventive method is realized, in a course, teacher arranges many subjobs to student, mutually comments system to receive student and carries The All Jobs handed over, and the Key for Reference of each subjob of correspondence, standards of grading and the operation sample that teacher submits to, described work Industry sample is comprised the operation of many places mistake by teacher by the portion of the job design arranged；For each subjob, mutually comment and be Corresponding Key for Reference, standards of grading and operation sample are issued each student by system, and will be random for the operation of every part of student's submission Give more than 2 students to be evaluated, evaluation result is fed back to system of mutually commenting by student.In simple terms it is simply that student is to mutually commenting System submits operation to, and teacher, to mutually commenting system to provide standards of grading, mutually comments system to distribute other students' to each student at random Operation is scored, and each student receives the operation of many parts of other students at random, and appraisal result is fed back to and mutually comments by scoring student System, mutually comments system to student data and the appraisal result that receives is processed.

Job eveluation is considered as to wrong Differential Diagnosiss process by the present invention, and estimator to the point scoring in submission operation is No correctly carry out Differential Diagnosiss, Key for Reference and standards of grading that diagnosis basis are issued for teacher.

The present invention mutually comments being implemented as of system:

After End-of-Course, mutually comment system synthesis student to multiple job eveluation result, process student by following items and submit to Data:

1) the th subjob that student txh submits to is made up of several point scorings, and dfh is the numbering of point scoring, this score The score value weight of point is m (th, dfh), and student pxh is scored to point scoring dfh as scoring student, df (pxh, txh, Th, dfh) appraisal result for student pxh, 0 represents not score, 1 expression score, and -1 represents and do not score, such as score then score value by m (th, dfh) is calculated；

2) evaluating ability of calculating student pxh:

Diagnostic sensitivity is defined to scoring student as follows:

In the All Jobs sample that student pxh evaluates, comprise n mistake, that is, have n point scoring evaluation of teacher must not be Point, student pxh is evaluated as not score to i point scoring in described n point scoring, then the diagnostic sensitivity of student pxh is fixed Justice is:

$se=\frac{i}{n}$

Specificity is defined to scoring student as follows:

In the All Jobs sample that student pxh evaluates, m point scoring evaluation of teacher is had to be score, student pxh is to described m The scoring of j point scoring in individual point scoring is score, then the Student Diagnosis specificity for pxh for the student number is defined as

$sp=\frac{j}{m}$

3) the operation score of calculating student txh:

3.1) for the point scoring that evaluation is consistent:

Submit in each point scoring of operation in student txh, the point scoring that all scoring students are unanimously chosen as score determines For score, the point scoring being unanimously chosen as not score is defined as not score；

3.2) error generation rate of estimation students' work:

Student's txh submission is obtained evaluating consistent point scoring, the number being wherein chosen as score is t₁, must not be chosen as The number dividing is t₂, then the error generation rate of student txh be:

$er=\frac{t_2}{t_1+t_2}$

3.3) for evaluating inconsistent point scoring, the operation not score that nonuniformity is evaluated is calculated according to Bayesian model Probability:

If a total of n scoring student independently scores to the same point scoring of student txh, corresponding diagnostic sensitivity It is respectively se with specificity₁,se₂,...,se_n,sp₁,sp₂,...,sp_nIf wherein front k Students ' Evaluation is divided into 0, rear n-k Individual Students ' Evaluation is divided into 1, if the corresponding event of scoring is a₁,a₂,...,a_kAnda₁,a₂,...,a_kRepresent k Raw point scoring is scored for 0,After expression, n-k student scores as 1 to point scoring, operation submitter txh's Error rate is er,

Remember branch not score event be b, according to condition probability formula, not scoring probability For:

$\begin{array}{c}p(b/a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})=\frac{p({\mathrm{ba}}_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})}{p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})}\\ =\frac{p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/b)p\left(b\right)}{p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})}\end{array}$

Score between due to each student separate, obtain:

$p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/b)=\underset{p=1}{\overset{k}{\&pi;}}p(a_p/b)\underset{q=k+1}{\overset{n}{\&pi;}}p(\stackrel{\&overbar;}{a_q}/b)=\underset{p=1}{\overset{k}{\&pi;}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\&pi;}}(1-{\mathrm{se}}_q)$

According to total probability formula, have:

$\begin{array}{c}p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})\\ =p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/b)p\left(b\right)+p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/\stackrel{\&overbar;}{b})p\left(\stackrel{\&overbar;}{b}\right)\\ =\underset{p=1}{\overset{k}{\&pi;}}p(a_p/b)\underset{q=k+1}{\overset{n}{\&pi;}}p(\stackrel{\&overbar;}{a_q}/b)p\left(b\right)+\underset{p=1}{\overset{k}{\&pi;}}p(a_p/\stackrel{\&overbar;}{b})\underset{q=k+1}{\overset{n}{\&pi;}}p(\stackrel{\&overbar;}{a_q}/\stackrel{\&overbar;}{b})p\left(\stackrel{\&overbar;}{b}\right)\\ =\underset{p=1}{\overset{k}{\&pi;}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\&pi;}}(1-{\mathrm{se}}_q)er+\underset{p=1}{\overset{k}{\&pi;}}(1-{\mathrm{sp}}_p)\underset{q=k+1}{\overset{n}{\&pi;}}{\mathrm{sp}}_p(1-er)\end{array}$

Therefore:

$p(b/a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n})=\frac{\underset{p=1}{\overset{k}{\&pi;}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\&pi;}}(1-{\mathrm{se}}_q)er}{\underset{p=1}{\overset{k}{\&pi;}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\&pi;}}(1-{\mathrm{se}}_q)er+\underset{p=1}{\overset{k}{\&pi;}}(1-{\mathrm{sp}}_p)\underset{q=k+1}{\overset{n}{\&pi;}}{\mathrm{sp}}_q(1-er)}$

3.4) score according to the inconsistent point scoring of determine the probability evaluation:

Given threshold α, ifThen this point scoring not score, if Then this point scoring score, ifThen mutually comment system commenting this point scoring Valency task is sent to teacher, and α corresponds to teacher and processes intensity, and α is more high, and the probability being sent to teacher's process is higher.As preferred Mode, to threshold alpha, arranges 80% < α < 100%.

In step 3.2) mistake in computation incidence rate when, student txh obtain evaluating consistent score count out fewer, then rear The Data processing in face, mutually comments the probability that the evaluation task evaluating inconsistent point scoring is sent to teacher by system higher.

Further, mutually comment system according to the scoring evaluation result of student and the final appraisal result meter being evaluated operation Calculate the scoring effective percentage of scoring student, marking system submits the final appraisal result of operation and commenting of this student to each student Divide effective percentage to be respectively provided with weight, be calculated the operation scores of each student by this two parts.Scoring effective percentage is permissible Form the constraint to scoring student, under the method according to the invention, student passes through mutually to comment system to realize interactional closed loop relation, Evaluation result done by one student is also subject to efficient evaluation of scoring simultaneously, on the one hand on original job eveluation basis On, also achieve the evaluation to student assessment ability, on the other hand also function to constrain the effect of the randomness of student assessment.

Below the present invention is illustrated with embodiment, exercise question numbering represents the object being evaluated, and point scoring is split for exercise question Least unit.

Embodiment:

120 students, halves everyone submitted 15 subjobs, totally 80 problem, containing 360 point scorings.

Teacher has simultaneously provided 30 parts of operation samples comprising typical fault, and every subjob correspondence has two parts of operation samples.

Every student scores to the 4 parts of operations distributed every time, wherein has a two parts of works providing from teacher Randomly draw in industry sample.

Mutually comment system to the operation appraisal result of each student by table 1 below record data:

Table 1

Operation sample appraisal result is by table 2 below record:

pxh	Scoring person's student number
		tbh	Job number
dfh	Point scoring is numbered
		pf	Scoring
jpf	Teacher scores

Calculation procedure:

1. all records in pair table 1

If pf1+pf2+pf3=0, zhf=0, dfgl=1

If pf1*pf2*pf3=1, zhf=1, dfgl=1

2. the error rate of each student of calculating:

To each student number xh, mistake in computation rate er (xh).

Note t0 is the number of the record meeting txh=xh and zhf ≠ -1 in table 1.T1 is txh=xh and zhf=0 in table 1 Record number.Then er (xh)=t1/t0.

3. the Sensitivity and Specificity of the Differential Diagnosiss of each student is calculated according to table 2

To each student number xh, calculate sensitivity se (xh) and the specificity sp (xh) of diagnosis.

Note t2 is the number of the record meeting pxh=xh and jpf=0 in table 2, and t3 is to meet pxh=xh and jpf in table 2 The number of=1 record, t4 is the number of the record meeting pxh=xh and jpf=0 and pf=0 in table 2, and t5 is to meet in table 2 The number of the record of pxh=xh and jpf=1and pf=1.

Then se (xh)=t4/t2,

Sp (xh)=t5/t3.

4. update the comprehensive score of zhf=-1 record in table 1

Calculated example:

1. student number 04, error rate is 10%, and the point scoring of submission is evaluated by student number 05,06,07.The discriminating of above-mentioned 3 people is examined Disconnected sensitivity is respectively 95%, 96%, 40%, and specificity is 96%, 98%, 92%.Three people's appraisal result are 0,0,1.(probability Threshold value takes α=0.9)

According to formula, this point scoring is that 0 probability is:

$p=\frac{0.95\&times;0.96\&times;(1-0.4)\&times;0.1}{0.95\&times;0.96\&times;(1-0.4)\&times;0.1+0.04\&times;0.02\&times;0.92\&times;0.9}=99.9\%$

Zhf=0, dfgl=99.9%.

2. student number 09, error rate is 20%, and the point scoring of submission is evaluated by student number 12,13,14.The discriminating of above-mentioned 3 people is examined Disconnected sensitivity is respectively 90%, 89%, 92%, and specificity is 93%, 94%, 95%.Three people's appraisal result are 0,1,1.(probability Threshold value takes α=0.9)

According to formula, this point scoring is that 0 probability is:

$p=\frac{0.9\&times;0.11\&times;0.08\&times;0.2}{0.9\&times;0.11\&times;0.08\&times;0.2+0.07\&times;0.94\&times;0.95\&times;0.8}=3\%$

Zhf=1, dfgl=97%.

3. student number 12, evaluate other people point scoring sum 360 × 4=1440, wherein 1200 evaluations are consistent with zhf, thenThis index reacts the evaluating ability of this student.

As the contrast of reliability of the present invention, on the premise of on the basis of evaluation of teacher, it is based on bayes model for evaluating Mutually comment efficiency, based on the 6th subjob in above-described embodiment, carried according to student by the system of mutually commenting that the present invention realizes The data handed over obtains appraisal result, and is checked by teacher's 120 parts of operations whole to the 6th time, and makees to match rank test.Knot Fruit according to the form below arranges:

Student number	Teacher appraises	Ranking	Mutually comment score	Mutually comment ranking	Difference	Rank
							001	86	23	90	23	0	4.5
002	79	96	82	90	-6	-65
							003	96	10	95	12	2	12.5
004	94	11	96	9	-2	-12.5

Wherein difference represents and realizes representing the student's ranking after mutually commenting by mutually commenting system, with the student's name after teacher's scoring Difference between secondary.

By the absolute value sequence of the difference of all students, residing sequence number and difference symbol form rank.Equal the taking of sequence number Meansigma methodss.If student number 003 difference is 2, sequence 12, student number 004 difference is -2, and sequence 13 takes average (12+13)/2=12.5, In conjunction with difference symbol, the two rank is respectively 12.5, -12.5.

It is that positive rank is added all ranks, obtain t⁺=3782, all ranks are negative addition, obtain t^-=3478.According to Equation below calculates z, and z is the statistic obeying standard normal distribution, c=120:

$z=\frac{t^{+}-c(c+1)/4}{\sqrt{c(c+1)(2c+1)/24}}$

Look into standard normal distribution table and obtain corresponding probit p=0.65, be non-small probability event during 5%≤p≤95%, because This, illustrate teacher appraise with the present invention mutually comment the system to process the score no significant difference that obtains it was demonstrated that the reliability of the present invention Property.

Claims (3)

1. a kind of implementation method mutually commenting system, in a course, teacher arranges many subjobs to student, it is characterized in that mutually commenting and is System receives the All Jobs that student submits to, and the Key for Reference of each subjob of correspondence, standards of grading and the work that teacher submits to Industry sample, described operation sample by teacher by the job design arranged the operation comprising many places mistake；For making each time Industry, mutually comments system that corresponding Key for Reference, standards of grading and operation sample issued each student, and every part of student is submitted to Operation is given more than 2 students at random and is evaluated, and evaluation result is fed back to system of mutually commenting by student；

After End-of-Course, mutually comment system synthesis student to multiple job eveluation result, process, by following items, the number that student submits to According to:

1) the th subjob that student txh submits to is made up of several point scorings, and dfh is the numbering of point scoring, this point scoring Score value weight is m (th, dfh), and student pxh is scored to point scoring dfh as scoring student, df (pxh, txh, th, Dfh) be student pxh appraisal result, 0 represents not score, 1 expression score, and -1 represents and do not score, such as score then score value by m (th, dfh) is calculated；

2) evaluating ability of calculating student pxh:

Diagnostic sensitivity is defined to scoring student as follows:

In the All Jobs sample that student pxh evaluates, comprise n mistake, that is, have n point scoring evaluation of teacher for not score, learn Raw pxh is evaluated as not score to i point scoring in described n point scoring, then the diagnostic sensitivity of student pxh is defined as:

$se=\frac{i}{n}$

Specificity is defined to scoring student as follows:

In the All Jobs sample that student pxh evaluates, m point scoring evaluation of teacher is had to be score, student pxh obtains to described m The scoring of j point scoring in branch is score, then the Student Diagnosis specificity for pxh for the student number is defined as

$sp=\frac{j}{m}$

3) the operation score of calculating student txh:

3.1) for the point scoring that evaluation is consistent:

Submit in each point scoring of operation in student txh, the point scoring that all scoring students are unanimously chosen as score is defined as Point, the point scoring being unanimously chosen as not score is defined as not score；

3.2) error generation rate of estimation students' work:

Student's txh submission is obtained evaluating consistent point scoring, the number being wherein chosen as score is t₁, it is chosen as not score Number is t₂, then the error generation rate of student txh be:

$er=\frac{t_2}{t_1+t_2}$

3.3) for evaluating inconsistent point scoring, according to the operation that Bayesian model calculating nonuniformity is evaluated, score is not general Rate:

If the independent same point scoring to student txh of a total of n scoring student scores, corresponding diagnostic sensitivity and examining Disconnected specificity is respectively se₁,se₂,...,se_n,sp₁,sp₂,...,sp_nIf wherein front k Students ' Evaluation is divided into 0, rear n-k Raw scoring is 1, if the corresponding event of scoring is a₁,a₂,...,a_kAnda₁,a₂,...,a_kRepresent k student couple Point scoring scores as 0,After expression, n-k student scores as 1 to point scoring, the mistake of operation submitter txh Rate is er,

Remember branch not score event be b, according to condition probability formula, not scoring probability For:

$\begin{array}{c}p\left(b/a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}\right)=\frac{p\left({\mathrm{ba}}_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}\right)}{p\left(a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}\right)}\\ =\frac{p\left(a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}/b\right)p\left(b\right)}{p\left(a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}\right)}\end{array}$

Score between due to each student separate, obtain:

$p(a_1{a}_2...a_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}...,\stackrel{\&overbar;}{a_n}/b)=\underset{p=1}{\overset{k}{\&pi;}}p(a_p/b)\underset{q=k+1}{\overset{n}{\&pi;}}p(\stackrel{\&overbar;}{a_q}/b)=\underset{p=1}{\overset{k}{\&pi;}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\&pi;}}(1-{\mathrm{se}}_q)$

According to total probability formula, have:

$\begin{array}{c}p\left(a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}\right)\\ =p\left(a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}/b\right)p\left(b\right)+p\left(a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}/\stackrel{\&overbar;}{b}\right)p\left(\stackrel{\&overbar;}{b}\right)\\ =\underset{p=1}{\overset{k}{\mathrm{\&pi;}}}p\left(a_p/b\right)\underset{q=k+1}{\overset{n}{\mathrm{\&pi;}}}p\left(\stackrel{\&overbar;}{a_q}/b\right)p\left(b\right)+\underset{p=1}{\overset{k}{\mathrm{\&pi;}}}p\left(a_p/\stackrel{\&overbar;}{b}\right)\underset{q=k+1}{\overset{n}{\mathrm{\&pi;}}}p\left(\stackrel{\&overbar;}{a_q}/\stackrel{\&overbar;}{b}\right)p\left(\stackrel{\&overbar;}{b}\right)\\ =\underset{p=1}{\overset{k}{\mathrm{\&pi;}}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\mathrm{\&pi;}}}\left(1-{\mathrm{se}}_q\right)er+\underset{p=1}{\overset{k}{\mathrm{\&pi;}}}\left(1-{\mathrm{sp}}_p\right)\underset{q=k+1}{\overset{n}{\mathrm{\&pi;}}}{\mathrm{sp}}_q\left(1-er\right)\end{array}$

Therefore:

$p\left(b/a_1{a}_2\mathrm{...}{a}_k\stackrel{\&overbar;}{a_{k+1}},\stackrel{\&overbar;}{a_{k+2}}\mathrm{...},\stackrel{\&overbar;}{a_n}\right)=\frac{\underset{p=1}{\overset{k}{\&pi;}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\&pi;}}(1-{\mathrm{se}}_q)er}{\underset{p=1}{\overset{k}{\&pi;}}{\mathrm{se}}_p\underset{q=k+1}{\overset{n}{\&pi;}}(1-{\mathrm{se}}_q)er+\underset{p=1}{\overset{k}{\&pi;}}(1-{\mathrm{sp}}_p)\underset{q=k+1}{\overset{n}{\&pi;}}{\mathrm{sp}}_q(1-er)}$

3.4) score according to the inconsistent point scoring of determine the probability evaluation:

Given threshold α, ifThen this point scoring not score, ifThen This point scoring score, ifThen mutually comment system by the evaluation of this point scoring Task is sent to teacher, and α corresponds to teacher and processes intensity, and α is more high, and the probability being sent to teacher's process is higher.

2. a kind of implementation method mutually commenting system according to claim 1, is characterized in that to threshold alpha, arrange 80% < α < 100%.

3. a kind of implementation method mutually commenting system according to claim 1, is characterized in that mutually commenting system according to scoring student Evaluation result and the final appraisal result being evaluated operation calculate the scoring effective percentage of scoring student, marking system is to each Student submits to the final appraisal result of operation and the scoring effective percentage of this student to be respectively provided with weight, is calculated by this two parts Operation scores to each student.