JP2005196281A - Already evaluated examination paper mixing method, already evaluated examination paper mixing device and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an already evaluated examination paper mixing method, an already evaluated examination paper mixing device and a computer program, mixing an examination paper previously decided with evaluation into a plurality of examination papers to be graded by a grader so as to evaluate the grader of a description type test. <P>SOLUTION: This already evaluated examination paper mixing device 1 stores the already evaluated examination paper decided with the evaluation of an correct answer or an erroneous answer about a plurality of questions having possibility for use as a an test question into an already evaluated examination paper DB 14a, extracts the already evaluated examination paper about each the question included in an actual test questions, and stores it into a question-classified already evaluated examination paper DB 14e. When outputting a plurality of non-evaluated examination papers answered by an examinee and not graded to an grading terminal 3 so as to make the grader grade the plurality of non-evaluated examination papers, the already evaluated examination paper mixing device 1 mixes the already evaluated examination paper selected from the question-classified already evaluated examination paper DB 14e into order of the plurality of non-evaluated examination papers and outputs it. Because the grader simultaneously grades the already evaluated examination paper too, grading performance of the grader can be evaluated. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for evaluating a scorer of a descriptive test, and relates to an already-evaluated answer mixing method in which an answer whose evaluation is determined in advance is mixed among a plurality of answers to be scored by the scorer, The present invention relates to an apparatus and a computer program.

  In a descriptive test, the test answer must be scored manually. In an examination such as a qualification examination that is taken by many examinees, an enormous amount of answers are scored within a predetermined period of time, so it is necessary to efficiently score a large number of graders. In Patent Document 1, a technique in which a plurality of answers for one question among a plurality of questions that are presented is displayed on a display device of a computer, and a grader scores each of the plurality of answers for the displayed one question. Is disclosed. By using this technique, the scorer only needs to repeatedly score the same question, so the scoring operation is simplified and the score can be efficiently scored.

In addition, accurate scoring is required for scoring test answers. Therefore, the technique disclosed in Patent Document 1 is an answer (hereinafter referred to as “existing answer”) in which an evaluation such as correctness or inferiority is determined in advance among a plurality of answers (hereinafter referred to as “unevaluated answers”) to be scored by the scorer. It is equipped with a function for obtaining a degree of coincidence between an evaluation made by a grader with respect to an already-evaluated answer and a predetermined evaluation. By using this technique, it is possible to objectively determine the scoring ability of the scorer, so that it is possible to improve the ability of the scorer using the determination result as an index. This makes it possible to score test answers more accurately.
JP 2000-122517 A

  In order to objectively determine the ability of the grader using the technique disclosed in Patent Document 1, it is necessary to prepare a plurality of unassessed answers in which already evaluated answers are appropriately mixed. However, Patent Document 1 does not disclose a method of mixing an already evaluated answer among a plurality of unassessed answers.

  The present invention has been made in view of such circumstances, and the purpose of the present invention is to previously accumulate a plurality of already-assessed answers for each question, and for each question included in the test question. It is possible to realize the evaluation of the scoring ability of the grader by extracting a plurality of already-evaluated answers, and interrupting and displaying the extracted already-evaluated answers on a plurality of unassessed answers displayed on the display device. An evaluation answer mixing method, an already evaluated answer mixing apparatus, and a computer program for realizing a computer as an already evaluated answer mixing apparatus.

  In the already-evaluated answer mixing method according to the first invention, evaluation is already determined in a plurality of unassessed answers to each of a plurality of questions included in a test question using a computer including a storage unit, a calculation unit, and an output unit. A plurality of already-assessed answers that are associated with each of a plurality of questions, stored in the storage unit, and associated with each of the plurality of questions included in the test question. An arbitrary number of already evaluated answers extracted from the plurality of already evaluated answers stored in the storage means is extracted by the calculation unit, and the test question includes the arbitrary number of already evaluated answers extracted by the calculation unit. A plurality of questions are stored in the storage unit in association with each of the questions, the plurality of unassessed answers to one question among a plurality of questions included in the test question, and stored in the storage unit in association with the one question. Said A predetermined number of previously evaluated answers retrieved from the number of previously evaluated answers and outputs in the output section.

  In the already-evaluated answer mixing method according to the second invention, a computer including a storage unit and a calculation unit is used to evaluate a plurality of unassessed answers for each of a plurality of questions included in a test question in a predetermined order. Is a method of mixing already-assessed answers that have already been determined, each of which is associated with each of a plurality of questions, a plurality of already-assessed answers to the question are stored in the storage unit, and each of a plurality of questions included in the test question An arbitrary number of already-evaluated answers are extracted from the plurality of already-evaluated answers stored in the storage means in association with the computing unit, and the arbitrary number of already-evaluated answers extracted by the computing unit are extracted from the test. The calculation unit stores the predetermined number of mixed positions for storing a predetermined number of already evaluated answers in the order of the plurality of unassessed answers in association with each of the plurality of questions included in the problem. Determined by and before Storing in the storage unit order specifying information for specifying the order of each of the plurality of unevaluated answers and the predetermined number of already evaluated answers, including mixing position specifying information for specifying each mixing position determined by the calculation unit. It is characterized by.

  In the already-evaluated answer mixing method according to the third invention, the computer further includes an output unit, wherein one of the plurality of questions included in the test question is selected by the calculation unit and selected by the calculation unit. Each of the predetermined number of mixed positions designated by the mixed position designation information included in the order designation information is stored in the storage unit in association with the one question. Each of the already-evaluated answers to be mixed is selected by the calculation unit, and the plurality of unassessed answers for the one question and the predetermined number of already-evaluated answers selected by the calculation unit are specified by the order designation information According to the order of output, the output unit sequentially outputs.

  In the already-assessed answer mixing method according to a fourth aspect of the invention, the computer further includes a receiving unit that receives the identification information of the grader, and stores the identification information of the grader in the storage unit in association with the rank of the grader. The degree of difficulty of the already evaluated answer is stored in the storage unit in association with each of the already evaluated answers, and when the predetermined number of already evaluated answers are selected by the computing unit, the identification information received by the receiving unit An already-evaluated answer associated with the degree of difficulty corresponding to the rank associated with is selected.

  An already-evaluated answer mixing apparatus according to the fifth invention is an apparatus that mixes an already-evaluated answer whose evaluation has already been determined in a predetermined order of a plurality of unassessed answers for each of a plurality of questions included in a test question. The first storage means stores a plurality of already evaluated answers to the question in association with each of the plurality of questions, and the first storage means stores in association with each of the plurality of questions included in the test question. Means for extracting an arbitrary number of already evaluated answers from the plurality of already evaluated answers, and storing the arbitrary number of already evaluated answers extracted by the means in association with each of the plurality of questions included in the test question. Second storage means, a means for determining the predetermined number of mixing positions for mixing a predetermined number of already evaluated answers in the order of the plurality of unassessed answers, and each mixing position determined by the means Specification of mixing position to be specified Including broadcast, characterized in that it comprises a means for storing the order designation information for designating the order of each of the plurality of unevaluated answer and previously evaluated answer of the predetermined number.

  An already-assessed answer mixing apparatus according to a sixth aspect of the invention is characterized in that: means for selecting one question from a plurality of questions included in the test question; and the second storage means in association with the one question selected by the means. From the stored arbitrary number of already evaluated answers, the already evaluated answers to be mixed in each of the predetermined number of mixing positions specified by the mixing position specifying information included in the order specifying information are selected. Output that sequentially outputs each of the plurality of unassessed answers to the one question and the predetermined number of already-evaluated answers selected by the already-evaluated answer selecting means according to the order specified by the order specifying information. And a means.

  The already-assessed answer mixing apparatus according to a seventh aspect of the present invention further comprises identification information storage means for storing the identification information of the grader in association with the rank of the grader, and identification information reception means for accepting the identification information, The storage means is adapted to store the degree of difficulty of the already evaluated answer in association with each of the already evaluated answers, and the already evaluated answer selecting means is associated with the identification information received by the identification information receiving means. It is characterized in that an already-evaluated answer associated with the degree of difficulty corresponding to the rank is selected.

  A computer program according to the eighth invention relates to each of a plurality of questions included in a test question, and stores a plurality of already-evaluated answers whose evaluations have already been determined for the question. A computer program that mixes an already-evaluated answer in a predetermined order of unassessed answers, and among the plurality of already-evaluated answers that are associated with each of a plurality of questions included in the test question on the computer A procedure for extracting an arbitrary number of already-evaluated answers from the computer, a procedure for storing the arbitrary number of already-evaluated answers in association with each of a plurality of questions included in the test question, and a computer having a predetermined number of already-evaluated answers The procedure for determining the predetermined number of mixing positions for mixing the answers in the order of a plurality of unevaluated answers, and the computer Comprising mixing location specifying information to specify the location, characterized in that it comprises a procedure for storing the order designation information designating a plurality of unevaluated answer and order of each of the previously evaluated answer of the predetermined number.

  A computer program according to a ninth invention is a program for causing a computer to select one question from a plurality of questions included in the test question, and the arbitrary program stored in the computer in association with the selected one question. A step of selecting each of the already evaluated answers to be mixed at each of the predetermined number of mixing positions designated by the mixing position designation information included in the order designation information from among a plurality of already evaluated answers, And a step of outputting each of the plurality of unevaluated answers to the question and the selected predetermined number of already evaluated answers in order according to the order designated by the order designation information.

  The computer program according to the tenth invention stores the identification information of the grader in association with the rank of the grader and stores the identification information received in a computer that stores the degree of difficulty of the already-evaluated answer in association with each of the already-evaluated answers The method further includes a step of selecting an already-evaluated answer associated with the degree of difficulty corresponding to the rank associated with the item as an already-evaluated answer to be output.

  In the first invention, in order to mix an already evaluated answer whose evaluation has already been determined among a plurality of unassessed answers of test questions that have not yet been scored, individual questions that may be used as test questions Memorize multiple evaluated answers for a question, extract and store multiple evaluated answers for each question included in a test question, and output an unassessed answer and an already evaluated answer for one question . When outputting the already evaluated answers, a predetermined number of already evaluated answers are extracted from a plurality of already evaluated answers stored in association with the question and output.

  In the second, fifth and eighth inventions, used as a test question in order to mix an already-evaluated answer whose evaluation has already been determined among a plurality of un-evaluated answers to a test question that has not yet been scored. Memorize multiple evaluated questions for each possible question, extract and store multiple evaluated questions for each question included in the exam questions, and pre-determine them in the order of the exam number, etc. Determine a mixing position for mixing a predetermined number of previously evaluated answers in the order of the unassessed answers, and store order designation information for specifying a plurality of unassessed answers and a predetermined number of previously evaluated answers .

  In the third, sixth and ninth inventions, a question to which an answer should be output for scoring is selected, and an unexamined answer is extracted from a plurality of already evaluated answers stored in association with the question. A predetermined number of already-evaluated answers to be mixed into the evaluation answer are selected, and an unevaluated answer or an already-evaluated answer for one question is sequentially output according to the order specified by the order designation information.

  In the fourth, seventh and tenth inventions, the rank indicating the ability of the scorer and the difficulty indicating the difficulty of legitimately evaluating the already evaluated answer are stored, and a predetermined number of already evaluated answers to be output. Is selected, an already evaluated answer with a difficulty corresponding to the rank of the grader is selected.

  In the 1st, 2nd, 5th and 8th inventions, memorize already evaluated answers that have been evaluated in advance for a plurality of questions that may be used as test questions. When a grader scores an unevaluated answer by extracting an already assessed answer for each included question and mixing it in the order of multiple unassessed answers that the test taker has answered and not yet graded The graded ability can be scored at the same time, and the grader's ability is objectively determined by comparing the graded score against the graded graded answer with a predetermined grade. be able to. Moreover, since the ability of the grader can be determined objectively, it is possible to improve the ability of the grader by using the determination result as an index, and the test answer can be scored more accurately.

  In the third, sixth and ninth inventions, it is not fixed which evaluated answer is mixed in the unassessed answer, and the position where the already evaluated answer is mixed in the unassessed answer is determined. In addition, when a grader scores an answer, it is easy to select an already-evaluated answer to be mixed according to the type of question or the grader by interrupting the already-evaluated answer at the mixing position and scoring. Can accurately determine the ability of the grader.

  In the fourth, seventh and tenth inventions, when each grader is assigned a rank, the degree of scoring is determined for each already evaluated answer, and the already evaluated answer is mixed with the already evaluated answer. By mixing the graded evaluation questions according to the grader's rank, the high-ranking graders who were previously determined to have high scoring ability scored the high-grade graded answers. It is possible to mark a low-ranked grader with a graded assessment with a low difficulty level, such as scoring an appropriate graded answer according to the grader's ability. The present invention has an excellent effect that it can be determined in detail.

Hereinafter, the present invention will be specifically described with reference to the drawings showing embodiments thereof.
The present invention is a technique for assisting a scorer in scoring an answer in a descriptive test and simultaneously determining the scoring ability of the scorer. Specifically, the grader's scoring ability is determined by mixing the already-evaluated answer in the unassessed answer to be scored by the grader and letting the grader score. In the following description, the present invention will be described by taking as an example the case where the present invention is applied to scoring a kanji test such as “Japanese Kanji Proficiency Test”.

  FIG. 1 is a block diagram showing an internal configuration of an already evaluated answer mixing apparatus 1 according to the present invention. The already evaluated answer mixing apparatus 1 is configured using a general-purpose computer such as a personal computer or a server apparatus. The already-assessed answer mixing device 1 includes a CPU (calculation unit) 11 that performs calculation, a RAM 12 that stores temporary information generated by the calculation, an external storage device 13 such as a CD-ROM drive, and a hard disk. And an internal storage device (storage unit) 14. The CPU 11 reads the computer program 20 of the present invention from the recording medium 2 such as a CD-ROM by the external storage device 13 and stores the read computer program 20 in the internal storage device 14. The computer program 20 is loaded from the internal storage device 14 to the RAM 12 as necessary. Based on the loaded computer program 20, the CPU 11 executes processing necessary for the already evaluated answer mixing device 1. The already-assessed answer mixing device 1 includes an input device 15 such as a keyboard, a mouse, or a tablet, and an output device 16 such as a liquid crystal display or a CRT display, and receives operations from an operator such as data input. It has become.

  In addition, an interface unit 17 is connected to the CPU 11. The interface unit 17 is connected to a scoring terminal 3 having an input unit such as a keyboard, a mouse, or a tablet, and an output unit such as a liquid crystal display or a CRT display. The scoring terminal 3 is configured to output an answer to be scored by the grader to the output unit, and to accept an input of scoring for the answer by the grader operating the input unit. The scoring terminal 3 functions as an output unit and output means according to the present invention.

  The internal storage device 14 includes an already-evaluated answer DB (first storage means) 14a that stores an already-evaluated answer for each question that may be given as a test question. FIG. 2 is a conceptual diagram showing an example of the contents stored in the already evaluated answer DB 14a. The image data obtained by electronically capturing handwritten characters is stored in association with the problem information indicating what the kanji problem is and the problem category information indicating what the kanji is being asked about. Yes. Individual questions are defined by problem information and problem category information. In the example in the figure, four image data are stored in the already-assessed answer DB 14a in association with the question “Kanji” in which the problem category is “Kanji” for writing this Kanji in the problem of the Chinese character “NA”. In addition, in relation to each image data, information on correct / incorrect category indicating whether this image data is a correct answer or an incorrect answer as a question answer, and the difficulty level when scoring this image data as an answer And information on the degree of difficulty indicating. In the example in the figure, the question with the question “NA” and the question category “Kanji” contains one correct answer image data and three incorrect answer image data. The difficulty level indicating the difficulty is stored in association with each other. The difficulty level indicates that the smaller the difficulty level, the more difficult it is to accurately score the already evaluated answer. Similarly, image data in which information about the correct / wrong category and the difficulty level is associated with each question defined by the information about the question and the problem category is stored in the already evaluated answer DB 14a.

  The internal storage device 14 stores test question data 14b indicating the content of the test question to be scored. FIG. 3 is a conceptual diagram showing an example of the contents of the test question data 14b. In the exam question data 14b, information indicating what grade and what grade the exam is in which year is recorded. And subquestion No. The information of the problem and the problem category is recorded in association with. In this way, a plurality of questions included in the test questions are specified.

  Further, the internal storage device 14 stores extraction condition data 14c indicating conditions for extracting from the already-evaluated answer DB 14a an already-evaluated answer to be mixed into an unassessed answer to the test question. FIG. 4 is a conceptual diagram showing an example of the contents of the extraction condition data 14c. In the extraction condition data 14c, the number of correct answers extracted and the number of incorrect answers extracted are recorded in association with each question defined by the same problems and problem categories as those recorded in the test question data 14b. In addition, the extraction condition data 14c records information on the homonym word classification indicating whether or not the image data of the homonym is extracted as an already evaluated answer, with YES or NO. Furthermore, the number of extracted homonyms is recorded for the question whose homonym category is YES. Note that the test question data 14b and the extraction condition data 14c may be data combined into one instead of separate.

  Further, the internal storage device 14 stores homonym data 14d indicating a problem having a homonym. FIG. 5 is a conceptual diagram showing an example of the contents of the homonym data 14d. In the homonym data 14d, problem information and kanji reading information in question are recorded in association with each other, and a plurality of problems that are synonyms with each other are associated with each other in the same reading. Are recorded. Further, the internal storage device 14 is provided with a question-by-question already-evaluated answer DB (second storage means) 14e for storing already-evaluated answers to individual questions included in the test question data 14b.

  The computer program 20 extracts an already-evaluated answer for the test question from the already-evaluated answer DB 14a and records it in the already-evaluated answer DB 14e for each question, and mixes the already-evaluated answer into the unassessed answer to the test question. A program for performing a mixing process for determining a position to be performed, a scoring process for allowing a grader to score an unevaluated answer mixed with an already-evaluated answer, and an evaluation process for evaluating the scorer are included.

  FIG. 6 is a flowchart showing a procedure of an already evaluated answer extraction process performed by the already evaluated answer mixing apparatus 1 of the present invention. The CPU 11 executes the following processing according to the computer program 20 loaded into the RAM 12. The CPU 11 reads the test question data 14b and the extraction condition data 14c stored in the internal storage device 14 into the RAM 12 (S101). Next, the CPU 11 selects one question from a plurality of questions recorded in the test question data 14b (S102), and has already evaluated the question based on the information of the question and problem category of the selected question. An answer is searched from the already-evaluated answer DB 14a, and the number of correct answers according to the information on the number of correct answers recorded in the extraction condition data 14c for this question and the number of errors according to the information on the number of incorrect answers extracted. The already evaluated answer answers are extracted from the already evaluated answer DB 14a (S103). The already evaluated answers extracted at this time include image data, correct answer classification information, and difficulty information. In this case, the same already evaluated answer may be extracted in duplicate so as to follow the information on the number of correct answers and the number of incorrect answers extracted.

  Next, the CPU 11 determines whether or not the homonym segment information recorded in the extraction condition data 14c for the selected question is YES (S104). If the information of the homonym classification is YES (S104: YES), the CPU 11 searches the homonym data 14d for the same reading problem as the selected question, and finds the information and selection of the searched problem. The number of already-evaluated answers having the same question category information as the question being answered is extracted from the already-evaluated answer DB 14a according to the number of homophones extracted in the extraction condition data 14c (S105). The already-evaluated answer of the homonym extracted at this time is an incorrect answer to the question even if it is an already-evaluated answer whose correct / incorrect information is recorded in the already-evaluated answer DB 14a. Is stored in the RAM 12 as an incorrect answer.

  When the information on the homonym classification is NO in step S104 (S104: NO) or when step S105 is completed, the CPU 11 has already evaluated the answers for all questions recorded in the test question data 14b. Is determined (S106). If there is a question for which an already evaluated answer has not yet been extracted (S106: NO), the CPU 11 returns the process to step S102, and selects the next question for which an already evaluated answer has not yet been extracted. When the already evaluated answers are extracted for all questions (S106: YES), the CPU 11 outputs a list of questions from which the already evaluated answers are extracted to the output device 16 (S107).

  FIG. 7 is a schematic diagram showing a screen example of a list of questions extracted from the already evaluated answers that the already evaluated answer mixing apparatus 1 outputs to the output device 16. It can be seen that a plurality of already-evaluated answers are extracted for each question similar to that recorded in the test question data 14b. The operator can check a list of already evaluated answers on the screen of the output device 16 and can operate the input device 15 to select any question.

  The CPU 11 monitors the reception of selection of any question by the operator operating the input device 15 (S108). If the selection of the question is received (S108: YES), the CPU 11 has already extracted the selected question. The evaluation answer is output to the output device 16 (S109).

  FIG. 8 is a schematic diagram showing a screen example of an already evaluated answer for one question output from the already evaluated answer mixing device 1 to the output device 16. In the figure, the already-evaluated answers extracted for the question whose problem is “Meet” and whose problem category is “Kanji” is shown, and image data of a plurality of extracted already-evaluated answers is output. The operator confirms each image data on the screen of the output device 16 and operates the input device 15 to change an already evaluated answer such as deletion of image data, correction of image data, or input of new image data. Can be input.

  The CPU 11 monitors reception of an instruction to change the already evaluated answer by the operator operating the input device 15 (S110), and if no change instruction is received (S110: NO), the process returns to step S107. . When the instruction to change the already evaluated answer is received (S110: YES), the CPU 11 receives the change of the image data of the already evaluated answer by the operator operating the input device 15 (S111). At this time, if the change instruction is deletion, the CPU 11 deletes the already evaluated answer that is specified to be deleted. If the change instruction is correction of the image data, the CPU 11 uses the input device 15 such as a tablet. Accept correction of image data on the screen used. When the change instruction is input of new image data, the CPU 11 accepts input of new image data on the screen using the input device 15 such as a tablet, and further, correct / error classification, difficulty level, etc. Information necessary for the already evaluated answer is accepted.

  After step S111 ends, the CPU 11 returns the process to step S109. When the selection of the question is not accepted in step S108 (S108: NO), the CPU 11 registers it on the question list screen in which the operator operates the input device 15 to extract the already evaluated answer as shown in FIG. The registration instruction reception by pressing the button is monitored (S112). When the registration instruction is not accepted (S112: NO), the CPU 11 returns the process to step S108. When the registration instruction is accepted (S112: YES), the CPU 11 records the already evaluated answer for each of the plurality of questions included in the test question in the question-by-question already evaluated answer DB 14e (S113), and performs the already evaluated answer extracting process. finish.

  FIG. 9 is a conceptual diagram showing an example of the contents stored in the question-based already evaluated answer DB 14e. Fiscal year, times, classes, major questions , And subquestion No. The image data is recorded in association with the information specifying the question, and the information about the right / wrong classification and the difficulty level is recorded. Furthermore, a mixed serial number for distinguishing each of a plurality of already evaluated answers associated with each question is recorded.

  The internal storage device 14 also includes an unevaluated answer DB 14f that records an unevaluated answer that is an answer of a test performed using a test question that indicates the content of the test question data 14b. The unassessed answer was created by a method such as taking the answer written on paper with a scanner, or by accepting the answer as electronic data by the examinee operating an input device such as a tablet during the test. It is data. The unassessed answer DB 14f records image data of the answer by the examinee in association with information specifying each question included in the test question and the examinee, and each of the plurality of questions included in the exam question is recorded. Unassessed answers are recorded in the order of examinee codes that identify examinees. The internal storage device 14 also includes a unevaluated answer DB 14g for mixing, which records an unevaluated answer to which information indicating a mixing position where the already evaluated answer is mixed is added to the unevaluated answer.

  FIG. 10 is a flowchart showing the procedure of the mixing process performed by the already-assessed answer mixing device 1 of the present invention. The CPU 11 executes the following processing according to the computer program 20 loaded into the RAM 12. The CPU 11 reads the unevaluated answer concerning the candidate to be scored from the unevaluated answer DB 14f to the RAM 12 (S21), and multiplies the number of examinees corresponding to the read unevaluated answer by a predetermined mixing rate, The number of already evaluated answers mixed is determined (S22). The mixing rate may be processed by the input device 15 by an operator's operation. Next, the CPU 11 determines a mixing position for mixing the number of already-assessed answers in the order of the unevaluated answers according to the examinee number (S23). At this time, the CPU 11 may determine the mixing position randomly using a random number, or may determine the mixing position according to a predetermined rule. Next, the CPU 11 adds mixing position information indicating the mixing position of the already evaluated answer to the determined mixing position with respect to the unevaluated answer to be scored (S24), and the mixing position information is added. The unevaluated answer (order designation information) is stored in the unevaluated answer DB 14g for mixing (S25), and the process ends.

  FIG. 11 is a conceptual diagram showing an example of the contents stored in the unevaluated answer DB 14g for mixing. Corresponding to the number of the output order that is output at the time of scoring, an unevaluated answer of each examinee identified by the examinee code is recorded. Further, a special code such as “9999” at the end is recorded as the mixing position information in association with the output order number corresponding to the mixing position.

  The internal storage device 14 also stores the grader data 14h in which the scorer code (identification information) for identifying the grader is recorded, and the difficulty level in which information for associating the grader with the difficulty level of the already evaluated answer is recorded. Corresponding data 14i is stored.

  FIG. 12 is a conceptual diagram showing an example of the content of the grader data 14h. In association with the grader code for identifying each grader, information indicating the name of the grader and the rank of the grader is recorded. The grader's rank indicates that the smaller the number, the higher the ability of the grader. FIG. 13 is a conceptual diagram showing an example of the content of the difficulty level correspondence data 14i. In association with the rank of the grader, the upper limit and the lower limit of the difficulty level of the already evaluated answer mixed in the unassessed answer at the time of scoring are recorded. Based on the grader data 14h and the difficulty correspondence data 14i, the difficulty level of the already evaluated answer when determining the ability of each grader is determined.

  The internal storage device 14 includes a scoring result DB 14j that records the scoring results of the scorer.

  FIG. 14 is a flowchart showing the procedure of scoring processing performed by the already-assessed answer mixing device 1 of the present invention. The CPU 11 executes the following processing according to the computer program 20 loaded into the RAM 12. The CPU 11 monitors the acceptance of the grader code when the grader operates the grader terminal 3 and inputs the grader code (S301), and if no grader code is accepted (S301: NO), the grader Continue monitoring code acceptance. When the grader code is accepted (S301: YES), the CPU 11 determines whether or not the accepted grader code is a valid grader code recorded in the grader data 14h (S302). If it is not a grader code (S302: NO), the process returns to step S301. When the accepted scorer code is a valid scorer code (S302: YES), the CPU 11 handles the difficulty according to the rank of the scorer recorded in the scorer data 14h in association with the scorer code. As recorded in the data 14i, the upper limit and the lower limit of the difficulty level of the already evaluated answer are determined (S303).

  Next, the CPU 11 searches a question included in the test question from the beginning and selects a question that has not yet been scored by another grader, or selects a question that is predetermined according to the grader code. The question corresponding to the rank of the grader is selected, or the question to be scored is determined by the grader specifying the question to be graded (S304). In the subsequent processing, for example, a process of outputting a plurality of answers to one question such as the sub-question 1 of the big question 5 to the scoring terminal 3 is performed, and the scorer scores a plurality of answers to the question. .

  Next, the CPU 11 selects an answer according to the output order recorded in the unevaluated answer DB 14g for mixing (S305), and whether or not the mixed position information is recorded at the position of the selected answer in the unevaluated answer DB 14g for mixing. Is determined (S306).

  If the mixed position information has been recorded (S306: YES), the CPU 11 determines the difficulty level in step S303 from among a plurality of already-evaluated answers recorded in the question-based already-evaluated answer DB 14e in association with the question. An already evaluated answer that falls within the determined upper and lower limits is selected (S307). Next, the CPU 11 creates output data 14m for recording answer data to be output and scored by the grader, and records the selected evaluated answer in the output data 14m (S308). When selecting an already evaluated answer, it may be selected randomly from the already evaluated answers corresponding to the difficulty level, or may be selected in the order of mixed serial numbers from the already evaluated answer answers corresponding to the difficulty level. .

  When the mixed position information is not recorded in step S306 and an unevaluated answer is recorded (S306: NO), the CPU 11 does not record the selected answer in the unevaluated answer DB 14g for mixing. The evaluation answer is recorded in the output data 14m (S309).

  After recording the already evaluated answer or the unevaluated answer in the output data 14m in step S308 or step S309, the CPU 11 determines whether all the answers to be output for scoring are recorded in the output data 14m. (S310). If there is an answer not recorded in the output data 14m (S310: NO), the CPU 11 returns the process to step S305, and performs a process of recording the next answer to be output in the output data 14m. When all the answers to be output for scoring are recorded in the output data 14m (S310: YES), the CPU 11 stores the output data 14m in the internal storage device 14 (S311).

  FIG. 15 is a conceptual diagram showing an example of the contents of the output data 14m. Corresponding to the number of the output order that is output at the time of scoring, an unevaluated answer of each examinee identified by the examinee code is recorded. Furthermore, the selected already evaluated answer is recorded in association with the output order number corresponding to the mixing position.

  Next, the CPU 11 outputs a predetermined number of unevaluated answers and already evaluated answers recorded in the output data 14m to the scoring terminal 3 in the order of output (S312). At this time, a predetermined number of answers to be scored by the scorer are output to the output unit of the scoring terminal 3 such as a CRT display. Next, the CPU 11 monitors the acceptance of scoring for the answer output to the output unit of the scoring terminal 3 when the scorer operates the input unit of the scoring terminal 3 (S313).

  FIG. 16 is a schematic diagram illustrating an output example in which the scoring terminal 3 outputs a predetermined number of answers. The answer to the selected question is output by a predetermined number corresponding to the size of the output screen of the output unit of the scoring terminal 3. The grader operates the input unit of the scoring terminal 3 and inputs the scoring results for each answer.

  The CPU 11 continues to monitor the reception of the grading when there is no grading reception with respect to the answer (S313: NO), and when the grading with respect to the output answer is received (S313: YES), the scorer takes charge. It is determined whether scoring for the question is completed (S314). If scoring has not been completed and there are still answers to be scored (S314: NO), the CPU 11 returns the processing to step S312 to give a predetermined number of answers to be scored by the scorer. And continue the scoring process. When the scoring is completed (S314: YES), the CPU 11 records the scoring result of the scorer in the scoring result DB 14j (S315), and ends the process. In the above scoring process, the process of outputting the output data 14m once and then outputting the answer to the scoring terminal 3 is performed. However, the present invention is not limited to this, and the unevaluated answer from the unevaluated answer DB 14g for mixing is used. Is directly read out and output, and when the mixed position information is recorded at the position of the answer, processing for selecting and outputting the already evaluated answer from the question-by-question already evaluated answer DB 14e based on the difficulty level is performed each time. Form may be sufficient.

  FIG. 17 is a conceptual diagram showing an example of the contents stored in the scoring result DB 14j. A score of a correct answer or an incorrect answer is recorded in association with each question of an unassessed answer for each examinee, and a scorer code of a scorer who scored the question is recorded. In addition, information about the correctness / incorrectness classification and the difficulty level is further recorded in association with the mixed already-assessed answer.

  FIG. 18 is a flowchart showing the procedure of the evaluation process performed by the already evaluated answer mixing apparatus 1 of the present invention. The CPU 11 executes the following processing according to the computer program 20 loaded into the RAM 12. The CPU 11 extracts the score of the already evaluated answer from the scoring result DB 14j (S41), and selects the scorer to be evaluated (S42). Next, the CPU 11 compares the information of the correct answer classification of the already evaluated answer and the information of the already evaluated answer with respect to the score of the already evaluated answer for the selected scorer, and the score of the scorer has already been evaluated. A correct answer rate indicating a ratio of correct answers to the determined evaluation is calculated for each difficulty level of the already evaluated answer (S43). Next, the CPU 11 generates evaluation information indicating the evaluation of the scorer according to the calculated accuracy rate (S44). At this time, for example, when the accuracy rate of the difficulty level of the same numerical value as the grader's rank is 90% or more and the accuracy rate of the difficulty level one level above is 50% or more, the evaluation of the scorer is excellent. When the accuracy rate of the difficulty level of the same number as the rank of the grader is 50% or less, the grader is not evaluated.

  Next, the CPU 11 determines whether or not the evaluation has been completed for all the graders (S45). If there is a grader that has not been evaluated yet and has not yet been evaluated (S45: NO). The process returns to step S42 to select a grader to be evaluated. When the evaluation has been completed for all the graders (S45: YES), the CPU 11 stores the evaluation data 14k in which the evaluation information for each grader is recorded in the internal storage device 14 (S46), and ends the processing. To do. At this time, if the accuracy rate of the difficulty level of the same numerical value as the rank of the grader is 90% or more and the accuracy rate of the difficulty level one level above is 50% or more, it is recorded in the grader data 14h. The grader data 14h is increased based on the evaluation data 14k, such as when the rank of the grader is increased and the accuracy rate of the difficulty level of the same number as the rank of the grader is 50% or less. You may perform the process to update. Based on the contents of the evaluation data 14k, the scoring ability of the scorer is determined, and guidance to the scorer is performed.

  As described above in detail, in the present invention, an already-evaluated answer in which an evaluation of a correct answer or an incorrect answer is determined in advance for a plurality of questions that may be used as a test question is stored and included in an actual test question. An already evaluated answer for each question is extracted and mixed in the order of a plurality of unassessed answers that have not yet been scored by the examinee. As a result, when the grader scores an unassessed answer, the already evaluated answer can be scored at the same time, and the evaluation of the correct answer or the incorrect answer given by the grader to the already evaluated answer is predetermined. By comparing the evaluation, the ability of the scorer can be objectively determined.

  Further, in the present invention, the position where the already evaluated answer is mixed in the unassessed answer is determined without fixing which evaluated answer is included in the unassessed answer, and the scorer answers the answer. When scoring the score, the already evaluated answer is interrupted at the mixing position and scored. This makes it easy to select an already evaluated answer to be mixed according to the type of question or the grader, and the ability of the grader can be accurately determined.

  Furthermore, in the present invention, the rank is set for each grader, the degree of scoring difficulty is set for each graded answer, and when the graded grade is mixed with the graded grade, Mix the already evaluated answers with the difficulty according to. As a result, high-ranked scorers who have previously been judged to have high scoring ability score highly evaluated answers that are highly difficult, and low-ranked scorers mark previously evaluated answers that are less difficult, etc. It becomes possible to score an appropriate already evaluated answer according to the ability of the grader, and the ability can be determined in more detail according to the ability of the grader.

  By using the present invention as described above, the ability of the grader can be objectively determined when the grader grades the answer to the test question. Therefore, the ability of the grader is improved using the judgment result as an index. This makes it possible to score test answers more accurately.

  In the present embodiment, an embodiment has been described in which processing for creating an unevaluated answer for mixing to which mixing position information is added in the mixing process and storing it in the unevaluated answer DB for mixing 14g is shown. Instead, information indicating the order of the unevaluated answer according to the examinee code and the mixing position is created and stored in the internal storage device 14 as the order designation information according to the present invention, and is not stored in accordance with the order designation information in the scoring process. The form which performs the process which reads an evaluation answer and an already evaluated answer may be sufficient.

  Further, in the present embodiment, the form of performing the process of selecting the already-evaluated answer to be output according to the degree of difficulty is shown, but the present invention is not limited to this, and any grader without using the degree of difficulty information For example, a method of selecting an already-evaluated answer to be output by a similar method may be used. In addition, although the form of determining the difficulty level for each of the already evaluated answers is shown, there is a difficulty in scoring the question itself, such as a kanji that is easily mistaken for others or a question that is difficult to judge whether the answer is correct or incorrect Since the difficulty level information is stored in association with the question, the grader selects the question to be graded according to the rank of the grader and the difficulty level of the question. Also good.

  Further, in the present embodiment, the present invention is described as an example in which the present invention is applied to scoring a Chinese character test, but the scope of the present invention is not limited to the Chinese character test, and is applicable to other tests. Is possible.

It is a block diagram which shows the internal structure of the already evaluated answer mixing device of this invention. It is a conceptual diagram which shows the example of the memory content of an already evaluated answer DB. It is a conceptual diagram which shows the example of the content of test question data. It is a conceptual diagram which shows the example of the content of extraction condition data. It is a conceptual diagram which shows the example of the content of a homonym data. It is a flowchart which shows the procedure of the already evaluated answer extraction process which the already evaluated answer mixing device of this invention performs. It is a schematic diagram which shows the example of a screen of the list of the questions which the evaluated evaluation answer mixing apparatus output to the output device which extracted the already evaluated evaluation answers. It is a schematic diagram which shows the example of a screen of the already evaluated answer about the one question which the already evaluated answer mixing apparatus output to the output device. It is a conceptual diagram which shows the example of the memory | storage content of problem-based already evaluated answer DB. It is a flowchart which shows the procedure of the mixing process which the already evaluated answer mixing apparatus of this invention performs. It is a conceptual diagram which shows the example of the memory content of the unevaluated answer DB for mixing. It is a conceptual diagram which shows the example of the content of grader data. It is a conceptual diagram which shows the example of the content of difficulty corresponding data. It is a flowchart which shows the procedure of the scoring process which the already evaluated answer mixing device of this invention performs. It is a conceptual diagram which shows the example of the content of the data for output. It is a schematic diagram which shows the example of an output which the scoring terminal output the predetermined number of answers. It is a conceptual diagram which shows the example of the memory content of scoring result DB. It is a flowchart which shows the procedure of the evaluation process which the already evaluated answer mixing device of this invention performs.

Explanation of symbols

1 Evaluated answer mixing device 11 CPU (calculation unit)
14 Internal storage device (storage unit)
14a Already evaluated answer DB (first storage means)
14e Question-based already evaluated answer DB (second storage means)
2 Recording medium 20 Computer program 3 Scoring terminal (output means)

Claims (10)

A method of using a computer having a storage unit, a calculation unit, and an output unit to mix an already-evaluated answer whose evaluation has already been determined among a plurality of unevaluated answers to each of a plurality of questions included in a test question. ,
A plurality of already evaluated answers to the question are stored in the storage unit in association with each of the questions,
Extracting an arbitrary number of already-evaluated answers from the plurality of already-evaluated answers stored in the storage means in association with each of a plurality of questions included in the test question by the calculation unit,
The arbitrary number of already evaluated answers extracted by the calculation unit are stored in the storage unit in association with each of a plurality of questions included in the test question,
A plurality of unassessed answers to one question among a plurality of questions included in the test question and a predetermined number taken out of the arbitrary number of already evaluated answers stored in the storage unit in association with the one question An already-evaluated answer mixing method, wherein the output unit outputs a number of already-evaluated answers. A method of mixing an already-evaluated answer whose evaluation has already been determined in a predetermined order of a plurality of unassessed answers for each of a plurality of questions included in a test question, using a computer including a storage unit and a calculation unit Because
A plurality of already evaluated answers to the question are stored in the storage unit in association with each of the questions,
Extracting an arbitrary number of already-evaluated answers from the plurality of already-evaluated answers stored in the storage means in association with each of a plurality of questions included in the test question by the calculation unit,
The arbitrary number of already evaluated answers extracted by the calculation unit are stored in the storage unit in association with each of a plurality of questions included in the test question,
The calculation unit determines the predetermined number of mixing positions for mixing a predetermined number of already evaluated answers in the order of the plurality of unassessed answers,
The storage unit stores order specifying information for specifying the order of each of the plurality of unevaluated answers and the predetermined number of already evaluated answers, including mixing position specifying information for specifying each mixing position determined by the calculation unit. An already-evaluated answer mixing method characterized by this. The computer further includes an output unit,
Select one question from a plurality of questions included in the test question in the calculation unit,
Among the arbitrary number of already-assessed answers stored in the storage unit in association with the one question selected by the calculation unit, the predetermined number of the specified number of the mixture position designation information included in the order designation information Select each of the already evaluated answers to be mixed in each of the mixing positions in the calculation unit,
The plurality of unassessed answers to the one question and the predetermined number of already-evaluated answers selected by the calculation unit are sequentially output from the output unit according to the order specified by the order designation information. Item 2. The method for mixing already evaluated answers according to Item 2. The computer further includes a reception unit that receives the identification information of the grader,
The identification information of the grader is stored in the storage unit in association with the rank of the grader,
The degree of difficulty of the already evaluated answer is stored in the storage unit in association with each of the already evaluated answers,
When selecting the predetermined number of already-evaluated answers in the calculation unit, the already-evaluated answer associated with the degree of difficulty according to the rank associated with the identification information received by the receiving unit is selected. The method for mixing already evaluated answers according to claim 3. A device that mixes an already evaluated answer whose evaluation has already been determined in a predetermined order of a plurality of unassessed answers for each of a plurality of questions included in a test question,
First storage means for storing a plurality of already-assessed answers to the question in association with each of the questions;
Means for extracting an arbitrary number of already evaluated answers from the plurality of already evaluated answers stored in the first storage means in association with each of a plurality of questions included in the test question;
Second storage means for storing the arbitrary number of already evaluated answers extracted by the means in association with each of a plurality of questions included in the test question;
Means for determining the predetermined number of mixing positions for mixing a predetermined number of already evaluated answers in the order of the plurality of unassessed answers;
Means for storing order designation information for designating the order of each of the plurality of unevaluated answers and the predetermined number of already evaluated answers, including mixing position designation information for designating each mixing position determined by the means; Evaluated answer mixing device characterized by. Means for selecting one question from a plurality of questions included in the test question;
The predetermined number designated by the mixed position designation information included in the order designation information from the arbitrary number of already evaluated answers stored in the second storage means in association with the one question selected by the means. An already-evaluated answer selection means for selecting each of the already-evaluated answers to be mixed in each of the mixing positions of
An output means for sequentially outputting each of the plurality of unassessed answers to the one question and the predetermined number of already-evaluated answers selected by the already-assessed answer selecting means according to the order designated by the order designation information. The already-assessed answer mixing apparatus according to claim 5. Identification information storage means for storing the identification information of the grader in association with the rank of the grader;
An identification information receiving means for receiving the identification information;
The second storage means is adapted to store the degree of difficulty of the already evaluated answer in association with each of the already evaluated answers,
The already-evaluated answer selecting unit is configured to select an already-evaluated answer associated with the degree of difficulty according to the rank associated with the identification information received by the identification information receiving unit. The already evaluated answer mixing apparatus according to 6. In relation to each of a plurality of questions included in a test question, a plurality of unassessed answers for each of the plurality of questions are determined in advance in a computer that stores a plurality of already-evaluated answers for which the evaluation has already been determined. A computer program that mixes already evaluated answers in the order,
A procedure for causing a computer to extract an arbitrary number of already evaluated answers from the plurality of already evaluated answers related to each of a plurality of questions included in the test question;
A procedure for causing the computer to store the arbitrary number of already evaluated answers in association with each of a plurality of questions included in the test question;
A procedure for causing the computer to determine the predetermined number of mixing positions for mixing a predetermined number of already evaluated answers in the order of a plurality of unassessed answers;
And a procedure for storing order designation information for designating the order of each of the plurality of unevaluated answers and the predetermined number of already evaluated answers, including mixing position designation information for designating each mixing position. A computer program. A procedure for causing a computer to select one question from a plurality of questions included in the test question;
In the computer, each of the predetermined number of mixed positions designated by the mixed position designation information included in the order designation information from among the arbitrary number of already evaluated answers stored in association with the selected one question. A procedure for selecting each of the already evaluated answers to be mixed,
And a step of causing the computer to sequentially output each of the plurality of unevaluated answers to the one question and the selected predetermined number of already evaluated answers in the order specified by the order specification information. Item 9. The computer program according to Item 8.   According to the rank associated with the received identification information, the computer stores the identification information of the grader in association with the rank of the grader, and stores the degree of difficulty of the already evaluated answer in association with each of the evaluated evaluations. The computer program according to claim 9, further comprising a procedure for selecting an already-evaluated answer associated with the degree of difficulty as an already-evaluated answer to be output.

Images