JP2003085296A - Method and device for evaluating test question and its program and storage medium with its program stored thereon - Google Patents

Abstract

PROBLEM TO BE SOLVED: To judge whether or not test questions have the proper level of difficulty by evaluating those test questions according to contents. SOLUTION: Test questions are managed in a storage means in a hierarchical structure according to the contents of the test questions, and the level of difficulty of the test questions is calculated from the answer data of examinees in the test, and the test questions whose level of difficulty does not match the vertical relation of the hierarchical structure are detected.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a test question evaluation method and apparatus, a test question evaluation program, and a storage medium storing the test question evaluation program, and more particularly, to improve accuracy of ability measurement in a test. By managing the exam questions in a hierarchical structure and comparing the difficulty levels of the upper and lower exam questions using the hierarchical structure, you can find inappropriate exam questions and correct or delete them.
The present invention relates to a test question evaluation method and apparatus for improving the accuracy of test ability measurement, a test question evaluation program, and a storage medium storing the test question evaluation program.

[0002]

2. Description of the Related Art As a conventional method for evaluating test questions,
SP table (as an example of the SP table, http: //www.chuoh-k
youiku.co.jp/soft/sp, http://www.eyotsuva.com/eyo/
kyoukoku / image / sousasp.pdf and Takahiro Sato "Introduction to SP Table" Meiji Books, Toshiaki Mori, Kiyomi Akita "Educational Evaluation"
Meiji books etc.) The SP table is a test result display method mainly used by the teacher to grasp the degree of understanding of the student, but is also used to find a defect in the test question from the answer data of the student.

As shown in FIG. 11, on the vertical axis of the SP table, the names of students are displayed from the top to the bottom in the order of the highest legal numbers, and on the horizontal axis, the exams are tested from the left to the right in the highest legal number. The problem is displayed. From the column of the test question answered correctly by each examinee, it is possible to comprehensively understand which test question the examinee has answered correctly and made an incorrect answer, and vice versa, which examinee has answered the correct and incorrect questions. be able to.

A specific method for evaluating a test question using the SP table is a test in which a test taker with a high ability gives a wrong answer but a test taker with a poor ability gives a correct answer. The problem is to find a test question that has an answer result that is difficult to think, such as a question, and speculate that there is some kind of failure.

In order to evaluate the test questions, there is also a method of managing the correct answer rate for each question ID in the question management database as shown in FIG.

[0006]

However, the conventional method for evaluating test questions using the SP table is difficult to compare and evaluate the test questions because the content and category of the test questions are not taken into consideration. There is a problem.

Originally, the test should be designed so that the ability of a population of examinees with a particular ability distribution can be properly evaluated. Candidates with high abilities receive high scores,
Candidates with low abilities should consist of a set of exam questions that yield low scores. To do this, each test question must have an appropriate degree of difficulty.
Exams should include exam questions that can be answered by basic knowledge and skills, as well as exam questions that require deep knowledge and skills.

In order to realize such a test design, a means is needed to assess the difficulty of the individual test questions.

There are two ways to assess whether an individual exam question is of adequate difficulty. One is a method of referring to the answer data of the examinee to find an exam question in which an examinee having a high ability gives a wrong answer but an examinee having a poor ability answers correctly. However, it seems that there is a defect in the examination problem even if one test taker with high ability accidentally makes a wrong answer and one test taker with low ability makes a correct answer. In other words, it is difficult to evaluate the examinee's entire answer.

The other is, as shown in FIG.
This is a method of comparing exam questions by managing the correct answer rate for each question in a question management database. From the answer data of the examinee, the difficulty of the test question may be the number of correct answers, for example. If the test question is correct in 5 out of 10, 0.5 is 0.2 if only 2 in 10 are correct, and the lower the number, the more difficult the test problem is. You can think about the degree. Considering this difficulty level, exam questions that require deep knowledge and skills should have lower numbers than exam questions that can be answered by basic knowledge and skills. Since this method does not refer to the individual answers of individual examinees, it enables a more comprehensive evaluation of examination questions.

However, there is a problem in the comparison of test questions. Comparing the exam questions of advanced network security technology with the exam questions of the underlying database language would not be a meaningful comparison.
This is because the areas of knowledge and skills required are different. Candidates with advanced network security technology may not have knowledge or skills about databases.

In the SP table, both the test questions concerning the network and the test questions concerning the database are simply arranged on the horizontal axis of the SP table in the order of the number of correct answers. Since all test questions are treated in the same row regardless of the content, no information is provided for selecting comparison targets, which test questions should be more difficult or simpler than which ones. Therefore, there is a problem that it is difficult to compare test questions.

The test questions related to advanced network security technology should be compared with the test questions that are the basic knowledge directly related to the content thereof, for example, the test questions in the terminology related to networks.

That is, in order to evaluate whether the test questions have an appropriate difficulty level, when comparing the difficulty levels of the test questions, it is necessary to provide a method capable of making a comparison according to the contents of the test questions. .

The present invention has been made in view of the above points, and it is possible to compare test questions according to their contents, and a test question evaluation method capable of determining whether or not the difficulty level is appropriate. An object is to provide an apparatus, a test question evaluation program, and a storage medium storing the test question evaluation program.

[0016]

FIG. 1 is a diagram for explaining the principle of the present invention.

According to the present invention (claim 1), in a test question evaluation method for evaluating a qualification test or a certification test using a computer, the test problems are stored in a storage means in a hierarchical structure according to the contents of the test problem. (Step 1), the difficulty of the test question is obtained from the answer data of the examinee in the test using the test question (step 2), and the test question in which the hierarchical relationship of the hierarchical structure and the difficulty are inconsistent is detected ( Step 3). The present invention (Claim 2) determines that there is a contradiction when an inconsistent test question is lower in difficulty than an inferior test question in a hierarchical structure when detecting an inconsistent test question. Flag exam questions in and exam questions below as inappropriate.

FIG. 2 is a block diagram showing the principle of the present invention.

The present invention (Claim 3) is a test question evaluation apparatus for evaluating a qualification test or a certification test using a computer, and a test question database 21 managed in a hierarchical structure according to the contents of the test question. And the answer analysis means 22 for obtaining the difficulty level of the test question from the answer data of the examinee in the test using the test question, and the test question database 2
Contradiction detection means 2 for detecting a test question in which the hierarchical relationship of 1 and the difficulty level obtained by the answer analysis means 22 are inconsistent
3 and 3.

The present invention (Claim 4) provides a contradiction detecting means 23.
In the above, in the case where the test question at the higher level of the hierarchical structure has a lower difficulty level than the test question at the lower level, a means to determine that there is a contradiction and an inappropriate flag for the test question at the higher level and the test question at the lower level are set. And means for standing.

The present invention (Claim 5) is a test question evaluation program for evaluating a qualification test or qualification test using a computer, which uses a test question database managed in a hierarchical structure according to the contents of the test question. An answer analysis process that finds the difficulty level of the test question from the answer data of the examinee for the given test question, and a test question in which the hierarchical structure of the test question database and the difficulty level determined in the answer analysis process are inconsistent And a contradiction detection process.

According to the present invention (Claim 6), in the contradiction detection process, if a test question at a higher level of the hierarchical structure has a lower difficulty level than a test question at a lower level, it is determined that there is a contradiction and a higher level. The exam questions in and the exam questions below are flagged as inappropriate.

The present invention (claim 7) is a storage medium in which a test question evaluation program for evaluating a qualification test or a certification test using a computer is stored, and is managed in a hierarchical structure according to the contents of the test problem. The answer analysis process that finds the difficulty level of the test question from the answer data of the test question given from the exam question database, and the hierarchical relationship of the hierarchical structure of the test question database and the difficulty level determined by the answer analysis process are inconsistent. And a contradiction detection process for detecting test questions.

According to the present invention (claim 8), in the contradiction detection process, when a test question at a higher level of the hierarchical structure is lower in difficulty than a test question at a lower level, it is determined that there is a contradiction and a higher level. The exam questions in and the exam questions below are flagged as inappropriate.

As described above, according to the present invention, by managing the examination questions in a hierarchical structure according to the contents, it is possible to ask for advanced knowledge and skills in the same contents, not in the questions of unrelated contents. Compare exam questions with exam questions that ask for basic knowledge or skills. This makes it possible to compare the difficulty levels of meaningful test questions.

As for the test question, which is detected to have inconsistency in difficulty, it is prevented from being used in the next test, is corrected, or is deleted, so that the accuracy of the ability measurement of the test is improved. Can be achieved.

[0027]

BEST MODE FOR CARRYING OUT THE INVENTION In order to realize comparative evaluation between test questions, the present invention proposes a method of appropriately classifying and managing test questions according to their contents and using them for comparison. Specifically, according to the content and category, the questions that require advanced knowledge and skills are ranked higher, and the exam questions that require basic knowledge and skills related to them are ranked lower. Take

FIG. 3 is a diagram showing a hierarchical structure of learning targets in one embodiment of the present invention. The knowledge that the test questions ask has a learning sequence at the time of learning. Candidates who cannot write the kanji "fire" It is usually unthinkable that the examinee can write the kanji "charcoal". Similarly, examinees who are unable to read and write kanji cannot read or narrate sentences because they have high and low skills to ask for exam questions. Therefore, it can be considered that the test questions have a hierarchical relationship of difficulty among related knowledge and skills.

FIG. 4 shows an example of a method for describing the hierarchical structure of a test question in one embodiment of the present invention, and FIG. 5 shows an example of a hierarchical structure model of a test question in the one embodiment of the present invention (part thereof). 1). FIG. 5 is a model of the hierarchical structure of the test questions represented by FIG. 3 and FIG. 4, and the hierarchical relationship of the difficulty of the test questions is shown using the numbers of the test questions: 1>11> It can be considered as 111 .1>11> 112 .1>12> 121 .1>12> 122.

In the exam questions managed in a hierarchical structure, the exam questions lower than one's own knowledge and level have a lower degree of difficulty than one's own knowledge and level, and the exam questions above one's knowledge and level are: It is designed to be more difficult than your knowledge and level.

The test problem in which the hierarchical relationship of these hierarchical structures and the difficulty level are inconsistent is considered to have some trouble. A defect is a mistake in which an examinee who has the ability to correct the question is used with inappropriate content or terminology that is beyond the scope of the question that the examinee had shown to the examinee in advance, or who should have the correct answer. That is, the contents such as a hook problem may be included.

FIG. 6 shows the configuration of a test question evaluation device according to an embodiment of the present invention.

The apparatus shown in the figure comprises a test question database 21, an answer analysis section 22, and a contradiction detection section 23.

The exam question database 21 classifies and manages exam questions according to their contents, and the exam questions that ask for advanced knowledge and skills are located in the upper rank, and the exam questions that ask basic knowledge and skills related thereto are in the lower rank. It is managed in a hierarchical structure so that it is located at.

The answer analysis unit 22 obtains the difficulty level of the test question from the answer data of the examinee after the test is performed.

The contradiction detection unit 23 detects a test question having a contradiction in the hierarchical relationship of the hierarchical structure and the difficulty level.

[0037]

Embodiments of the present invention will be described below with reference to the drawings.

First, the operation of the test question evaluation system shown in FIG. 6 will be described.

The test question database 21 is a database that stores test questions created by the test organizer. The test questions are classified according to the content, as shown in FIG.
Numbered. In FIG. 4, the number of the test question that requires the most advanced knowledge and skill is "1", the related test question related to the LAN is "11", and the applied test question related to security is "12". ] Is divided. Further, test questions asking basic knowledge and skills related to LAN are designated as “111” and “112” respectively, and test questions asking basic knowledge and skills regarding security as “121” and “122” respectively. FIG. 5 shows an example of modeling the hierarchical structure described in this way. The number of trees having a hierarchical structure is not limited to one in one test, and a plurality of trees having a hierarchical structure as shown in FIG. 5 exist.

Further, in the hierarchical structure, one test question is not always located immediately above one test question.
There may be multiple exam questions immediately above the exam question, where one exam question is an exam question that asks for subordinate knowledge or skills associated with multiple exam questions. An example of the test question management method at this time is shown in FIG. In the figure,
Assign any number to the test questions and describe the test questions immediately above and below each test question as parents and children. This allows
Describe the hierarchical relationship between each test question. FIG. 8 shows an example of modeling the hierarchical structure described in this way.

The answer analysis unit 22 obtains the difficulty level of the test question from the answer of the examinee, and adds the difficulty level data to the test question in the test question database 21. The difficulty of the test question can be calculated by using the correct answer rate or using item response theory (Kenji Otomo "Introduction to Item Response Theory" Daishukan Shoten, Hiroshi Ikeda "Modern Test Theory" Asakura Shoten) it can.

The correct answer rate represents the number of correct answers by all the examinees, and takes a value of 0 to +1.00 from the most difficult problem to the simplest problem.

The item response theory is a mathematical model showing the test taker's reaction to each test question. Generally, the test taker's ability is represented by θ, and the probability of correct answer to each item is a monotone of θ. It is represented by an increasing function p (θ), and the difficulty of the test question is represented by b. Candidate's ability θ and difficulty of exam questions b
Can be expressed on the same scale. That is, it can be interpreted that the ability to answer the difficulty level b is θ. b is + 3.00-from the most difficult problem to the easiest problem
Takes a value of 3.00. Under the item response theory, the difficulty of the test question does not depend on the ability of the examinee, which is a great difference from the difficulty using the correct answer rate.

The contradiction detection section 23 compares the difficulty levels of the test questions obtained by the answer analysis section 22 according to the hierarchical relationship of the hierarchical structure of the test questions. FIG. 8 is a diagram for explaining the contradiction detection method according to the embodiment of the present invention. here,
It is assumed that the difficulty determined by the item response theory is added to the test question. The difficulty is compared with the exam question that is immediately below it from the highest exam question.

In the example of FIG. 8, first, "problem 1" and "problem 11" are compared. Next, "Problem 1" and "Problem 12"
To compare. Next, "Problem 11" and "Problem 111",
"Problem 11" and "Problem 112", "Problem 12" and "Problem 121", "Problem 12" and "Problem 122" are compared in this order.

The contradiction detection section 23 detects a contradiction when the difficulty level of the lower-level test question is higher than that of the higher-level test question.

In the example of FIG. 8, "Problem 12" is "Problem 1".
22 ”, but the difficulty level is low, so it is detected that there is a contradiction. When there are a plurality of hierarchically structured trees, similar inconsistency detection is performed in each tree.

The detection result is reflected on the test question database 21, and the "problem 12" and "question 122" are flagged as inappropriate as shown in FIG.

As shown in FIGS. 7 and 10, when there are a plurality of test questions immediately above one test question, not only are "problem 1" and "problem 5" compared, but ,
"Problem 2" is also compared with "Problem 5", and similar contradiction detection is performed.

The degree of difficulty and the inappropriateness flag are described on the test question database 21 as shown in FIG. The inappropriate flag can be canceled when the test organizer corrects or deletes the test question on the test question database 21.

Further, in the present invention, the configuration of the test question evaluation device in the above-mentioned embodiment can be constructed as a program and installed in a computer used as the test question evaluation device, or distributed through a network. Is.

A hard disk, a floppy (registered trademark) disk, or a CD connected to the computer used as the test question evaluation device for the constructed program.
-The present invention can be easily realized by storing it in a portable storage medium such as a ROM and installing it when carrying out the present invention.

The present invention is not limited to the above embodiments, and various modifications and applications are possible within the scope of the claims.

[0054]

As described above, according to the present invention, the test questions can be compared according to the contents, and it is possible to judge whether the difficulty level is appropriate. This makes it possible to detect a test question of which the difficulty level is not appropriate. By not using, correcting, or deleting the detected test questions for the next test, it is possible to improve the accuracy of the ability estimation of the examinees.

[Brief description of drawings]

FIG. 1 is a diagram for explaining the principle of the present invention.

FIG. 2 is a principle configuration diagram of the present invention.

FIG. 3 is a diagram showing a hierarchical structure of learning targets according to an embodiment of the present invention.

FIG. 4 is an example of a method of describing a hierarchical structure of a test question according to an embodiment of the present invention.

FIG. 5 is an example of a hierarchical structure model of a test question in one embodiment of the present invention.

FIG. 6 is a configuration diagram of a test question evaluation device according to an embodiment of the present invention.

FIG. 7 is an example of a method of describing a test question hierarchical structure according to an embodiment of the present invention.

FIG. 8 is a diagram for explaining a contradiction detection method according to an embodiment of the present invention.

FIG. 9 is a diagram showing a method of describing a difficulty level and an inappropriate flag according to an embodiment of the present invention.

FIG. 10 is an example of a hierarchical structure model of a test question according to an embodiment of the present invention.

FIG. 11 is a diagram for explaining an SP table in the conventional technique.

FIG. 12 is an example of a description of a problem management database in the related art.

[Explanation of symbols]

21 Exam question database 22 Answer analysis means, answer analysis section 23 Contradiction detection means, conflict detection unit

Continued front page    (72) Inventor Akito Nagatsu             2-3-1, Otemachi, Chiyoda-ku, Tokyo             Inside Telegraph and Telephone Corporation

Claims (8)

[Claims] 1. A test question evaluation method for evaluating a qualification test or qualification test using a computer, wherein the test questions are managed in a storage means in a hierarchical structure according to the contents of the test questions, A test question evaluation method, wherein the difficulty level of the test question is obtained from the answer data of the examinee in the test using, and the test question in which the hierarchical relationship of the hierarchical structure and the difficulty level are contradictory is detected. 2. When detecting the contradictory test questions, if the higher-ranked test questions in the hierarchical structure have a lower difficulty level than the lower-ranked test questions, it is determined that there is a contradiction, and The test question evaluation method according to claim 1, wherein an inappropriate flag is set for a test question and the subordinate test questions. 3. A test question evaluation device for evaluating a qualification test or qualification test using a computer, which uses a test problem database managed in a hierarchical structure according to the contents of the test problem, and uses the test problem. An answer analysis unit that determines the difficulty level of the test question from the answer data of the examinee in the test, and a test question in which the hierarchical structure of the test question database and the difficulty level obtained by the answer analysis unit are inconsistent And a contradiction detecting means for performing the test question evaluation device. 4. The contradiction detection means determines that there is a contradiction when a test question at a higher level of the hierarchical structure has a lower difficulty level than a test question at a lower level, and the test question at a higher level. 4. The test question evaluation device according to claim 3, further comprising means for setting an inadequate flag for the test question at the lower level. 5. An examination question evaluation program for evaluating a qualification examination or a certification examination using a computer, wherein an examination question is given from an examination question database managed in a hierarchical structure according to the contents of the examination question. Answer analysis process for determining the difficulty level of the test question from the answer data of the person, and contradiction detection for detecting a test question in which the hierarchical structure of the test question database and the difficulty level obtained in the answer analysis process are inconsistent A test question evaluation program characterized by having a process. 6. The process of detecting a contradiction includes a process of determining that there is a contradiction when a test question at a higher level of the hierarchical structure is lower in difficulty than a test question at a lower level, and the test question at a higher level. 6. The test question evaluation program according to claim 5, further comprising: and a process of setting an inappropriate flag to the test question at the lower level. 7. A storage medium for storing a test question evaluation program for evaluating a qualification test or a certification test using a computer, which is provided from a test question database managed in a hierarchical structure according to the contents of the test question. Answer analysis process for obtaining the difficulty level of the test question from the answer data of the examinee for the test question, and the test question in which the hierarchical relationship of the test question database and the difficulty level obtained in the answer analysis process are inconsistent A storage medium storing a test question evaluation program, characterized by having a contradiction detection process for detecting. 8. The contradiction detection process comprises a process of determining that there is a contradiction when a test question at a higher level of the hierarchical structure has a lower difficulty level than a test question at a lower level, and a test question at the higher level. 8. A storage medium storing a test question evaluation program according to claim 7, further comprising: and a process of setting an inappropriate flag to the test question at the lower level.