JP7393827B1 - Question optimization system, question optimization method, and question optimization program - Google Patents

Abstract

[Problem] It is possible to select questions that suppress variations in difficulty of questions and to reflect test results in question selection. [Solution] The question optimization system includes a storage unit that stores each question registered as a test question candidate and statistical information of the question and each test result based on the test implementation history, and a storage unit that stores question candidates from the storage unit. A question selection method that includes a method using a predetermined averaging method and a method using a predetermined function that can output each candidate question as a probability value according to the test operator's specifications. a question generation section that selects questions from the question pool and generates test questions; and a question generation section that calculates individual statistical information about the test results of the generated test questions, and stores in the storage section. and an analysis unit that reflects the statistical information. [Selection diagram] Figure 3

Description

The present disclosure relates to a question optimization system, a question optimization method, and a question optimization program.

Conventionally, in the field of learning, there are techniques for analyzing a user's learning situation and techniques for evaluating exam questions.

For example, there is a technique (see Patent Document 1) that estimates a student's (user's) understanding level or correct answer probability with greater accuracy than conventional M-IRT. In this technique, a weighted average using weights of the probability of correct answer based on item response theory is calculated and used to calculate the probability of a student's correct answer regarding a test item.

Furthermore, for example, there is a technique related to a test parameter estimation method (see Patent Document 2) that further reduces the estimated value error of a test subject's (user) ability value parameter (discrimination ability) in item response theory. In addition to normalizing the value of the ability value parameter, this technique moves the value on the variable axis of the distribution function while maintaining the magnitude relationship so that the value distribution approaches the normal distribution. There is.

Japanese Patent Application Publication No. 2016-109981 Japanese Patent Application Publication No. 2011-257676

Here, the purpose of the conventional technology is to improve the accuracy of estimating the correct answer probability and ability value parameters for the user. On the other hand, in recent examinations, tests using computers called CBT (Computer Based Testing) tests have come to be implemented. The CBT exam tends to be conducted more frequently because it can adopt a format that allows examinees to take the exam on any date they choose. On the other hand, the number of questions that can be prepared is limited because creating questions requires a great deal of cost.

While the frequency of exams is increasing in this way, there is a risk that the difficulty level of questions may vary from exam to exam. If there is variation in the difficulty of questions from exam to exam, it will lead to inequality among test takers and affect the reliability and quality of the exam. Therefore, there is a need for a method for selecting questions in exams that can suppress variations in the difficulty level of questions asked. Test administrators also need a system that can generate test questions using this method and reflect the test results from the generated test questions in question selection.

The present disclosure has been made in view of the above circumstances, and provides a question optimization system, a question optimization method, and a question optimization method that make it possible to select questions that suppress variations in difficulty of questions and reflect test results in question selection. The purpose is to provide a question optimization program.

The question optimization system of the present disclosure includes a storage unit that stores each question registered as a candidate for a test question and statistical information of the question and each test result based on the test implementation history, and a storage unit that stores statistical information of the question and each test result based on the test implementation history; A question pool containing each question is obtained, and according to the test administrator's specifications, question selection methods are used, including a method using a predetermined average method and a method using a predetermined function that can output each candidate question as a probability value. A question generation unit that selects questions from the question pool and generates test questions; and a question generation unit that selects questions from the question pool and generates test questions; and and an analysis section that reflects the statistical information. With this configuration, it is possible to select questions that suppress variations in the difficulty level of the questions to be asked, and to reflect test results in the question selection.

Further, in the question optimization system, when the method using the predetermined averaging method is used as the problem selection method, the problem generation section includes an acquisition section, a first judgment section, a random setting section, and a calculation section. , a question setting section and a second determining section, the acquiring section acquiring the question pool and the number N of questions to be asked, and the first determining section determining the average difficulty level of the question pool. It is determined whether or not a predetermined random selection condition that satisfies a tolerance is satisfied, and when the random selection condition is satisfied, the random setting section empties the set and randomly selects a question from the question pool; the calculation unit calculates an acquisition target difficulty level based on the current number of questions in the set, an average difficulty level of the current questions in the set, and a target difficulty level when the random selection condition is not satisfied; A predetermined number of questions corresponding to the acquisition target difficulty level are selected from the question pool as selection candidates, a question is selected from the selection candidates, and the question setting section adds the selected questions to the set and selects the questions. The second determining section selects the questions of the number N of questions for processing by the first determining section, the random setting section, the calculating section, and the question setting section. You can repeat the process until. By using such an averaging method, it is possible to achieve optimal problem selection that suppresses variations in the difficulty level of the questions to be asked.

Further, in the question optimization system, when the method using the predetermined function is used as the problem selection method, the problem generation section includes an acquisition section, a quantile calculation section, an item information amount calculation section, and a quantile calculation section. The acquisition unit includes a place selection unit, a conversion unit, a question setting unit, and a determination unit, and the acquisition unit acquires a question pool including each candidate question and the number N of questions to be asked. The quantile calculation unit calculates each quantile as a quantile list for a predetermined probability distribution set in advance for the question, and the item information calculation unit calculates each quantile for each of the quantiles. , calculates the amount of item information for each question in the question pool at the quantile, and the quantile selection unit selects the quantile according to a predetermined order based on the center of the quantile list. The conversion unit inputs a value based on the amount of item information of each question at the selected quantile into a predetermined function that can output each input question as a probability value, and calculates the probability value. the question setting unit selects a question according to the converted probability value and excludes the selected question from the question pool, and the determining unit selects the question according to the converted probability value, and the determination unit The processing of the section and the question setting section can be repeated until the number N of questions is selected. By selecting questions with a high amount of item information at each quantile and conducting the test, it is possible to suppress variations in difficulty from test to test and improve the quality of the test. can.

In addition, the question optimization method of the present disclosure selects each question that is a candidate for a test from a storage unit that includes each question registered as a candidate for a test question and statistical information of the questions and each test result based on the test implementation history. Depending on the test operator's specifications, either a method using a predetermined average method or a method using a predetermined function that can output each candidate question as a probability value is selected. is used to select questions from the question pool, generate test questions, calculate individual statistical information about the test results of the generated test questions, and reflect it in the statistical information in the storage unit. have the computer execute it.

In addition, the question optimization program of the present disclosure calculates each question that is a candidate for an exam from a storage unit that includes each question that is registered as a candidate for an exam, as well as statistical information on the questions and each exam result based on the exam implementation history. Depending on the test operator's specifications, either a method using a predetermined average method or a method using a predetermined function that can output each candidate question as a probability value is selected. is used to select questions from the question pool, generate test questions, calculate individual statistical information about the test results of the generated test questions, and reflect it in the statistical information in the storage unit. have the computer execute it.

According to the question optimization system, question optimization method, and question optimization program of the present disclosure, it is possible to select questions with a high amount of item information and to suppress variations in difficulty of the questions to be asked. This effect can be obtained.

FIG. 1 is a diagram showing an overview of the optimization cycle of the question optimization system. FIG. 2 is a diagram showing the hardware configuration of the question optimization system of this embodiment. FIG. 3 is a block diagram showing the configuration of the question optimization system of this embodiment. FIG. 4 is a flowchart showing the flow of question optimization processing by the question optimization system. FIG. 5 shows details of the configuration of the problem generation section of the first embodiment. FIG. 6 is a flowchart showing the flow of problem generation processing according to the first embodiment. FIG. 7 shows details of the configuration of the problem generation section of the second embodiment. FIG. 8 is a diagram showing an example of calculating quantiles using a normal distribution. FIG. 9 is a graph showing the probability of correct answer p _j (θ) with respect to the ability value θ for each item j. FIG. 10 is a graph showing the amount of item information I _j (θ) with respect to the ability value θ for each item j. FIG. 11 is a flowchart showing the flow of problem generation processing according to the second embodiment. FIG. 12 is a graph verifying the distribution of the average and standard deviation. FIG. 13 is a graph verifying the distribution of the average and skewness. FIG. 14 is a graph verifying the distribution of the mean and kurtosis. FIG. 15 is a graph examining the frequency with which each question was selected. FIG. 16 is a graph verifying the amount of test information and the expected test score. FIG. 17 is a graph showing the amount of item information with respect to the ability value θ of all questions that can be selected in the verification.

An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In addition, the same reference numerals are given to the same or equivalent components and parts in each drawing. Furthermore, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

An overview of each embodiment of the present disclosure will be described (hereinafter, since the details of each embodiment are the same, they will simply be referred to as the present embodiment). The method of this embodiment relates to a method of optimizing question questions that mainly involves conducting a test as a premise. Question optimization will be realized below by reflecting the question selection method according to the first embodiment and the second embodiment and the analysis of the test results of the test conducted using the test questions generated from the selected questions. The test is mainly assumed to be, for example, a test using CBT, but it is also applicable to methods other than CBT.

FIG. 1 is a diagram showing an overview of the optimization cycle of the question optimization system. The test administrator registers each candidate question (question candidate) in the question optimization system. The question optimization system selects questions using a predetermined question selection method from statistical information and generates test questions. The test administrator conducts the test using the generated test questions. The question optimization system analyzes test takers' test results and reflects them in statistical information. The question optimization system accumulates the latest statistical information that reflects the individual statistical information of test results, and uses it to generate test questions for subsequent tests. Through this cycle of question asking and analysis, the test questions generated by the question optimization system are optimized to suppress variations in the difficulty level of the questions. Test administrators can conduct tests with optimized test questions.

The hardware configuration of the question optimization system 100 of this embodiment, which is common to each embodiment, will be described.

FIG. 2 is a diagram showing the hardware configuration of the question optimization system 100 of this embodiment. As shown in FIG. 1, the question optimization system 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication unit. It has an interface (I/F) 17. Each configuration is communicably connected to each other via a bus 19.

The CPU 11 is a central processing unit that executes various programs and controls various parts. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above components and performs various arithmetic operations according to programs stored in the ROM 12 or the storage 14. In this embodiment, the ROM 12 or the storage 14 stores a question optimization program.

The ROM 12 stores various programs and data. The RAM 13 temporarily stores programs or data as a work area. The storage 14 is constituted by a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display section 16 may employ a touch panel system and function as the input section 15.

The communication interface 17 is an interface for communicating with other devices such as terminals. For this communication, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used.

Each functional configuration of the question optimization system 100 will be explained. FIG. 3 is a block diagram showing the configuration of the question optimization system 100 of this embodiment. Each functional configuration is realized by the CPU 11 reading out a question optimization program stored in the ROM 12 or the storage 14, loading it into the RAM 13, and executing it.

As shown in FIG. 3, the question optimization system 100 functionally includes a storage section 102, a question generation section 110, and an analysis section 112.

Note that, as will be explained in the first embodiment and the second embodiment, which will be described later, the internal configuration and processing details of the question generation unit 110 are different.

Common to the question selection methods of each embodiment, each question selected as an exam question is selected from a question pool consisting of questions prepared for each field by the exam administrator who administers the exam. Target questions selected from the question pool are referred to as "questions," and questions that appear on general exams that are not related to processing are simply referred to as "questions." By selecting questions and generating test questions using the question selection method of each embodiment, question optimization is realized that allows test administrators to select desired questions.

The storage unit 102 stores questions registered as test question candidates by the test administrator, statistical information, and various information necessary for processing by the question generation unit 110 and analysis unit 112, which will be described later. The statistical information is information about questions and each test result, and has information about each question and each test result. For each question, a predetermined pool of questions to be asked is determined in advance by the test administrator. Regarding the statistical information, individual statistical information of the test results is calculated in the analysis section described later. In addition, in the question optimization system 100, by storing each question and statistical information of each question in the storage unit 102 based on the test implementation history in this way, the storage unit 102 can be used as an item bank that can treat questions as items. Function. Note that the test administrator can add and register questions in the storage unit 102.

Explain statistical information. Information regarding each question includes the correct answer rate, point biserial correlation coefficient, discrimination index, a value which is discrimination power in item response theory, and b value which is difficulty level. When item response theory is not used, the point biserial correlation coefficient and the discrimination index correspond to the discrimination power (a value), and the correct answer rate corresponds to the degree of difficulty (b value). In this way, each question has a predetermined discrimination power (a value) and difficulty level (b value) set in advance.

Information regarding each test result includes reliability coefficient, standard deviation, mean score, median, maximum value, minimum value, and test taker score distribution. Note that these statistics are just examples, and statistics that can be obtained from each question and each test result can be used.

The question generation unit 110 acquires a question pool containing each candidate question from the storage unit 102, and selects questions from the question pool using a predetermined question selection method in accordance with the specifications of the test administrator. , generate exam questions. The predetermined problem selection method is a method using either a method using a predetermined average method described in the first embodiment or a method using a predetermined function described in the second embodiment. The predetermined averaging method is a method that considers the average of the a value and the b value, and the predetermined function is a function that can output each candidate question as a probability value. Details will be described later in each embodiment.

The analysis unit 112 calculates individual statistical information regarding the test results of the test questions generated by the question generation unit 110, and reflects the statistical information in the storage unit 102. Regarding the information regarding each question among the statistical information, it is sufficient to reflect the individual elements of the information regarding each question described above regarding the questions included in the test results. Regarding information regarding test results among the statistical information, individual elements of the above-mentioned information regarding test results may be reflected as information on the test results.

Note that the test operator can refer to the statistical information in the storage unit 102 to check each numerical value of the statistical information regarding the analysis result of the analysis unit 112. Further, in the question optimization system 100, an application may be introduced that allows statistical information to be confirmed in a graph display on the display unit 16.

(Processing flow)
Next, the operation of the question optimization system 100 as a question optimization method will be explained. FIG. 4 is a flowchart showing the flow of question optimization processing by the question optimization system 100. The CPU 11 reads the question optimization program from the ROM 12 or the storage 14, expands it to the RAM 13, and executes it, thereby performing the question optimization process. The CPU 11 functions as each part of the question optimization system 100 to execute the following processes.

In step S100, the CPU 11 acquires a question pool containing each candidate question from the storage unit 102, and selects a question from the question pool using a predetermined question selection method according to the designation of the test administrator. and generate test questions.

In step S102, the CPU 11 obtains the test results of the test questions generated by the question generation unit 110. Note that since it is assumed that there is a certain period of time, such as several days to several weeks, from generation of test questions to implementation of the test, steps S100 and S102 are performed with a certain period of time in between.

In step S<b>104 , the CPU 11 calculates individual statistical information regarding the test results of the test questions, and reflects the calculated information in the statistical information in the storage unit 102 .

As described above, the question optimization system 100 updates the statistical information of the test results and uses the updated statistical information to generate the next test question, thereby optimizing the questions.

(First embodiment)
A first embodiment will be described. The first embodiment is a method in which a predetermined averaging method is used as a problem selection method by the problem generating section 111.

FIG. 5 shows details of the problem generation unit 110 of the first embodiment. The question generation unit 110 of the first embodiment includes an acquisition unit 210, a first determination unit 212, a random setting unit 214, a calculation unit 216, a question setting unit 218, and a second determination unit 220. be done.

The acquisition unit 210 acquires the question pool and the number N of questions to be asked from the storage unit 102. The number of questions N may be obtained by receiving input from the user, or may be set in advance.

Here, the symbols used in the processing by the problem generation unit 110 of the first embodiment are defined as follows. Note that the set is a reference destination to which questions are added, and is emptied by the random setting unit 214 at the start and during repetition of the process.
Tb: Target difficulty level Ta: Target discrimination power SC: Current number of questions in the set SAb: Average difficulty level of the current questions in the set SAa: Average discrimination power of the current questions in the set L: Maximum number of selection candidates (setting item)
ε: Tolerance of SAb and Tb (setting item)

The first determination unit 212 determines whether a predetermined random selection condition is satisfied. In the present embodiment, the predetermined random selection condition is that "the question is the first question" or "the question is not the final question and |SAb-Tb|≦ε is satisfied." In other words, in the process, the random selection condition is satisfied if the average difficulty level (b value) within the set satisfies the allowable error. If the allowable error is not satisfied, the question is selected by calculation by the calculation unit 216.

The random setting unit 214 empties the set and randomly selects a question from the question pool when the random selection condition is satisfied.

The calculation unit 216 performs calculations using discrimination power (a value) and difficulty level (b value) when the random selection condition is not satisfied. Note that the configuration may be such that the calculation using the discriminative power (a value) is omitted by setting whether or not the discriminating power is specified and setting the discriminating power to no specification. The calculation using the difficulty level (b value) is the calculation of the acquisition target difficulty level and the selection using selection candidates.

First, calculation using discrimination power (a value) will be explained. Before calculating using the difficulty level (b value), the calculation unit 216 calculates the acquired target discrimination power based on the current number of questions in the set, the average discrimination power of the current questions in the set, and the target discrimination power. do. The acquisition target discrimination power is calculated, for example, by "acquisition target discrimination power=(SC+1)×Ta−SC×SAa."

Next, the calculation unit 216 creates a question pool in which selectable questions are limited to questions whose discrimination power is equal to or greater than the value of the acquired target discrimination power. The calculation unit 216 uses a limited question pool in the next calculation using the difficulty level (b value).

Next, calculation using the difficulty level (b value) will be explained. The calculation unit 216 calculates the acquisition target difficulty level based on the current number of questions in the set, the average difficulty level of the current questions in the set, and the target difficulty level. The acquisition target difficulty level is calculated by, for example, "acquisition target difficulty level = (SC+1) x Tb - SC x SAb".

The calculation unit 216 selects questions with difficulty levels close to the acquisition target difficulty level from the question pool limited above as selection candidates, and selects questions from the selection candidates. Selection candidates may be selected from a pool of questions limited to a maximum of L questions in order of difficulty to obtain the target. A threshold value may be set for the proximity to the acquisition target difficulty level. This method of selecting selection candidates is an example of selecting a predetermined number of questions according to the acquisition target difficulty level of the present disclosure as selection candidates. Questions to be selected from the selection candidates may be automatically selected from the selection candidates by setting conditions for identification and difficulty level, or may be selected at random, or questions may be manually selected from the selection candidates.

The question setting unit 218 adds the selected question to the set and removes the selected question from the question pool.

The second determination unit 220 repeats the processing of the first determination unit 212, the random setting unit 214, the calculation unit 216, and the question setting unit 218 until N questions are selected.

Next, the operation of the problem generation process according to the first embodiment will be explained. FIG. 6 is a flowchart showing the flow of problem generation processing according to the first embodiment.

In step S200, the CPU 11 acquires the question pool and the number N of test questions to be asked from the storage unit 102.

In step S202, the CPU 11 determines whether a predetermined random selection condition is satisfied. If the random selection conditions are met, the process moves to step S204, and if the random selection conditions are not satisfied, the process moves to step S28.

In step S204, the CPU 11 empties the set.

In step S206, the CPU 11 randomly selects a question from the question pool. Next, the process moves to step S220.

In step S208, the CPU 11 determines whether or not the discrimination power is specified. If there is a designation, the process moves to step S210, and if there is no discrimination power designation, the process moves to step S214.

In step S210, the CPU 11 calculates the obtained target discrimination power based on the current number of questions in the set, the average discrimination power of the current questions in the set, and the target discrimination power.

In step S212, the CPU 11 creates a question pool in which selectable questions are limited to questions whose discrimination power is equal to or greater than the value of the acquired target discrimination power.

In step S214, the CPU 11 calculates the acquisition target difficulty level based on the current number of questions in the set, the average difficulty level of the current questions in the set, and the target difficulty level.

In step S216, the CPU 11 selects up to L questions from the question pool with difficulty levels close to the acquisition target difficulty level as selection candidates. Note that if step S212 has been passed, a limited question pool is used.

In step S218, the CPU 11 selects one question from the selection candidates.

In step S220, the CPU 11 adds the selected question to the set.

In step S222, the CPU 11 excludes the selected question from the question pool.

In step S224, the CPU 11 determines whether the number of selected questions is equal to or greater than the number of questions N (number of selected questions≧N). If the number is N or more, the process ends, and if it is not N or more, the process returns to step S202 and the next process is repeated.

As described above, the question optimization system 100 according to the first embodiment makes it possible to select questions that suppress variation in difficulty of questions to be asked and to reflect test results in question selection.

(Second embodiment)
A second embodiment will be described. The second embodiment is a method in which a predetermined function is used in the problem selection method of the problem generating section 111. As explained in the above-mentioned subject, in recent years, as tests are being conducted more frequently, it is necessary to maintain test quality by conducting tests with a high amount of item information. Therefore, in this embodiment, in order to stably realize question selection with a higher amount of item information, we use a method of selecting questions by converting the amount of item information of each question into a probability value using a softmax function. Use. A softmax function is an example of a predetermined function of this disclosure.

An overview of the processing flow of the method of this embodiment will be explained. Assuming that the number of questions to be asked is N, questions for the exam questions are selected by performing the following steps (1) to (6). Details of each process will be described later.
(1): Find the N+1 quantile from a normal distribution set in advance.
(2): Calculate the amount of item information at each quantile obtained in (1) for each question in the question pool.
(3): Select in order from the center of the quantile list obtained in (1).
(4): The amount of item information for each question at the quantile selected in (3) is input to the softmax function and converted into a probability value.
(5): Select questions with the probability obtained in (4), and exclude the selected questions from the question pool.
(6): Repeat (3) to (5) N times.

FIG. 7 shows details of the configuration of the problem generation section 110 of the second embodiment. The problem generation unit 110 of the second embodiment includes an acquisition unit 310, a quantile calculation unit 312, an item information amount calculation unit 314, a quantile selection unit 316, a conversion unit 318, and a question setting unit 320. , and a determining unit 322.

The acquisition unit 310 acquires the question pool and the number N of exam questions to be asked from the storage unit 102. The number of questions N may be obtained by receiving input from the user, or may be set in advance.

The quantile calculation unit 312 calculates each quantile as a quantile list for a predetermined normal distribution. The processing of the quantile calculation unit 312 corresponds to step (1) of the above procedure. The normal distribution is set by a predetermined mean and standard deviation for questions, and is expressed as a probability distribution of ability values θ. The quantile can be calculated from the cumulative total of probability density values of a probability density function (pdf). The quantile calculation unit 312 calculates each quantile from the cumulative total of probability density values. By calculating each quantile, the ability value θ corresponding to each quantile is determined, and the difficulty and discrimination questions corresponding to the ability value θ of the quantile are selected from each question in the question pool. Can be extracted.

FIG. 8 shows an example of calculating quantiles using a normal distribution. The horizontal axis shows the ability value θ, and the vertical axis shows the probability density value (pdf). The difficulty level θ is set at 0 as the average starting point, and the further to the left corresponds to questions with lower difficulty, and the further to the right corresponds to questions with higher difficulty. The quantile calculation unit 312 can calculate N quantiles by dividing the area of the portion surrounded by the probability density function of the normal distribution and the x-axis (the lower area of the normal distribution) into N+1 equal parts. Vertical lines drawn in the normal distribution indicate each quantile. The quantiles found from the probability density function become denser as they are closer to the center of the normal distribution, and sparser as they are farther away. Note that the normal distribution is an example of "a predetermined probability distribution set in advance for a question" in the present disclosure. Further, as the probability distribution, it is sufficient that quantiles can be obtained from the value of the probability density, and in addition to the normal distribution, a uniform distribution, a beta distribution, etc. may be used.

Furthermore, the parameters of the normal distribution include an average and a standard deviation, and the acquisition unit 310 can also accept settings for the average and standard deviation to set the position and shape of the normal distribution. By changing the mean, you can move the entire position of the quantile list left or right. Furthermore, by changing the standard deviation, the length (degree of density) between each quantile in the quantile list can be changed.

For each quantile calculated by the quantile calculation unit 312, the item information amount calculation unit 314 calculates the item information amount for each question in the question pool at that quantile. The processing by the item information amount calculation unit 314 corresponds to step (2) of the above procedure. Each question in the question pool is represented as item j. The item information amount I _j (θ) can be calculated according to the following equation (2), where the probability of a correct answer to item j with respect to the subject's ability p _j (θ) is expressed by the following equation (1).

...(1)

...(2)

Here, D is a constant (=1.7), aj is the discrimination power of item _j , bj is the difficulty level of item _j , and θ is the ability value. These values may be stored in the storage unit 102 as the statistical information of the questions by estimating the a value and b from the test results, etc. in advance.

FIG. 9 is a graph showing the probability of correct answer p _j (θ) with respect to the ability value θ for each item j. The horizontal axis is the ability value θ, the vertical axis is the correct answer probability p _j (θ), and the item characteristic curve (ICC) for each item j is represented. In the example of Figure 9, an item characteristic curve showing the transition of the probability of correct answer p _j (θ) with respect to the ability value θ when the discrimination ability (a value) and difficulty level (b value) are given for each of the five items j is displayed. has been done.

The item information amount is an index representing which ability value θ of the subject each item j is suitable for. The item information amount I _j (θ) takes the maximum value when θ=b _j . FIG. 10 is a graph showing the amount of item information I _j (θ) with respect to the ability value θ for each item j.

The quantile selection unit 316 selects quantiles in a predetermined order based on the center of the quantile list. The processing of the quantile selection unit 316 corresponds to step (3) of the above procedure. For selection according to the order, for example, the quantiles may be selected in the order of their proximity to the center of the normal distribution, that is, the peak of the probability density value. Note that the predetermined number of quantiles can be arbitrarily selected from among a predetermined number of quantiles in order of closestness, and after completing the selection of the predetermined number, the next predetermined number of quantiles in order of closestness are selected. The selection may be made according to the order.

The conversion unit 318 inputs the item information amount of each question at the selected quantile to a softmax function that can output each input question as a probability value, and converts it into a probability value. The processing of the conversion unit 318 corresponds to step (4) of the above procedure. The output y _i of the softmax function is, when a set of n real numbers is input, it is converted into a probability value while maintaining the magnitude relationship of the input value (item information amount of each question), and the output is a set of n real numbers that become the probability value. This is a function that outputs a tuple. The softmax function converts the input real number into a probability value such that the larger the input real number is compared to other real numbers, the greater the probability value of the corresponding output real number. Conversion using the softmax function follows equation (3) below.

...(3)

y _i is an output value (probability value) output for number i. e is the Napier number, x _i is the input value of number i, and x _k is the input value of number k. When a set of n real numbers (x ₁ , . . . , x _n ) is input to the softmax function, a set of n real numbers (y ₁ , . . . , y _n ) is output. Since each y _i is between 0 and 1 and the sum of y ₁ , . . . , y _n is 1, y _i can be interpreted as a probability value. The larger the value of x _i is compared to other x, the larger the value of y _i becomes. For example, if a set of three real numbers (1, 2, 3) is given as an input value, the output will be (0.09003057, 0.24472847, 0.66524096), etc.

Note that the item information amount is an example of a value based on the item information amount of the present disclosure. In addition to the item information amount, the amount added to the item information amount can also be determined using the difference between the ability value θ of the quantile regarding the question and the b value that is the difficulty level. A value based on the amount of item information to be input to the softmax function can be designed as appropriate by adding, subtracting, or weighting an additional amount to the amount of item information. Note that the predetermined function is not limited to the softmax function, but any function that can input the value of each question and output it as a probability value can be used. For example, each question may be designed to be output as a probability value using a softmin function that increases the probability as the input value is lower, or may be used in combination with a softmax function.

The question setting section 320 selects a question according to the probability value converted by the conversion section 318, and excludes the selected question from the question pool. The process of the question setting unit 320 corresponds to step (5) of the above procedure. The question setting unit 320 may select questions based on the probability value of each question. In addition, there are restrictions on the number of questions selected from the question pool for each field. Therefore, for each field, if the number of questions selected in that field reaches a predetermined number, the question setting unit 320 excludes that field from selection targets in subsequent repeated selections.

The determination unit 322 repeats the processes of the quantile selection unit 316, the conversion unit 318, and the question setting unit 320 until N questions are selected. The process of the determination unit 322 corresponds to step (6) of the above procedure.

Next, the operation of the problem generation process according to the second embodiment will be explained. FIG. 11 is a flowchart showing the flow of problem generation processing according to the second embodiment.

In step S300, the CPU 11 acquires the question pool and the number N of test questions to be asked from the storage unit 102.

In step S302, the CPU 11 calculates each quantile as a quantile list for a predetermined normal distribution.

In step S304, for each of the quantiles calculated in step S302, the CPU 11 calculates the amount of item information for each question in the question pool at that quantile, according to equation (2) above.

In step S306, the CPU 11 selects quantiles in a predetermined order based on the center of the quantile list.

In step S308, the CPU 11 inputs the item information amount of each question at the selected quantile into a softmax function that can output each input question as a probability value, and calculates the probability value according to the above equation (3). Convert to

In step S310, the CPU 11 selects a question according to the probability value converted in step S308, and excludes the selected question from the question pool.

In step S312, the CPU 11 determines whether the number of selected questions is equal to or greater than the number of questions N (number of selected questions≧N). If it is N or more, the process ends, and if it is not N or more, the process returns to step S306, selects the next quantile, and repeats the process.

As explained above, the question optimization system 100 according to the second embodiment selects questions with a high amount of item information, suppresses variations in difficulty of questions to be asked, and selects questions in the test results. enable reflection.

[Verification example]
Verification was performed when problem selection was performed using the problem selection methods of the first and second embodiments. The selection methods for comparison verification were (A) random [random], (B) average [avg], and (C) function [softmax]. (B) Average is the method of the first embodiment in which questions are selected so as to approach the target average. The function (C) is the method of the second embodiment using softmax. The settings were verified as 200 total questions, 50 questions, and 200 simulations. The effectiveness of the (C) function method of the second embodiment will be mainly described below.

FIGS. 12 to 14 show the distribution of a values and b values when questions are selected using each verification method. In both figures, the upper graph is the distribution of a values, and the lower graph is the distribution of b values. Note that avg in the legend is the average method in (B), and avg on the horizontal axis of the graph indicates the average of the a value or b value of the selected question. Standard deviation (std), skewness (skew), and kurtosis (kurtosis) are each compared to the average (avg). FIG. 12 is a graph verifying the distribution of the average and standard deviation. FIG. 13 is a graph verifying the average and skewness distribution. FIG. 14 is a graph verifying the distribution of the mean and kurtosis. In both graphs, it was confirmed that the b values were aggregated into the average in (B) average. In addition, compared to other methods, the (C) function method (B) does not concentrate too much on the average but is appropriately aggregated toward the average, and (A) has less variation than random and is appropriately aggregated. This was confirmed.

FIG. 15 is a graph examining the frequency with which each question was selected. The graph represents the frequency with which each question was selected in the simulation. (C) It was confirmed that the function method selected more appropriate questions and a wider variety of questions than other methods.

FIG. 16 is a graph verifying the amount of test information and the expected test score. The test information amount is the total amount of item information of the selected questions. T(θ) represents the expected test score of the subject with the ability value θ. In both tests, the more the curves are aggregated, the less likely there will be a difference for test takers. It was confirmed that the test information and score information T(θ) were more concentrated in the (C) function method than in other methods. It was confirmed that the (B) average method was also more aggregated than the (A) random method. FIG. 17 is a graph showing the amount of item information with respect to the ability value θ of all questions that can be selected in the verification.

Note that the present invention is not limited to the embodiments described above, and various modifications and applications can be made without departing from the gist of the present invention.

Furthermore, although the present specification has been described as an embodiment in which a program is installed in advance, it is also possible to provide the program by storing it in a computer-readable recording medium.

100 Question optimization system 102 Storage unit 110 Question generation unit 112 Analysis units 210, 310 Acquisition unit 212 First judgment unit 214 Random setting unit 216 Calculation unit 218 Question setting unit 220 Second judgment unit 312 Quantile calculation unit 314 Item information Quantity calculation section 316 Quantile selection section 318 Conversion section 320 Question setting section 322 Judgment section

Claims (7)

a storage unit that stores statistical information of each question registered as a candidate for an exam question and the questions and each exam result based on the exam implementation history;
A question pool containing each candidate question is acquired from the storage unit, and a question selection method using a predetermined function that can output each candidate question as a probability value according to the specifications of the test administrator is implemented. a question generation unit that selects questions from the question pool and generates test questions using the question pool;
an analysis unit that calculates individual statistical information regarding the test results of the generated test questions and reflects the statistical information in the storage unit;
including;
In generating the test questions using the question selection method,
The question generation unit includes an acquisition unit, a quantile calculation unit, an item information amount calculation unit, a quantile selection unit, a conversion unit, a question setting unit, and a determination unit,
The acquisition unit acquires a question pool including each question to be asked and the number N of questions to be asked;
The quantile calculation unit calculates a quantile list based on the probability distribution of the ability value, which is a predetermined probability distribution set in advance for the question, and the number of questions N. Calculate each,
For each of the quantiles, an ability value indicating the horizontal axis of the corresponding probability distribution is determined,
The item information amount calculation unit calculates, for a combination of each of the quantiles and each question of the question pool, based on the discrimination ability and difficulty level of the question as an item, and the ability value of the quantile. Calculate the amount of item information,
The quantile selection unit selects the quantile points according to a predetermined order based on the center of the quantile list,
For the selected quantile, the conversion unit inputs the value of the amount of item information of each question at the quantile into a predetermined function that can output the input value as a probability value, and converts it into a probability value. ,
The question setting unit selects a question according to a probability value calculated from the item information amount of each question at the selected quantile, and excludes the selected question from the question pool;
The determination unit repeats the processing of the quantile selection unit, the conversion unit, and the question setting unit until the number N of questions is selected.
Question optimization system. The predetermined function is a function that, when inputting a set of n real numbers, outputs a set of n real numbers that is a probability,
While maintaining the size relationship of the item information amount of each question, each real number is converted to the above probability value so that the larger the input real number is compared to other real numbers, the greater the probability value of the corresponding output real number. The question optimization system according to claim 1, which performs conversion. 2. The question optimization system according to claim 1, wherein the acquisition unit receives settings of an average and standard deviation for the question, as well as the question pool, and sets the probability distribution. The questions selected from the question pool have restrictions on the number of questions for each field,
2. The question setting unit, for each field, when the number of questions selected in the field reaches a predetermined number, excludes the field from selection targets in subsequent repeated selections. Question optimization system. The statistical information includes information regarding each question and information regarding each test result in the test implementation history,
The information regarding each question includes at least one of a correct answer rate, a point biserial correlation coefficient, a discrimination index, an a value which is a discrimination power in item response theory, and a b value which is a difficulty level,
The question optimization system according to claim 1, wherein the information regarding each test result includes at least one of a reliability coefficient, a standard deviation, an average score, and a score distribution of test takers. A question pool containing each question that is a candidate for questions to be asked is acquired from a storage unit that includes each question that has been registered as a candidate for an exam, as well as statistical information on the questions and each test result based on the exam implementation history, and According to the specification, select questions from the question pool using a question selection method using a predetermined function that can output each candidate question as a probability value, and generate test questions;
Calculating the individual statistical information regarding the test results of the generated test questions and reflecting it in the statistical information in the storage unit,
In generating the test questions using the question selection method,
Obtain a question pool containing each candidate question and the number N of questions to be asked,
A predetermined probability distribution set in advance for questions, regarding the probability distribution of ability values, each of the quantiles is calculated as a quantile list from the probability distribution and the number of questions N,
For each of the quantiles, an ability value indicating the horizontal axis of the corresponding probability distribution is determined,
For a combination of each of the quantiles and each question of the question pool, calculate the amount of item information based on the discrimination power and difficulty level as an item of the question and the ability value of the quantile,
selecting the quantile points according to a predetermined order based on the center of the quantile list;
For the selected quantile, input the value of the amount of item information for each question at the quantile into a predetermined function that can output the input value as a probability value, converting it into a probability value,
selecting a question according to a probability value calculated from the item information amount of each question at the selected quantile, and excluding the selected question from the question pool;
repeating the selection of the quantile, the conversion, and the selection and exclusion of questions until the number N of questions is selected;
A question optimization method that allows the computer to perform the processing. A question pool containing each question that is a candidate for questions to be asked is acquired from a storage unit that includes each question that has been registered as a candidate for an exam, as well as statistical information on the questions and each test result based on the exam implementation history, and According to the specification, select questions from the question pool using a question selection method using a predetermined function that can output each candidate question as a probability value, and generate test questions;
Calculating the individual statistical information regarding the test results of the generated test questions and reflecting it in the statistical information in the storage unit,
In generating the test questions using the question selection method,
Obtain a question pool containing each candidate question and the number N of questions to be asked,
A predetermined probability distribution set in advance for questions, regarding the probability distribution of ability values, each of the quantiles is calculated as a quantile list from the probability distribution and the number of questions N,
For each of the quantiles, an ability value indicating the horizontal axis of the corresponding probability distribution is determined,
For a combination of each of the quantiles and each question of the question pool, calculate the amount of item information based on the discrimination power and difficulty level as an item of the question and the ability value of the quantile,
selecting the quantile points according to a predetermined order based on the center of the quantile list;
For the selected quantile, input the value of the item information amount of each question at the quantile into a predetermined function that can output the input value as a probability value for each question, converting it into a probability value,
selecting a question according to a probability value calculated from the item information amount of each question at the selected quantile, and excluding the selected question from the question pool;
repeating the selection of the quantile, the conversion, and the selection and exclusion of questions until the number N of questions is selected;
A question optimization program that allows the computer to perform the processing.