KR101370735B1 - Computerized adaptive testing system and method using bayesian networks - Google Patents

Abstract

Disclosed are a computer-based adaptive evaluation system and method using a Bayesian network. The computerized adaptive testing (CAT) system is structured by clustering an evaluation item database in which a plurality of evaluation items for estimation of learning are divided into a plurality of detailed learning areas and stored in the evaluation item database. By doing so, an association information generation unit generating association information between the detailed learning area of each subject to be evaluated and each evaluation item, a Bayesian network unit generating a Bayesian network based on the generated association information, and the generated Bayesian network A learning degree estimator for estimating the degree of learning of the user with respect to the evaluation item and the total degree of learning and the degree of learning of the user for each detailed learning area may be included.

Description

Computer-based Adaptive Evaluation System and Method using Bayesian Networks {COMPUTERIZED ADAPTIVE TESTING SYSTEM AND METHOD USING BAYESIAN NETWORKS}

Embodiments of the present invention relate to a computer-based adaptive evaluation system and method using a Bayesian network that performs Computerized Adaptive Testing (CAT) using a Bayesian network.

Various assessment methods have been developed to measure the student's current level for customized learning according to the learner's level. Most schools, however, now apply the traditional paper test to their evaluation methods.

The problem with the paper-based exam is that the number of questions inevitably increases in order to measure the level of learners of various types, and as the number of questions increases, the test time becomes longer, thereby increasing the burden on the learners. In addition, it is difficult to create individualized questions for the learner's type, so the contents of the test are easily composed of the items including most of the hierarchies, which is a waste of time by requiring learners who do not belong to the type to respond to unnecessary questions. And lowering the motivation of the learner.

In order to solve this problem, computer-based tests began to receive attention as one of the solutions, and students' cognitive processes and computer cognitive processes were developed through computerized adaptive test (CAT), which combines the computer's computing power and test theory. Research is underway to determine the condition.

In the CAT, the following questions are selected and answered according to the response result of a given question, so that the candidate's ability can be estimated more precisely. In addition, the CAT can evaluate the subject's ability more accurately with fewer questions than the paper-based test, and it is very economically advantageous because the test is carried out and the result is confirmed at the same time as the test.

Korean Patent Application Publication No. 10-2001-0103810 (published November 24, 2001) includes a computer-based educational evaluation system and its operation method. Based on the information, a questionnaire corresponding to the test list is prepared for each subject, and the test sheet is composed according to the test method selected by the test method among paper test, computer based test (CBT), and CAT. Based on the test analysis program, providing and storing the results of item analysis and subject capability estimation to the subject are described.

However, it is difficult to refine the characteristics of learners of various types with such evaluation methods. CAT, like the paper test score, defines the learner's characteristics with a single parameter, the ability parameter. The method of classifying learner's grade based on only one number is the ability for education to consider the characteristics of each learner. There is a lack of evaluation. In addition, learners' responses to the questions are assumed to be independent of each other, so that if they are not in the real world, that is, if there are highly related questions, they should submit a similar item to the question that was answered without reflecting the result of the already answered question. It is not efficient to quickly measure the level of learners.

Therefore, in the composition of items, CAT simply recommends the problem of difficulty that suits the learner's ability by using only item response theory (IRT). It cannot be evaluated and it is difficult to estimate learner's abilities quickly and accurately.

In addition, since the existing IRT-based CAT evaluates only the overall ability of the learner, there is a problem in that it cannot effectively estimate the learner's ability in each detailed learning area.

A computer-based adaptive evaluation system and method using Bayesian network that can estimate learner's ability quickly and accurately in consideration of the correlation between questions is provided.

A computer-based adaptive evaluation system and method using a Bayesian network that can accurately evaluate a learner's learning status in a short time with a small number of questions is provided.

The computerized adaptive testing (CAT) system includes an evaluation item database in which a plurality of evaluation items for estimation of learning are divided into a plurality of detailed learning areas, and clustered evaluation items stored in the evaluation item database. An association information generation unit for generating a detailed learning region of a subject to be evaluated and an association information between each assessment item by structuring the relationship between the learning region and each item in a Bayesian network topology, and a Bayesian network based on the generated association information. A learning degree estimator for estimating a learning degree of the user for each evaluation item and each detailed learning area by using a Bayesian network unit for generating a) and an association relationship between the generated Bayesian network and items, wherein the association information generation unit includes: Above three Structuring the learning area and the test items in the form of a tree, and related items associated with the specific, it can be reflected in the response result each time the test items The response by the user.

According to one side, each node of the Bayesian network using an evaluation-maximization (EM) algorithm based on the evaluation result database that stores the evaluation results of past users of the evaluation item and the evaluation results stored in the evaluation result database And a CPT generator for generating a conditional probability table for each probability calculation, wherein the Bayesian network unit generates the Bayesian network using the association information and the conditional probability table, and the EM algorithm utilizes the evaluation result. We can estimate the hidden variable.

According to another aspect, the method may further include an evaluation item recommending unit for recommending an item suitable for the user among the evaluation items based on the estimated degree of learning.

The computerized adaptive testing (CAT) method comprises clustering a plurality of evaluation items divided into a plurality of detailed learning areas to structure a relationship between the learning areas and the items in a Bayesian network topology, and the structured detail. Generating association information between a learning area and each assessment item; generating a Bayesian network based on the generated association information; and using the generated Bayesian network and item-specific associations; Estimating the degree of learning of the user for each sublearning area, wherein the structuring includes: structuring the sublearning area and the evaluation item in the form of a tree, and the relation of each item is evaluated by the user. Whenever this response is received, the response result may be reflected.

Instead of the existing Item Response Theory (IRT), we can estimate learner's ability quickly and accurately by using the Bayesian network which considers the relationship between items.

Since the association between questions is structured through the Bayesian network, it is possible to accurately assess the learner's learning status in a short time with a small number of questions, and to estimate not only the overall learner's ability but also the learner's ability in each detailed learning area.

1 is a block diagram illustrating a computer-based adaptive evaluation system using a Bayesian network according to an embodiment of the present invention.
FIG. 2 is a diagram for structuring detailed learning areas of subjects and respective evaluation items using a Bayesian network using associations with areas of knowledge in accordance with an embodiment of the present invention.
3 is a diagram illustrating CPT for each node of a Bayesian network according to one embodiment of the present invention.
4 to 7 are exemplary diagrams illustrating a process of evaluating a user using a computer-based adaptive evaluation system using a Bayesian network according to an embodiment of the present invention.
8 is a graph illustrating an experimental result of a computer-based adaptive evaluation system using a Bayesian network according to an embodiment of the present invention.
9 is a flowchart illustrating a computer-based adaptive evaluation method using a Bayesian network according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In the related art, only item response theory (IRT) was used for estimating the learner's ability. In other words, there is a problem that the items that are unnecessary in the actual evaluation are evaluated without considering the correlation between the items at all. In order to solve this problem, the present invention uses Bayesian network, which can calculate probabilities for all other variable values by using some observed variable values. Take advantage.

Since the association between all questions is defined through the Bayesian network, the response result of one item affects the percentage of correct answers of the other item. That is, when the examinee responds to a question, the result of the response is propagated to all nodes (question node, learning area node) connected to the Bayesian network through the Bayesian network. Through this, information on several questions and categories of each area can be obtained at the same time. For each item, it is possible to grasp the probability of fitting the item and the like, and for each category, the probability of clearly understanding the area can be estimated.

In addition, using the Bayesian network probability estimation technique, it is possible to efficiently calculate and estimate the probability of all remaining variable values using some observed variable values. That is, when the examinee responds to an item, the probability of all other variables (other item variables and learning area variables) can be calculated based on the response. As a result, the computer-based adaptive evaluation system using the Bayesian network according to the present invention performs an effective evaluation using the following characteristics of the Bayesian network.

(1) Efficiently express associations between variables through Bayesian networks

→ Through this, the relationship between the question and the knowledge area can be expressed effectively.

(2) When the values of some variables are observed in the Bayesian network, the observed values of some variables can be used to calculate probabilities for all other variables.

¡Æ when a user responds to an item, the probability of the remaining nodes is inferred based on the response

1 is a block diagram illustrating a computer-based adaptive evaluation system using a Bayesian network according to an embodiment of the present invention.

The computerized adaptive testing (CAT) system using a Bayesian network includes an evaluation item database 110, an association information generating unit 120, an evaluation result database 130, a CPT generating unit 140, and a Bayesian network unit. 150, a learning degree estimator 160, an evaluation item recommender 170, and a user interface 180 may be included.

In the evaluation item database 110, a plurality of evaluation items for estimating the degree of learning may be divided into a plurality of detailed learning areas and stored.

The association information generation unit 120 clusters the evaluation items stored in the evaluation item database 110 to structure the association between knowledge areas (learning areas) and items by Bayesian network topology. Association information between the detailed learning area and each assessment item may be generated. Through this, unlike the CAT using the conventional IRT, it is possible not only to estimate the overall learning ability, but also to estimate the learner's ability in each detailed learning area, and the correlations between the items can be considered in the actual evaluation to perform more accurate and efficient learning evaluation. . The association information generated by the association information generation unit 120 will be described in more detail with reference to FIG. 2 to be described later.

Meanwhile, the evaluation result database 130 stores training data necessary for generating a CPT, that is, actual evaluation results of past students on corresponding evaluation items.

The CPT generation unit 140 uses a conditional probability table (CPT) for calculating a probability for each node of the Bayesian network based on the evaluation-maximization (EM) algorithm based on the evaluation result stored in the evaluation result database 130. Create A process of generating the conditional probability table by the CPT generator 140 will be described in more detail with reference to FIG. 3.

Bayesian network unit 150 generates a Bayesian network. As an example, the Bayesian network unit 150 may complete the Bayesian network using the association information generated by the association information generator 120 and the conditional probability table generated by the CPT generator.

The learning degree estimator 160 estimates not only the user's learning degree for the evaluation item but also the overall learner's learning degree and the learner's learning degree for each detailed learning area using the Bayesian network generated by the Bayesian network unit 150. can do.

The evaluation item recommender 170 recommends an item suitable for the user among evaluation items stored in the evaluation item database 110 based on the learning degree estimated by the learning degree estimator 160.

The user interface 180 displays a question and an evaluation result recommended by the evaluation item recommendation unit 170 or receives a response to the evaluation item from the user and transfers the response to the learning estimator 130 to the user for the evaluation item. To estimate the degree of learning and the response is reflected in the item that is recommended through the evaluation item recommender 160.

Hereinafter, the operation of the computer-based adaptive evaluation system using the Bayesian network will be described in more detail.

FIG. 2 is a diagram for structuring detailed learning areas of subjects and respective evaluation items using a Bayesian network using associations with areas of knowledge in accordance with an embodiment of the present invention.

For example, when a detailed learning area of a subject such as a function change rate, limit, derivative, partial derivative and integral is related as shown in FIG. 2 (a), the association information generation unit of the CAT system according to the present invention The evaluation items stored in the evaluation item database are clustered and structured in a tree form using a Bayesian network, as shown in FIG. 2 (b), the detailed learning areas (function change rate, limit, derivative, partial derivative, integral) and each evaluation item ( Q1 to Q6) may generate association information.

For example, if the user is wrong about the rate of change and the limit of the function, the user is less likely to solve the differential problem. Conversely, users who match partial and integral problems are more likely to solve differential problems. If so, the user is automatically more likely to understand the derivatives.

Also, through the probability inference technique of Bayesian network, if the user responds to the second problem (observed value), the probability value of all remaining nodes can be inferred based on the result of the response. Through this, it is possible to effectively estimate the probability of fitting different items and the probability of understanding in each detailed learning area with only a few items.

When the Bayesian network is applied to CAT and used for algorithms to select the next problem according to the user's response and algorithms for evaluating the student's abilities in each area, fewer questions are generated in a shorter time than the conventional evaluation method. Not only can they quickly and accurately assess learners, but they can also estimate overall learners 'abilities and learners' abilities for each subfield.

3 is a diagram illustrating CPT for each node of a Bayesian network according to one embodiment of the present invention.

As shown, the CAT system can generate a CPT for each node by performing an EM algorithm using evaluation results of existing users and the association information of the Bayesian network shown in FIG. 2.

Although the topology of the Bayesian network is completed through the clustering process of knowledge in the association information generation unit, a CPT (Conditional Probability Table) in which conditional probability values are defined for each node of the Bayesian network may be required to perform the actual probability estimation. have. To this end, the CPT generation unit of the computer-based adaptive evaluation system using the Bayesian network according to the present invention automatically utilizes the CPT by using the evaluation results, which the users stored in the evaluation result database have taken in the past, as training data. Create

The evaluation results stored in the evaluation results database for the CPT generation of the Bayesian network are based on the results of the students' answers (correct or incorrect) for each question. At this time, the achievement of each student's detailed learning area is not given in the evaluation result. As such, a variable value that is not given to the actual evaluation result is called a hidden variable.

The CPT generator may use an EM (Expectation Maximization) algorithm to generate the CPT of the learning area node using the evaluation result. This allows the automatic generation of CPTs for each node in the Bayesian network, which is necessary for the overall evaluation.

EM algorithm is a technique for estimating hidden variables of a given model using given training data. It is used in various fields such as Bayesian network, Hidden Markov Model (HMM) and Gaussian mixture model.

The computer-based adaptive evaluation system using the Bayesian network according to the present invention requires only the correct answer data of the test that the real users took as learning data. The capability data of the remaining users in each domain is not needed and can be calculated automatically by the Bayesian network. This will automatically generate CPTs for each node needed to perform the actual evaluation, resulting in a complete Bayesian network.

4 to 7 are exemplary diagrams illustrating a process of evaluating a user using a computer-based adaptive evaluation system using a Bayesian network according to an embodiment of the present invention.

The conventional CAT system estimates the learner's ability using the IRT. Based on this, the appropriate questions are selected and students' proficiency is evaluated. However, this conventional method is not effective because it does not reflect the relations among the items properly and also it is impossible to estimate the ability values for each detailed learning area.

However, the computer-based adaptive evaluation system using the Bayesian network according to the present invention estimates the learner's abilities using the Bayesian network rather than the IRT, and recommends the best questions based on the learner's ability. Estimate the capability of each area.

Bayesian networks can estimate the probability values of all the nodes based on the observations of some nodes. In this system, the principles of Bayesian network are applied to the actual evaluation process. The evaluation process is briefly described as follows.

(1) The user responds to a given question. → observations of some variables are given

(2) Based on this, the user's stats for each detailed learning area and the probability of matching the other items are automatically calculated. ¡Æ the probability values of all remaining nodes are estimated based on the given observations

(3) Based on the estimated abilities of each item, the item information function is used to select the optimal item that matches the user's ability. ¡Æ instead of recommending items based on existing IRT-estimated stats, Bayesian networks are used to estimate stats

(4) The process of (1) ~ (3) is repeated until the evaluation of the user's stats by total and detailed area is completed.

The item information function is a function indicating how accurately the item is estimated by the user, and can be expressed as in Equation 1 below.

는 문항 i에서 사용자의 능력치 정도를 나타낸다.
here

Denotes the user's ability level in question i.

When the user responds to a given question, the probability values of all other variables (detailed learning area nodes and item nodes) are automatically calculated according to the principles of the Bayesian network.

At this time, since all items are structured in relation to each other, the response of the item affects the probability value of all other related items. This effectively solves the problem of overlooking the linkage between items that were considered to be a disadvantage of the existing IRT-based CAT. In addition, the response result for a given item is propagated to all nodes, and the probability values of the learner's detail region and all nodes of each item are automatically calculated.

Once the probability of matching each item is calculated, the system uses the item information function to select the optimal item for the user's ability. If the existing CAT evaluates the user's capability using the IRT, the computer-based adaptive evaluation system using the Bayesian network according to the present invention estimates the user's capability using the Bayesian network in all parts of user evaluation and item recommendation.

4 through 7 illustrate an example of a process of evaluating a user using a computer-based adaptive evaluation system using a Bayesian network. As shown in FIG. 4, whenever a user responds to a question presented, the CAT system according to the present invention estimates the user's ability value for each learning detail region. Since the Bayesian network can estimate the probability values of all the remaining variables even with only some given variable values, the CAT system according to the present invention utilizes the characteristics of the Bayesian network so that each time the user responds to the presented question, each learning subregion The ability of star learners can be effectively calculated. 5 is a diagram illustrating the learner's ability value for each detailed learning area. "true" indicates the probability (capability) that the learner correctly understands in each learning area. In FIG. 6, in contrast to FIG. 4, when a user incorrectly answers a given question, the user's ability value is reduced for each subregion.

The CAT system using the Bayesian network automatically terminates the evaluation when the learner's ability reaches a certain level. Through this, it is more efficient than the paper-based test that requires solving all the questions, and the Bayesian network probability inference technique can estimate the learner's ability much faster with fewer items than the conventional IRT-based CAT. In addition, the learner's ability in each detailed learning area can be estimated as well as the overall learner's ability. 7 is a diagram illustrating a state in which all learning is completed.

8 is a graph illustrating an experimental result of a computer-based adaptive evaluation system using a Bayesian network according to an embodiment of the present invention.

FIG. 8 is a graph showing the results of using the maximum likelihood estimation as a method of estimating the subject's ability parameters. Experiments were performed on items of type A (FIG. 8A) and items of type B (FIG. 8B), respectively. Results are shown. Referring to the drawings, it can be seen that the CAT system (BN: Bayesian Network) according to the present invention has fewer items required for estimating the learner's abilities than the conventional technologies (Naive, IRT (Score), IRT (Theta)). have. Through this, it can be confirmed that the CAT system using Bayesian network can effectively evaluate the learner's ability at a faster time and with fewer questions than the conventional technology.

As a result of the experiment, the correlation coefficient of this technique finally converged to 0.9 or more. This indicates that the present invention can determine the learner's ability to the same level as the paper test. Therefore, it can be confirmed that the present invention can accurately determine the learner's ability even with a smaller number of questions than other conventional evaluation methods.

9 is a flowchart illustrating a computer-based adaptive evaluation method using a Bayesian network according to an embodiment of the present invention.

A computer-based adaptive evaluation system using a Bayesian network first clusters and structures a plurality of evaluation items divided into a plurality of detailed learning areas (S910). In operation S920, correlation information between the structured detailed learning area and each evaluation item is generated. Thereafter, a conditional probability table (CPT) is generated for each node of the Bayesian network based on the evaluation result and the correlation information stored in the evaluation result database (S930). Then, using the Bayesian network completed through such a process, the overall learning degree of the user, the learning degree of each subregion, and the learning degree of each item are estimated (S940). Thereafter, by recommending an appropriate item to the user among the evaluation items based on the estimated learning degree using the Bayesian network (S950), the user may perform the evaluation through the user interface and be notified of the evaluation result.

The computer-based adaptive evaluation system and method using the Bayesian network according to the present invention can be used to improve the accuracy and performance of evaluation in CAT-based evaluation systems such as iBT TOEFL, GRE, GMAT, which are already widely used, and TOEIC, TEPS, iTEMPS It can also be used effectively in other qualification tests such as MOS. In addition, since the present invention can determine the ability of the learner on the same basis as the paper-based test, it is also possible to expect the effect of gradually replacing the paper-based test, which takes time and cost.

In addition, this technique can not only accurately assess the learner's ability with a very small number of questions in a short time, but also can evaluate the learner's ability by detailed learning area, which can be widely used in learning diagnosis systems and learning management systems. have.

The computer-based adaptive evaluation method according to the present invention can be implemented in the form of program instructions that can be executed by various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains various modifications and variations from such descriptions. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

100: CAT system
110: Evaluation Item Database
120: association information generation unit
130: evaluation results database
140: CPT generator
150: Bayesian Network Division
160: learning degree estimator
170: Evaluation Item Recommendation
180: user interface

Claims (7)

In Computerized Adaptive Testing (CAT) system,
An evaluation item database in which a plurality of evaluation items for estimating the degree of learning are divided into a plurality of detailed learning areas and stored;
An association information generation unit configured to cluster the evaluation items stored in the evaluation item database and structure the relationship between the learning area and each item in a Bayesian network topology to generate correlation information between the detailed learning area of each subject to be evaluated and each evaluation item;
A Bayesian network unit generating a Bayesian network based on the generated association information; And
A learning degree estimator for estimating the learning degree of the user for each of the evaluation items and each detailed learning area by using the generated Bayesian network and the relationship for each item.
Lt; / RTI >
The association information generation unit,
The detailed learning area and evaluation items are structured in a tree form,
The relationship between the items,
Whenever one evaluation item is answered by the user, the response result is reflected,
The learning degree estimator,
Whenever one evaluation item is answered by the user, the probability of fitting each evaluation item according to the response result is estimated, and the user's understanding probability for each detailed learning area is estimated.
The learning degree of the user,
And a probability of matching each evaluation item and a user's understanding of each detailed learning area. The method of claim 1,
An evaluation result database in which evaluation results of past users of the evaluation item are stored; And
CPT generation unit for generating a conditional probability table for calculating a probability for each node of the Bayesian network based on an evaluation-maximization (EM) algorithm based on the evaluation result stored in the evaluation result database.
Further comprising:
The Bayesian network unit,
Generate the Bayesian network using the association information and the conditional probability table;
The EM algorithm,
And estimating a hidden variable using the evaluation result. 3. The method of claim 2,
An evaluation item recommending unit for recommending an item suitable for the user among the evaluation items based on the estimated degree of learning;
The computer-based adaptive evaluation system further comprising. In Computerized Adaptive Testing (CAT) method,
Clustering a plurality of evaluation items divided into a plurality of detailed learning areas to structure a relationship between the learning areas and the items in a Bayesian network topology;
Generating association information between the structured detailed learning area and each evaluation item; And
Creating a Bayesian network based on the generated association information;
Estimating the degree of learning of the user for each of the evaluation items and the detailed learning areas by using the generated Bayesian network and the relationship for each item
Lt; / RTI >
The structuring step,
The detailed learning area and evaluation items are structured in a tree form,
The relationship between the items,
Whenever one evaluation question is answered by the user, the response result is reflected,
Estimating the learning degree of the user,
Whenever one evaluation item is answered by the user, the probability of fitting each evaluation item according to the response result is estimated, and the user's understanding probability for each detailed learning area is estimated.
The learning degree of the user,
And a probability of matching each evaluation item and a user's understanding of each detailed learning area. 5. The method of claim 4,
Generating a conditional probability table for calculating a probability for each node of the Bayesian network using an Expectation-Maximization (EM) algorithm based on an evaluation result of the evaluation item;
Further comprising:
Generating the Bayesian network,
Generating the Bayesian network using the association information and the conditional probability table;
The EM algorithm,
And estimating a hidden variable using the evaluation result. 6. The method of claim 5,
After estimating the degree of learning,
Recommending an item suitable for the user among the evaluation items based on the estimated degree of learning;
The computer-based adaptive evaluation method further comprising. A computer-readable recording medium having recorded thereon a program for executing a computer-based adaptive evaluation method according to any one of claims 4 to 6.

Images