CN111241243A

CN111241243A - Knowledge measurement-oriented test question, knowledge and capability tensor construction and labeling method

Info

Publication number: CN111241243A
Application number: CN202010032981.3A
Authority: CN
Inventors: 王志锋; 刘继斌; 左明章; 叶俊民; 罗恒; 闵秋莎; 童名文; 田元; 夏丹
Original assignee: Central China Normal University
Current assignee: Seal Cutting Time Technology Wuhan Co ltd
Priority date: 2020-01-13
Filing date: 2020-01-13
Publication date: 2020-06-05
Anticipated expiration: 2040-01-13
Also published as: CN111241243B

Abstract

The invention belongs to the technical field of education data mining, and discloses a knowledge measurement oriented test question, knowledge and capacity tensor construction and labeling method, which combines a Q matrix and bloom cognition field education target classification to divide the mastering of knowledge points into six cognition capacity levels: knowing, comprehending, applying, analyzing, synthesizing and evaluating to construct test questions, knowledge and capacity tensors; and adopting an active learning strategy to construct an interpretable test question label prediction model, obtaining an interpretable label prediction information entropy, inputting an unlabeled sample into the prediction model by utilizing the trained interpretable test question label prediction model, and feeding back the label prediction information entropy with stronger interpretability, thereby performing man-machine cooperation. The method reduces the influence of the subjectivity of the manual marking on the TKA tensor, has high marking accuracy and efficiency, and greatly reduces the labor cost of experts. The method has strong mobility, can be applied to examination and marking of test question knowledge points of various subjects, and has better applicability.

Description

Knowledge measurement-oriented test question, knowledge and capability tensor construction and labeling method

Technical Field

The invention belongs to the technical field of education data mining, and particularly relates to a knowledge measurement-oriented test question, knowledge and capacity tensor construction and labeling method.

Background

Currently, the closest prior art: in the educational field, diagnostic assessment of an individual's knowledge structure, cognitive processing skills, or cognitive processes is commonly referred to as cognitive diagnostic assessment. Knowledge or cognitive processing skills are collectively called attributes, and the knowledge structure and cognitive processing skills of learners are potential variables that cannot be directly observed. In the theory of cognitive diagnostic evaluation, attributes are the most fundamental concepts, and refer to those potential, implicit psychological traits that affect the outward performance of the subject. In a teaching test, the subject is a learner, the behavior of the subject is shown as the response of the learner on the test question, and one of the main factors influencing the response of the learner is the mastery or cognitive level of the learner on the knowledge point. Therefore, diagnostic analysis is required for learners, and the main evaluation target is to analyze the mastery condition or cognitive level of learners on each content field, especially knowledge points.

The cognitive diagnosis evaluation needs to design a test to induce the external expression of the internal cognitive characteristics of the learner so as to realize the judgment of the internal cognitive characteristics, the process is mainly realized by the representation of test questions and knowledge, and in the aspect of the representation of the relation between subject knowledge and the test questions, three methods are mainly adopted:

1) student-question table (Student-Problem Chart, S-P table) characterization method;

2) a test question knowledge association Q matrix representation method;

3) a characterization method based on knowledge space theory.

The S-P table is a chart structure analysis method, and converted test and exercise score data are arranged into a two-dimensional table, so that the learner and the response situation of test questions are subjected to visualization processing. The S-P table realizes the direct mapping relation between the learner and the test question score, but does not characterize the knowledge points associated with the back of the test question.

The test question knowledge association Q matrix is used for expressing the knowledge points corresponding to each test question, if the test question examines the knowledge points, the knowledge points are recorded with 1, and if not, the knowledge points are recorded with 0. The Q matrix intuitively expresses the knowledge points corresponding to the test questions, but does not relate to the cognitive level and level of the test questions for examining the knowledge points.

A representation method based on knowledge space theory provides a method for expressing knowledge structure, which is a psychological theory for testing knowledge state and knowledge structure of students. In this theory, the knowledge state of a certain domain is represented by a set of questions that the subject can answer, and the knowledge structure is a set of knowledge states that constitute knowledge of a certain domain. The knowledge space theory method describes the mapping relation between the test question and the knowledge structure or the knowledge space, but does not relate to the representation of the knowledge mastering degree and the cognitive level of the learner.

In the Q matrix estimation technique, the following ideas are generally considered:

the simple examination method or the multi-scorer method for the project is implemented by an educator to review and analyze the obtained examination Q matrix so as to further determine the examination Q matrix, and the result has certain subjectivity.

And the other is an iterative estimation method based on the project parameters, which is a method for purifying and refining the Q matrix by combining the project parameters under the precondition of model fitting. However, in most cases, the error of the project parameter obtained by the initially incorrectly labeled Q matrix is large, which will seriously affect the accuracy of the estimation result of the Q matrix based on the project parameter.

And thirdly, performing Q matrix estimation based on response data of learners, wherein the method mainly comprises a nonparametric estimation method and a parametric estimation method, and the method is proved to have good effect through experiments, but the use of the method has strict limiting conditions and is low in applicability.

In summary, the problems of the prior art are as follows:

(1) in the prior art, for the representation of the relation between subject knowledge and test questions, the current research mainly focuses on the representation of the relation between a learner and the test questions and the relation between the test questions and knowledge, and the relation between the test questions and knowledge and ability is not comprehensively considered and represented, so that a subsequent diagnosis model cannot obtain effective support of a data layer, and the knowledge mastering degree and knowledge cognitive level of the learner cannot be diagnosed.

(2) The traditional study diagnosis is to diagnose and analyze learners according to the answer response matrix of learners and the test question-knowledge point association Q matrix, only to judge whether knowledge points are mastered, but not to estimate the knowledge mastering degree and level, and to provide less field information for interpretable modeling.

(3) In the prior art, most of the labeling of test questions is manually completed, and in an online learning platform, the number of the unlabelled test questions is huge, if the test questions are labeled by education experts in the subject field, the consumed labor cost is high, the labeling time is long, the accuracy is low, the subsequent test question diagnosis effect is poor, and the parameter estimation is inaccurate.

The difficulty of solving the technical problems is as follows:

(1) how to comprehensively consider and characterize the relation of test question-knowledge-ability to specifically diagnose the knowledge mastery degree or cognitive level of the learner, thereby obtaining effective support of a data layer for a subsequent diagnosis model and further accurately diagnosing the knowledge mastery degree and knowledge cognitive level of the learner.

(2) How to establish an accurate labeling model, so that the model can learn related labeling processes and training parameters by using the existing labeled samples, realize the automatic labeling of the unlabeled samples by the model, and widen the application range of the model.

(3) How to utilize the active learning strategy to enable the machine and the education experts to collaborate, and carry out man-machine collaborative semi-supervised labeling aiming at the accurate labeling task so as to improve the speed and accuracy of test question labeling and reduce the labor cost of the experts.

The significance of solving the technical problems is as follows: the invention relates to a method for constructing and labeling test questions, knowledge and capacity tensors for knowledge measurement, which combines psychology measurement and Blume cognition field education target classification to construct test questions, knowledge and capacity tensors for knowledge level measurement of learners; preprocessing operations such as text word segmentation and word stop removal are carried out on the test questions, a word vector model is used for vectorizing the test questions, and an interpretable knowledge label prediction model and an interpretable capability label prediction model are respectively constructed by combining a knowledge point library, a capability hierarchical library, test question word vectors, a bidirectional LSTM neural network and a convolutional neural network; aiming at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and an educational expert and a constructed interpretable test question label prediction model are used for performing human-computer cooperative semi-supervised labeling.

The invention is to combine Q matrix and bloom cognition field education target classification, and divide the mastery of knowledge points into six cognitive ability levels: knowing, comprehending, applying, analyzing, synthesizing and evaluating to construct a test question-knowledge-ability tensor. The TKA tensor provides knowledge in the education field for knowledge mastering degree and cognition level mining, and provides interpretable foundation for subsequent modeling from a data level. The TKA tensor concept is a product combining cognitive psychology and psychology, reflects TKA tensors of test questions, knowledge (or cognitive attributes) and cognitive level, is a bridge connecting cognitive psychology and psychology, provides a strong theoretical basis for subsequent knowledge and cognitive level mining of learners, and improves the trust of learners on mining results, so that weak links of learners are attributed according to evidence.

The invention adopts an active learning strategy to construct an interpretable test question label prediction model, obtains an interpretable label prediction information entropy, inputs unmarked samples into the prediction model by utilizing the trained interpretable test question label prediction model, and feeds back the label prediction information entropy with stronger interpretability. The establishment of the test question label prediction model can be explained, the TKA tensor marked by the field experts with abundant experience is taken as a basis, self parameters are optimized through continuous learning, the effect identical to the expert marking condition is expected to be achieved, the manual labor is liberated, the influence of the subjectivity of manual marking on the TKA tensor is reduced, the marking accuracy and efficiency are high, and the labor cost of the experts is greatly reduced.

The invention has strong mobility, can be applied to examination and marking of test question knowledge points of various subjects, and has better applicability.

Disclosure of Invention

Aiming at the problems in the prior art, the invention provides a knowledge measurement-oriented test question, knowledge and capacity tensor construction and labeling method.

The invention is realized in this way, and a knowledge measurement oriented test question, knowledge and capability tensor construction and labeling method comprises the following steps:

combining a Q matrix and bloom cognitive domain education target classification, and constructing test questions, knowledge and capacity tensors for knowledge level measurement of learners;

performing text word segmentation and stop word preprocessing operation on the test questions, vectorizing the test questions by using a word vector model, and respectively constructing an interpretable knowledge label prediction model and an interpretable capability label prediction model by combining a knowledge point library, a capability hierarchical library, test question word vectors, a bidirectional LSTM neural network and a convolutional neural network;

and thirdly, constructing an interpretable test question label prediction model by adopting an active learning strategy to perform man-machine cooperative semi-supervised labeling, obtaining an interpretable label prediction information entropy, inputting an unlabeled sample into the constructed interpretable test question label prediction model, and performing interpretable label prediction information entropy feedback.

Further, the first step specifically comprises:

step a): in knowledge measurement under an actual learning situation, on one hand, a refined capability level corresponding to the knowledge mastery degree of students is required to be measured, on the other hand, the measured capability level is required to have better explanatory property in the education field, and test questions, knowledge and capability tensors representing comprehensive relations among the test questions, the knowledge and the capability are constructed according to the requirement of combining psychology measurement and bloom cognition field education target classification;

step b): the TKA tensor is defined as P ═ P_tka}_T×K×AThe method comprises the steps that a test question T examines a capability level a corresponding to a knowledge point K, wherein T is a test question space (T is more than or equal to 0 and less than or equal to T), K is a knowledge space (K is more than or equal to 0 and less than or equal to K), and A is a capability space (a is more than or equal to 0 and less than or equal to A);

step c): the test question space T is defined as a set formed by the number sequences of the tested test questions;

step d): the knowledge space K is defined as a set of knowledge points examined by the test question space T;

step e): the competence space A is defined as a set of knowledge cognitive competence levels, and the knowledge is classified according to the education targets in the bloom cognitive domain, and the knowledge is divided into six cognitive competence levels: know, comprehend, use, analyze, synthesize, evaluate. "know" is defined as understanding knowledge and remembering, describing the knowledge of concrete knowledge or abstract knowledge, remembering it in a form similar to some knowledge initially encountered by learners; "comprehended" is defined as an understanding of something, but does not require an extensive understanding, but rather is rudimentary, even more facial, superficial; defining "application" as the primary direct application of learned definitions, laws, principles, requiring learning to correctly apply abstract concepts to appropriate problem-solving scenarios without knowledge of the problem-solving model; the analysis is defined as decomposing the knowledge into various element parts, thereby more clearly defining the interrelation among concepts and making the organization structure of the knowledge clearer; the comprehensive processing is defined as that all decomposed elements are comprehensively processed on the basis of analysis and recombined according to needs, so that problems are comprehensively and creatively solved, the special expression is required, the characteristics and the originality are emphasized, and the higher level is required; the evaluation is defined as the highest level of the educational objective in the bloom cognitive domain, requires rational and profound judgment on the value of the essence of knowledge, integrates the internal and external data and information, and makes an inference conforming to objective facts.

Further, the second step specifically comprises:

step 1): preprocessing the test questions to be labeled, wherein the preprocessing comprises text word segmentation and stop word removal;

step 2): text segmentation is carried out on test questions, and mixed segmentation is carried out on the test question text by adopting a method combining a bidirectional maximum matching method and statistics based on a mixed dictionary;

step 3): firstly, establishing a mixed dictionary containing Chinese, English, formulas, special symbols and the like by using a bidirectional maximum matching method, carrying out bidirectional matching on a character string to be segmented and entries in the dictionary one by one, and cutting the entries from the character string to be segmented if the matching is successful, thereby completing preliminary segmentation; then, a statistical probability model is trained by using a large amount of word-segmented texts by using a statistical word segmentation method, and because a word is often a high-frequency combination of several continuous characters, when the co-occurrence frequency of the several continuous characters reaches a certain degree, the word is considered to exist, so that the word segmentation of the text which is not segmented is realized;

step 4): stopping words for the mixed word segmentation result; removing words which are irrelevant to the subjects of sentences and test question texts and do not contribute to the test question labeling task, and treating the words with low frequency as stop words, wherein the words do not contribute to the test question labeling task; according to the rule, a deactivation word bank is established, words appearing in the deactivation word bank are deleted, and words with low frequency are deleted;

step 5): processing the test questions by using a continuous Word bag model of a Word vector model Word2vec, and vectorizing the pre-processed input test questions; the CBOW model predicts a word vector of a target word according to the word vectors of a plurality of words in the context of the target word, and vectorizes the test question;

step 6): the CBOW model architecture comprises an input layer, a projection layer and an output layer; input context { x ] with input layer encoded by one-hot₁,...x_CThe size of a window is C, the size of a vocabulary table is V, a projection layer is an N-dimensional vector, and an output layer outputs word vector representation of a target word y; the one-hot encoded input vector is connected to the projection layer through a weight matrix W with dimension V multiplied by N, and the projection layer is connected to the output layer through a weight matrix W' with dimension N multiplied by V;

step 7): in the CBOW model, a loss function is defined, given the conditional probability of an output word of an input context, taking the logarithm to calculate as:

step 8): and (3) obtaining an updating rule of the output weight W' by differentiating the formula:

step 9): similarly, the update rule of the weight W is:

step 10): and calculating the output of the projection layer h according to the weight updating rule:

step 11): the inputs to each node at the output level are computed,

j-th column representing output matrix W':

step 12): calculating the output of the output layer, output y_jThe following were used:

step 13): the CBOW model obtains more grammar information through context learning, and obtains test word vector output from scale test data;

step 14): calculating a knowledge tag prediction information entropy and a capability tag prediction information entropy by using the test word vector as input and combining a knowledge point library and a capability level library;

step 15): the BilSTM network adopts two LSTMs to obtain the text characteristics of different test questions from opposite directions, and the calculation formula is as follows:

wherein a is₁,a₂,b₁And b₂G (-) is a hidden layer activation function, which is a weight coefficient,

for the forward hidden layer output at time t,

for the backward hidden layer output at the time t, finally fusing the hidden layer outputs in two directions at each time to construct a final output h_t：

Wherein c is₁And c₂Is the weight coefficient, f (-) is the output laserA live function;

step 16): the convolution layer introduces a plurality of convolution kernels to carry out convolution operation, the window width d of the convolution kernels is consistent with the output width of the BilSTM network, and the ith value of the convolution output vector v is calculated as follows:

v_i＝W·H_i:(i+j-1)+b,W∈R^j×d；

wherein, W is the weight coefficient of the convolution layer, H is the text characteristic of the BilSTM output test question, b is the bias term, and j is the number of convolution kernels;

step 17): a mean pooling strategy is used in a pooling layer, the average of characteristic values is taken as output in the field, representative information in the whole window information is obtained, and the characteristic dimension of the test question text and the number of model network parameters are reduced;

step 18): acquiring the prediction probability of the knowledge tag and the capability tag at a Softmax layer, and inputting the test question text features extracted in the steps 15-17) into a Softmax function to acquire the prediction probability P of the knowledge tag_kCapability tag prediction probability P_a：

Wherein O is_kOutput vector O for knowledge tag prediction model_KThe kth element of (1), O_aOutput vector O for capability tag prediction model_AE (-) is an exponential function;

step 19): predicting probability P from knowledge tag_kAnd capability tag prediction probability P_aConstruction of knowledge tag prediction information entropy E_KCapability tag prediction information entropy E_A：

The larger the label prediction information entropy is, the more uncertain the result of the test question label prediction is.

Further, step 14) specifically includes: firstly, extracting test question text characteristics by adopting a bidirectional long-time memory neural network, then introducing a convolutional neural network to optimize the test question text characteristics so as to obtain deeper text characteristic representation of the test question, then further cascading and inputting the test question text representation to a Softmax layer to obtain the prediction probability of a knowledge tag and a capability tag, and finally calculating the prediction information entropy of the knowledge tag and the prediction information entropy of the capability tag.

Further, in step 16), the convolutional layer result is input to the activation function, so that the test question data has the distinguishing capability in the fitting process.

Further, the third step comprises: step i): extracting a small part of test question samples from the training set to be labeled by experts related to the education field, labeling the knowledge labels and the capability labels of the test questions, and constructing the TKA tensor of the test questions;

step ii): training an interpretable test question label prediction model by using the labeled test question samples of the education experts, and taking the remaining unlabelled test question samples as the input of the model to obtain the knowledge label prediction information entropy and the capability label prediction information entropy of the unlabelled samples;

step iii): constructing a TKA tensor label prediction joint information entropy E of the test question:

the larger the test question label prediction joint information entropy is, the more uncertain the label prediction result is.

Step iv): and training a new interpretable test question label prediction model by using more education expert labeling samples, and repeating the steps i) to iii) for a plurality of times until the test question label prediction model is stable in performance, namely the label prediction joint information entropy is smaller than a threshold lambda, and outputting the test question label predicted by the model at the moment.

Further, the third step comprises: step i) is preceded by: for 5263 test question samples, the training set and test set were divided in a 7:3 ratio.

Another object of the present invention is to provide a computer program product stored on a computer readable medium, comprising a computer readable program for providing a user input interface to implement the method for constructing and labeling test questions, knowledge and capacity tensors for knowledge-oriented measurement when the computer program product is executed on an electronic device.

Another object of the present invention is to provide a computer-readable storage medium, which includes instructions that, when executed on a computer, cause the computer to execute the method for constructing and labeling the test question-knowledge-capability tensor oriented knowledge measurement.

Another object of the present invention is to provide a system for constructing and labeling test questions, knowledge, and capacity tensors oriented to knowledge measurement, which implements the method for constructing and labeling test questions, knowledge, and capacity tensors oriented to knowledge measurement, comprising:

the TKA tensor construction module is used for constructing test questions, knowledge and capacity tensors for knowledge level measurement of learners by combining the Q matrix and the class of education targets in the bloom cognitive domain;

the interpretable test question label prediction model module is used for carrying out preprocessing operations such as text word segmentation and word stop removal on a test question, vectorizing the test question by using a word vector model, and respectively constructing an interpretable knowledge label prediction model and an interpretable capability label prediction model by combining a knowledge point library, a capability level library, a test question word vector, a bidirectional LSTM neural network and a convolutional neural network;

the TKA tensor labeling module for the human-computer collaborative test questions aims at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and education experts and a constructed interpretable test question label prediction model are used for human-computer collaborative labeling.

In summary, the advantages and positive effects of the invention are: the method combines a Q matrix and the classification of educational targets in the bloom cognitive field to construct test questions, knowledge and capacity tensors for knowledge level measurement of learners; preprocessing operations such as text word segmentation and word stop removal are carried out on the test questions, a word vector model is used for vectorizing the test questions, and an interpretable knowledge label prediction model and an interpretable capability label prediction model are respectively constructed by combining a knowledge point library, a capability hierarchical library, test question word vectors, a bidirectional LSTM neural network and a convolutional neural network; aiming at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and an educational expert and a constructed interpretable test question label prediction model are used for performing human-computer cooperative semi-supervised labeling. The method fully utilizes computing resources, education experts only need to label a small number of test questions, and most of the test questions are labeled by the algorithm, so that the labor cost of the experts is greatly reduced, and meanwhile, the speed and the efficiency of labeling the test questions are improved. Secondly, the method has strong mobility and can be widely applied to test question labeling tasks of various subjects. Thirdly, the knowledge level of the learner is mined by utilizing the labeled TKA tensor, the concrete level of the class of the education targets in the bloom cognitive domain corresponding to the mastery degree of the learner at the specific knowledge point can be measured, the measurement result has better explanatory property in the education domain, on one hand, the trust of the user on the measurement result is improved, on the other hand, the effective evidence-based attribution can be carried out on weak links of the learning, and the method can be widely applied to the fields of education data mining, education measurement, online learning, psychological measurement and the like.

Compared with the prior art, the invention has the advantages that:

the invention is to combine Q matrix and bloom cognition field education target classification, and divide the mastery of knowledge points into six cognitive ability levels: knowing, comprehending, applying, analyzing, synthesizing and evaluating to construct a test question-knowledge-ability tensor. The TKA tensor provides knowledge in the education field for knowledge mastering degree and cognitive level, and provides interpretable foundation for subsequent modeling from a data level. The TKA tensor concept is a product combining cognitive psychology and psychology, reflects TKA tensors of test questions, knowledge (or cognitive attributes) and cognitive level, is a bridge connecting cognitive psychology and psychology, provides a strong theoretical basis for subsequent knowledge and cognitive level mining of learners, and improves the trust of learners on mining results, so that weak links of learners are attributed according to evidence.

The invention adopts an active learning strategy to construct an interpretable test question label prediction model, obtains an interpretable label prediction information entropy, inputs unmarked samples into the prediction model by utilizing the trained interpretable test question label prediction model, and feeds back the label prediction information entropy with stronger interpretability. The establishment of the test question label prediction model can be explained, the TKA tensor marked by the field experts with abundant experience is taken as a basis, self parameters are optimized through continuous learning, the effect identical to the expert marking condition is expected to be achieved, the manual labor is liberated, the influence of the subjectivity of manual marking on the TKA tensor is reduced, the marking efficiency is improved, and the labor cost of the experts is greatly reduced.

The method has strong mobility, can be applied to test question labeling of various subjects, and has better applicability.

Aiming at 5263 test question samples, a training set and a test set are divided according to a ratio of 7:3, representative test question samples are extracted from the training set and are submitted to experts relevant to the education field for labeling, and then the test question samples in the training set which are not labeled and the test question samples in all the test sets are predicted in the following two ways.

The collaborative labeling method comprises the following steps: and carrying out test question labeling on the collaborative interpretable test question label prediction model and related experts in the education field.

The random sampling method comprises the following steps: and (4) adopting random sampling to be labeled by related experts in the education field, and then predicting the residual test question samples.

The results of the experiment are shown in the table:

TABLE 1 interpretable test question label prediction model labeling TKA tensor result

The method combines a Q matrix and the classification of educational targets in the bloom cognitive field to construct test questions, knowledge and capacity tensors for knowledge level measurement of learners; preprocessing operations such as text word segmentation and word stop removal are carried out on the test questions, a word vector model is used for vectorizing the test questions, and an interpretable knowledge label prediction model and an interpretable capability label prediction model are respectively constructed by combining a knowledge point library, a capability hierarchical library, test question word vectors, a bidirectional LSTM neural network and a convolutional neural network; aiming at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and an educational expert and a constructed interpretable test question label prediction model are used for performing human-computer cooperative semi-supervised labeling.

The invention is to combine Q matrix and bloom cognition field education target classification, and divide the mastery of knowledge points into six cognitive ability levels: a is₁Know, a₂Figure A shows₃Application, a₄Analysis, a₅Synthesis, a₆And evaluating and constructing a tensor of test questions, knowledge and capability. The TKA tensor provides knowledge in the education field for knowledge mastering degree and cognition level mining, and provides interpretable foundation for subsequent modeling from a data level.

The invention adopts an active learning strategy to construct an interpretable test question label prediction model, obtains an interpretable label prediction information entropy, inputs unmarked samples into the prediction model by utilizing the trained interpretable test question label prediction model, and feeds back the label prediction information entropy with stronger interpretability.

The invention utilizes the interpretable test question label prediction model to ensure that a small number of representative samples are labeled by education experts, and other large number of samples are labeled by a prediction algorithm, thereby realizing the cooperative labeling of the label prediction model and the education experts and greatly improving the TKA labeling efficiency.

In conclusion, the TKA tensor concept is established, an interpretable basis is provided for subsequent modeling from a data level, and evidence-based attribution of weak learning links is facilitated; the TKA tensor is automatically labeled by using the interpretable test question label prediction model, so that manual labor is greatly liberated, the labor cost of experts is reduced, the model is more suitable for data through man-machine collaborative labeling, labeling errors are effectively reduced, and the labeling efficiency is improved; the method has strong mobility, can be applied to test question labeling of various disciplines, and has better applicability.

Drawings

Fig. 1 is a flowchart of a method for constructing and labeling test questions, knowledge and capacity tensors based on knowledge measurement according to an embodiment of the present invention.

Fig. 2 is a schematic diagram of a method for constructing and labeling test questions, knowledge and capacity tensors for knowledge measurement according to the embodiment of the present invention.

Fig. 3 is a diagram of the structure composition of the test question, knowledge, capability (TKA) tensor provided by an embodiment of the present invention.

Fig. 4 is a schematic structural diagram of a model for predicting interpretable test question labels according to an embodiment of the present invention.

Fig. 5 is a schematic diagram of a system for constructing and labeling test questions, knowledge and capacity tensors oriented to knowledge measurement according to an embodiment of the present invention.

In the figure: 1. a TKA tensor construction module; 2. the interpretable test question label prediction model building module; 3. and a man-machine cooperation labeling module.

FIG. 6 is a line drawing of the TKA tensor labeling experiment result in an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

The invention aims to solve the problem that the relation of test questions, knowledge and capability cannot be comprehensively considered and characterized in the prior art, so that a subsequent diagnosis model cannot obtain effective support of a data layer; the labeling of the test questions is completed manually, the number of the unlabelled test questions is huge on an online learning platform, and if the test questions are labeled by education experts in the subject field, the labor cost is high, the labeling time is long, the accuracy is low, the subsequent test question diagnosis effect is poor, and the parameter estimation is inaccurate.

Aiming at the problems in the prior art, the invention provides a knowledge measurement oriented test question, knowledge and capacity tensor construction and labeling method, which is described in detail in the following by combining the attached drawings.

As shown in fig. 1, the method for constructing and labeling test questions, knowledge and capacity tensors based on knowledge measurement according to the embodiment of the present invention includes: combining the Q matrix and the class of education targets in the bloom cognitive domain, and constructing test questions, knowledge and capacity tensors for knowledge level measurement of learners; preprocessing operations such as text word segmentation and word stop removal are carried out on the test questions, a word vector model is used for vectorizing the test questions, and an interpretable knowledge label prediction model and an interpretable capability label prediction model are respectively constructed by combining a knowledge point library, a capability hierarchical library, test question word vectors, a bidirectional LSTM neural network and a convolutional neural network; aiming at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and an educational expert and a constructed interpretable test question label prediction model are used for performing human-computer cooperative semi-supervised labeling.

The method specifically comprises the following steps:

s101: and combining the Q matrix and the class of education targets in the bloom cognitive domain to construct test questions, knowledge and capacity tensors for knowledge level measurement of learners.

S102: the method comprises the steps of carrying out preprocessing operations such as text word segmentation and word stop removal on test questions, vectorizing the test questions by using a word vector model, and respectively constructing an interpretable knowledge label prediction model and an interpretable capability label prediction model by combining a knowledge point library, a capability hierarchical library, test question word vectors, a bidirectional LSTM neural network and a convolutional neural network.

S103: aiming at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and an educational expert and a constructed interpretable test question label prediction model are used for performing human-computer cooperative semi-supervised labeling.

Fig. 2 is a schematic diagram of a method for constructing and labeling test questions, knowledge and capacity tensors oriented to knowledge measurement according to an embodiment of the present invention.

As the preferred embodiment of the present invention. Step S101 specifically includes:

step 1.1: knowledge measurement in an actual learning situation requires measurement of a refined capability level corresponding to the knowledge mastery degree of a student on the one hand, and requires the measured capability level to have better explanatory property in the education field on the other hand, so that a TKA tensor representing the comprehensive relation among test questions, knowledge and capability is constructed by combining psychometrics and bloom cognition field education target classification.

Step 1.2: the TKA tensor is defined as P ═ P_tka}_T×K×AThe method indicates that the test question T examines the capability level a corresponding to the knowledge point K, wherein T is a test question space (T is more than or equal to 0 and less than or equal to T), K is a knowledge space (K is more than or equal to 0 and less than or equal to K), and A is a capability space (a is more than or equal to 0 and less than or equal to A).

Step 1.3: the test question space T is defined as a collection of numbered sequences of the test questions under examination.

Step 1.4: the knowledge space K is defined as a set of knowledge points examined by the test question space T.

Step 1.5: the competence space A is defined as a set of knowledge cognitive competence levels, and the knowledge is classified according to the education targets in the bloom cognitive domain, and the knowledge is divided into six cognitive competence levels: know, comprehend, use, analyze, synthesize, evaluate. "know" is defined as understanding knowledge and remembering, describing the knowledge of concrete knowledge or abstract knowledge, remembering it in a form similar to some knowledge initially encountered by learners; "comprehended" is defined as an understanding of something, but does not require an extensive understanding, but rather is rudimentary, even more facial, superficial; defining "application" as the primary direct application of learned definitions, laws, principles, requiring learning to correctly apply abstract concepts to appropriate problem-solving scenarios without knowledge of the problem-solving model; the analysis is defined as decomposing the knowledge into various element parts, thereby more clearly defining the interrelation among concepts and making the organization structure of the knowledge clearer; the comprehensive processing is defined as that all decomposed elements are comprehensively processed on the basis of analysis and recombined according to needs, so that problems are comprehensively and creatively solved, the special expression is required, the characteristics and the originality are emphasized, and the higher level is required; the evaluation is defined as the highest level of the educational objective in the bloom cognitive domain, requires rational and profound judgment on the value of the essence of knowledge, integrates the internal and external data and information, and makes an inference conforming to objective facts.

As shown in fig. 3, the embodiment of the present invention provides a structure composition diagram of the test question, knowledge and capability (TKA) tensor.

As the preferred embodiment of the present invention. Step S102 specifically includes:

step 2.1: the test questions to be labeled are preprocessed, and the preprocessing mainly comprises text word segmentation and stop words.

Step 2.2: and performing text word segmentation on the test question. Based on a mixed dictionary, a method combining a bidirectional maximum matching method and statistics is adopted to perform mixed word segmentation on the test question text.

Step 2.3: firstly, a mixed dictionary containing Chinese, English, formulas, special symbols and the like is established by utilizing a bidirectional maximum matching method, the character string to be segmented and the entries in the dictionary are subjected to one-by-one bidirectional matching, and if the matching is successful, the entries are cut out from the character string to be segmented, so that the preliminary segmentation is completed. Then, a statistical probability model is trained by using a large amount of word-segmented texts by using a statistical word segmentation method, and because a word is often a high-frequency combination of several continuous characters, when the co-occurrence frequency of the several continuous characters reaches a certain degree, the word is considered to exist, so that word segmentation of the unsingulated test question text is realized, and finer segmentation is realized.

Step 2.4: and stopping words according to the mixed word segmentation result. And words which are irrelevant to the subjects of sentences and test question texts and do not contribute to the test question labeling task are removed, and in addition, words with low frequency also do not contribute to the test question labeling task, so the words with low frequency are also treated as stop words. And according to the two rules, establishing a disabled word bank, deleting words appearing in the disabled word bank, and deleting words with low frequency.

Step 2.5: and processing the test questions by using a continuous Word bag model of the Word vector model Word2vec, and vectorizing the pre-processed input test questions. The CBOW model predicts the word vectors of the target words according to the word vectors of a plurality of words in the context of the target words, and therefore vectorizes the test questions.

Step 2.6: the CBOW model architecture comprises an input layer, a projection layer and an output layer. The input layer is an input encoded by one-hotContext { x₁,...x_CAnd the size of a window is C, the size of a vocabulary table is V, a projection layer is an N-dimensional vector, and an output layer outputs word vector representation of a target word y so as to solve the problem of sparsity of data storage. The one-hot encoded input vector is connected to the projection layer by a weight matrix W of dimension V × N, and the projection layer is connected to the output layer by a weight matrix W' of dimension N × V.

Step 2.7: in the CBOW model, a loss function is defined, i.e. the conditional probability of an output word given an input context, logarithmically calculated as:

step 2.8: and (3) obtaining an updating rule of the output weight W' by differentiating the formula:

step 2.9: similarly, the update rule of the weight W is:

step 2.10: and calculating the output of the projection layer h according to the weight updating rule:

step 2.11: the inputs to each node at the output level are computed,

j-th column representing output matrix W':

step 2.12: calculating the output of the output layer, output y_jThe following were used:

step 2.13: the CBOW model obtains more grammar information through context learning, and obtains high-quality test word vector output from the scale test data.

Step 2.14: the method comprises the steps of using a test question word vector as input, combining a knowledge point library and a capability level library, firstly extracting test question text features by using a bidirectional long-time memory neural network (BilSTM), then introducing a convolutional neural network to optimize the test question text features, thereby obtaining deeper text feature representation of the test question, secondly further inputting the test question text representation to a Softmax layer in a cascade mode, thereby obtaining prediction probabilities of a knowledge tag and a capability tag, and finally calculating prediction information entropy of the knowledge tag and prediction information entropy of the capability tag.

Step 2.15: the BilSTM network adopts two LSTMs to obtain text characteristics of different test questions from opposite directions, and the calculation is defined as:

for the forward hidden layer output at time t,

Wherein c is₁And c₂F (-) is the output activation function for the weight coefficients.

Step 2.16: the convolution layer introduces a plurality of convolution kernels to carry out convolution operation, the window width d of the convolution kernels is consistent with the output width of the BilSTM network, and the ith value of the convolution output vector v is calculated as follows:

v_i＝W·H_i:(i+j-1)+b,W∈R^j×d

wherein, W is the weight coefficient of the convolution layer, H is the text characteristic of the BilSTM output test question, b is the bias term, and j is the number of convolution kernels. The result of the convolutional layer is input into the activation function, so that the test question data has certain distinguishing capacity in the fitting process, and the performance of the label prediction model is improved. As shown in fig. 4, a schematic structural diagram of a model for predicting interpretable test question tags is provided in the embodiment of the present invention.

Step 2.17: and a mean pooling strategy is used in a pooling layer, namely, the average of characteristic values is taken as output in the field, so that representative information in the whole window information can be obtained, the characteristic dimension of the test question text and the number of model network parameters are reduced, and the adaptability of the label prediction model is improved.

Step 2.18: obtaining the prediction probability of the knowledge label and the capability label at the Softmax layer, inputting the test question text feature representation extracted by the network into the Softmax function, and then obtaining the prediction probability P of the knowledge label_kCapability tag prediction probability P_a：

Wherein O is_kOutput vector O for knowledge tag prediction model_KThe kth element of (1), O_aOutput vector O for capability tag prediction model_AIs an exponential function.

Step 2.19: predicting probability P from knowledge tag_kAnd ability toProbability of label prediction P_aConstruction of knowledge tag prediction information entropy E_KCapability tag prediction information entropy E_A：

The label prediction information entropy has good physical explanation, and when the label prediction information entropy is larger, the result of the test question label prediction is more uncertain.

As the preferred embodiment of the present invention. Step 3, aiming at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and an educational expert and a constructed interpretable test question label prediction model are utilized to perform human-computer cooperative semi-supervised labeling, and the method specifically comprises the following steps:

step 3.1: and aiming at 5263 test question samples, dividing a training set and a test set according to a 7:3 ratio.

Step 3.2: and aiming at the unlabelled test questions, extracting a small part of test question samples from the training set, submitting the samples to the labeling of experts related to the education field, and labeling the knowledge labels and the capability labels of the test questions, thereby constructing the TKA tensor of the test questions.

Step 3.3: training an interpretable test question label prediction model by using the labeled test question samples of the education experts, and taking the residual unlabeled test question samples as the input of the model, thereby obtaining the knowledge label prediction information entropy and the capability label prediction information entropy of the unlabeled samples.

Step 3.4: constructing a TKA tensor label prediction joint information entropy E of the test question:

when the joint information entropy of the test question label prediction is larger, the result of label prediction is uncertain, so that the test questions with larger information entropy are recommended to education experts for labeling so as to eliminate the uncertainty of the prediction model to the label prediction information.

Step 3.5: and training a new interpretable test question label prediction model by using more education expert labeling samples, and repeating the steps from 3.3 to 3.4 for a plurality of times until the test question label prediction model is stable in performance, namely the label prediction joint information entropy is smaller than a threshold lambda, and outputting the test question label predicted by the model at the moment. Therefore, a small number of representative samples can be ensured to be labeled by education experts, other large number of samples are labeled by the test question label prediction model, the test question label prediction model and the education experts are labeled in a coordinated mode, and the labeling efficiency of the TKA tensor of the test questions is greatly improved.

As shown in fig. 5, the system for constructing and labeling test questions, knowledge and capacity tensors based on knowledge measurement according to the embodiment of the present invention includes:

and the TKA tensor construction module is used for constructing test questions, knowledge and capacity tensors for knowledge level measurement of learners by combining the Q matrix and the class of education targets in the bloom cognitive domain.

The interpretable test question label prediction model module is used for carrying out preprocessing operations such as text word segmentation and word stop removal on a test question, vectorizing the test question by using a word vector model, and respectively constructing an interpretable knowledge label prediction model and an interpretable capability label prediction model by combining a knowledge point library, a capability level library, a test question word vector, a bidirectional LSTM neural network and a convolutional neural network; the TKA tensor labeling module for the human-computer collaborative test questions aims at the TKA tensor labeling task with accurate test questions, an active learning strategy is adopted, and education experts and a constructed interpretable test question label prediction model are used for human-computer collaborative labeling.

The invention is further described below with reference to the experiments and results.

Aiming at 5263 test question samples, a training set and a test set are divided according to a ratio of 7:3, a small part of test question samples are extracted from the training set and are submitted to relevant experts in the education field for labeling, and then the test question samples in the training set which are not labeled and the test question samples in all the test sets are predicted by using a collaborative labeling method and a random sampling method. The collaborative labeling method collaborates with the interpretable test question label prediction model and the related experts in the education field to label the test questions. The random sampling method adopts random sampling to be labeled by related experts in the education field, and then the residual test question samples are predicted.

The results of the experiment are shown in the table:

table 2 interpretable test question label prediction model labeling TKA tensor result

In order to compare the cognitive diagnosis effect by using the TKA tensor, the knowledge mastering condition of the learner and the knowledge cognition level mining result of the learner are verified through experiments under the condition based on the TKA tensor and the traditional Q matrix. In the experiment, a multi-level cognitive diagnosis model is adopted to mine test question parameters, knowledge mastering conditions of learners and knowledge cognitive levels of the learners.

Root Mean Square Error (RMSE) is used for evaluating the condition of test question parameters mined by different models, the Root Mean Square Error measures the deviation between an estimated value and a real value, and the smaller the value is, the closer the mining result is to the real value is. The calculation formula is as follows:

evaluating the accuracy of the knowledge mastery or knowledge cognition mode of the mined learner by using a mode criterion rate (PMR) and An AAMR (AAMR), and judging the correct proportion of the learner in a certain knowledge mastery or knowledge cognition mode of the mined learner by using a mode criterion rate measuring model; the marginal attribute criterion rate is used for measuring the correct proportion of the knowledge mastery or cognition level judgment in the knowledge mastery or cognition level modes of all the mined learners. The larger the value is, the higher the accuracy of the mined learner cognitive mode is, and the calculation formula is as follows:

the results of the experiment are shown in the table:

table 3 comparison of mining results using TKA tensors with conventional Q matrices

As can be seen from the experimental results and fig. 6, for the TKA tensor labeling result, with the continuous increase of the expert labeling proportion in the training set, the two labeling methods generally show an increasing trend in the labeling precision of the test question samples in the training set that are not labeled and the test question samples in the test set.

For the collaborative labeling method, when the labeling proportion of experts in the training set reaches 3%, the labeling precision of the test question samples in the training set which are not labeled can reach more than 90%, and when the labeling proportion of experts in the training set reaches 5%, the labeling precision of the test question samples in the test set can reach more than 80%, and along with the improvement of the labeling proportion of the experts in the training set, the labeling precision of the test question samples in the training set which are not labeled and the labeling precision of the test question samples in the test set are continuously improved.

For the random sampling method, under the condition that the labeling proportion of experts in the training set is the same, the labeling precision of the test question samples in the training set which are not labeled and the labeling precision of the test question samples in the test set are lower than that of the cooperative labeling method. Along with the improvement of the expert labeling proportion in the training set, the labeling precision of the test question samples in the training set which are not labeled and the test question samples in the test set generally shows an ascending trend, but a vibration point exists, and the method is unstable in performance.

Aiming at the mining results of the test question parameters and the learner parameters, the test question parameters mined based on the TKA tensor are closer to the true values, and compared with the mining results based on the traditional Q matrix, the knowledge mastering of the mined learners is higher in accuracy. In addition, the knowledge cognition level of the learner can be mined based on the TKA tensor, reliable explanatory information is given, and the knowledge cognition level is close to the true value, but the knowledge cognition level of the learner cannot be mined based on the traditional Q matrix.

In summary, in the aspect of test question sample labeling, the TKA tensor labeled by the collaborative labeling method has higher test question labeling precision on the test question sample not labeled in the training set and on the test set than the TKA tensor labeled by the random sampling method, so that the labeling efficiency of the TKA tensor is effectively improved, and powerful and interpretable data support is provided for the subsequent knowledge mastering condition of learners, knowledge cognitive level mining of learners, and the like. In the aspect of mining the test question parameters and the learner parameters, the test question parameters mined based on the TKA tensor are closer to the real condition, the knowledge mastering condition and the knowledge understanding level of the learner can be accurately mined, the mining granularity is more refined, the learner can timely make targeted remedial measures, and personalized learning is better performed.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When used in whole or in part, can be implemented in a computer program product that includes one or more computer instructions. When loaded or executed on a computer, cause the flow or functions according to embodiments of the invention to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, the computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL), or wireless (e.g., infrared, wireless, microwave, etc.)). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that includes one or more of the available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents and improvements made within the spirit and principle of the present invention are intended to be included within the scope of the present invention.

Claims

1. A knowledge measurement oriented test question, knowledge and capability tensor construction and labeling method is characterized by comprising the following steps:

and step three, constructing an interpretable test question label prediction model by adopting an active learning strategy to carry out man-machine collaborative labeling, obtaining an interpretable label prediction information entropy, inputting an unlabeled sample into the constructed interpretable test question label prediction model, and carrying out interpretable label prediction information entropy feedback.

2. The knowledge measurement oriented test question, knowledge and capability tensor construction and labeling method of claim 1, wherein the first step specifically comprises the following steps:

step a): combining the Q matrix and the class of education targets in the bloom cognitive domain, and constructing test questions, knowledge and capacity tensors representing comprehensive relations among the test questions, the knowledge and the capacity;

step b): the examination question, knowledge and ability tensor is P ═ P_tka}_T×K×AThe method comprises the steps of representing a test question T to examine a capability level a corresponding to a knowledge point K, wherein T is a test question space, T is more than or equal to 0 and less than or equal to T, K is a knowledge space, K is more than or equal to 0 and less than or equal to K, A is a capability space, and a is more than or equal to 0 and less than or equal to A;

step c): the test question space T is a set formed by the serial number sequences of the tested test questions;

step d): the knowledge space K is a set of knowledge points examined by the test question space T;

step e): the ability space A is a set of knowledge cognitive ability levels, is classified according to the education targets in the bloom cognitive domain, and divides knowledge mastery into six cognitive ability levels: a is₁Know, a₂Figure A shows₃Application, a₄Analysis, a₅Synthesis, a₆And evaluating, namely constructing test questions, knowledge and capacity tensors for knowledge level measurement of learners.

3. The knowledge measurement oriented test question, knowledge and capability tensor construction and labeling method of claim 1, wherein the second step specifically comprises the following steps:

step 5): processing the test questions by using a Continuous Word Bag model (CBOW) Of a Word vector model Word2vec, and vectorizing the pre-processed input test questions; the CBOW model predicts a word vector of a target word according to the word vectors of a plurality of words in the context of the target word, and vectorizes the test question;

step 9): similarly, the update rule of the weight W is:

step 11): the inputs to each node at the output level are computed,

j-th column representing output matrix W':

for the forward hidden layer output at time t,

Wherein c is₁And c₂F (-) is the output activation function, which is the weight coefficient;

v_i＝W·H_i:(i+j-1)+b,W∈R^j×d；

4. The test question, knowledge and capability tensor construction and labeling method oriented to knowledge measurement as recited in claim 3, wherein the step 14) specifically comprises: firstly, extracting test question text characteristics by adopting a bidirectional long-time memory neural network, then introducing a convolutional neural network to optimize the test question text characteristics so as to obtain deeper text characteristic representation of the test question, then further cascading and inputting the test question text representation to a Softmax layer to obtain the prediction probability of a knowledge tag and a capability tag, and finally calculating the prediction information entropy of the knowledge tag and the prediction information entropy of the capability tag.

5. The knowledge measurement oriented test question, knowledge and capability tensor construction and labeling method of claim 3, wherein in the step 16), the convolutional layer result is input into the activation function, so that the test question data has the distinguishing capability in the fitting process.

6. The test question, knowledge and capacity tensor construction and labeling method oriented to knowledge measurement as recited in claim 1, wherein the third step comprises:

step i): for the unlabelled test questions, randomly extracting a small part of test question samples from the training set, submitting the small part of test question samples to an education expert for manual labeling of knowledge labels and capability labels of the labeled test questions, and constructing a TKA tensor of the test questions;

the larger the test question label prediction joint information entropy is, the more uncertain the label prediction result is;

step iv): and training a new interpretable test question label prediction model by using more education expert labeling samples, and repeating the steps ii) to iii) for a plurality of times until the test question label prediction model is stable in performance, namely the label prediction joint information entropy is smaller than a threshold lambda, and outputting the test question label predicted by the model at the moment.

7. The test question, knowledge and capacity tensor construction and labeling method oriented to knowledge measurement as recited in claim 1, wherein the third step comprises: step i) is preceded by: for 5263 test question samples, the training set and test set were divided in a 7:3 ratio.

8. A computer program product stored on a computer readable medium, comprising a computer readable program for providing a user input interface for implementing the method for question, knowledge, capability tensor construction and annotation for knowledge-oriented measurements as claimed in any one of claims 1 to 7 when executed on an electronic device.

9. A computer-readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method for question, knowledge, capability tensor construction and annotation for knowledge measurement oriented as claimed in any one of claims 1 to 7.

10. A knowledge measurement oriented test question, knowledge and capability tensor construction and labeling system for implementing the knowledge measurement oriented test question, knowledge and capability tensor construction and labeling method of any one of claims 1 to 7, wherein the knowledge measurement oriented test question, knowledge and capability tensor construction and labeling system comprises: