KR20230047351A - Method for questions assessment and device for questions assessment using the same - Google Patents

Abstract

The present invention relates to a question evaluation method and a device using the same. The question evaluation method comprises the steps of: receiving an arbitrary question to be solved by a user; determining a potential concept for the question; predicting a probability of the user getting the correct answer to the question; sharing a hidden concept layer with a concept evaluation model for predicting a probability of the user understanding a concept; predicting a probability of the user understanding the determined concept; and providing, to a user device, the predicted probability of getting the correct answer and the probability of understanding the concept. According to the present invention, prediction performance of a question evaluation model and a concept evaluation model can be mutually improved.

Description

Item evaluation method and device using it {METHOD FOR QUESTIONS ASSESSMENT AND DEVICE FOR QUESTIONS ASSESSMENT USING THE SAME}

The present invention relates to an item evaluation method and a device using the same, and more particularly, to an item evaluation method configured to evaluate a user's correct answer to a question to be solved by a user and an understanding of a concept of the item, and an item evaluation using the same It's about the device.

Recently, various learning methods that can proceed with learning according to the learner's situation regardless of time and space have emerged, breaking away from the conventional learning method in which the learner actually faces the teacher and learns through the guidance of the teacher.

In particular, the e-Learning learning method is an active learning method in which learners themselves can lead their own learning in a ubiquitous information and communication environment using the Internet, away from the classroom-centered learning environment. The method is gradually expanding its weight in overall education, and various types of e-learning technologies have been proposed according to this trend.

However, these conventional technologies are only focused on simply delivering learning matters to learners unilaterally, and in any order according to the learner's understanding changed by learning and the learning goal actively selected by the learner. No technology has been suggested regarding whether to proceed with learning, what learning content to select, and what level of difficulty to provide to learners.

In particular, as the conventional e-learning learning environment has limitations in carrying out efficient learning in consideration of individual learners' knowledge abilities, learning efficiency may be lower than that of 1:1 offline classes that provide optimized learning for individual learners. there is.

The background description of the invention has been prepared to facilitate understanding of the present invention. It should not be construed as an admission that matters described in the background art of the invention exist as prior art.

Registered Patent Publication No. 10-2015075 (2019.08.27.)

On the other hand, the inventors of the present invention, as a way to overcome the limitations of the conventional learning system as described above, paid attention to the development of an educational data mining tool for predicting the performance of each learner and advising the learner's learning. . In particular, the inventors of the present invention noted that knowledge tracing can be an important factor in improving the learning ability of individual students.

In this case, the knowledge tracing is a task of modeling a student's knowledge state, and may be expressed as a scalar value or vector representing a master level of knowledge components (KC), for example, a student's ability level.

Therefore, the inventors of the present invention could recognize that the achievement level of a learner is better understood in an e-learning learning environment based on the estimated state of knowledge of the learner, and learning materials can be selected accordingly. Furthermore, the inventors of the present invention could expect to prepare a better learning plan for maximizing learning efficiency by applying a learning analysis tool to an e-learning learning environment.

As a result, the inventors of the present invention have developed an item evaluation system configured to evaluate items provided in an e-learning learning environment based on a learner's state of knowledge.

At this time, the inventors of the present invention tried to apply an item evaluation model configured to predict whether or not a user answered correctly to an item and a concept evaluation model configured to evaluate the degree of understanding of concepts of the item to the item evaluation system.

More specifically, the inventors of the present invention determine the concept for the item based on latent concepts, and evaluate the item configured to predict whether or not the item is correct based on the determined concept and the concept learning status for the learner. The model could be applied to the item evaluation system.

Furthermore, the inventors of the present invention have attempted to apply a concept evaluation model configured to share at least one of a latent concept and a concept learning state with an item evaluation model and predict a level of understanding of a concept for an item to an item evaluation system.

At this time, the inventors of the present invention share 'latent concept' and/or 'concept learning state', which are hidden concept layers of the item evaluation model, to develop a concept evaluation model that enhances the interpretability of these layers. As applied, it was expected that an accurate analysis of the learner's learning status could be performed.

Furthermore, the inventors of the present invention found that the item evaluation system had higher item prediction performance than when a single item evaluation model was applied, as the concept evaluation model configured to interpret and evaluate the understanding of the learner was applied together. Could.

More specifically, in the single item evaluation model, the evaluation ability of items for concepts with a low frequency of appearance may be low due to overfitting due to items having concepts that frequently appear in the learning stage. That is, the performance of a single item evaluation model may deteriorate depending on the balance of item items for each concept.

On the other hand, the item evaluation system of the present invention, which is composed of an item evaluation model and a concept evaluation model, has a higher ability to evaluate items for concepts with a lower frequency of appearance than a single model, as it shares the concept learning state for the user, and Even if the evaluation is repeated, the diagnostic performance can be maintained at a high level.

Therefore, the inventors of the present invention can provide a customized e-learning environment for individual learners by providing the correct answer of the question output by each model and the degree of understanding of the concept related to the question by the item evaluation system. could have expected more.

Therefore, the problem to be solved by the present invention is to determine the concept of the received item using the item evaluation model, predict whether or not the item is correct, and predict the level of understanding of the concept related to the item using the concept evaluation model. It is to provide a configured item evaluation method and a device using the same.

The tasks of the present invention are not limited to the tasks mentioned above, and other tasks not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, an item evaluation method according to an embodiment of the present invention is provided. The method, as an item evaluation method implemented by a processor, includes receiving an arbitrary item to be solved by a user from a user device, determining a latent concept for the item, a plurality of latent concepts and the user's concept learning state. Using an item evaluation model composed of a hidden concept layer including a hidden concept layer and an item solving ability determination layer, predicting a user's correct answer probability for the item, a concept evaluation model for predicting a user's concept understanding probability using the hidden concept layer. estimating the user's concept understanding probability for the determined concept using the concept evaluation model composed of the shared hidden concept layer and the concept comprehension capability determination layer, and the predicted correct answer probability and concept understanding probability and providing the item to a user device, wherein the item evaluation model determines the plurality of latent concepts based on an input layer into which an item vector corresponding to the item is input and a degree of similarity between the item vector and the vectorized plurality of latent concepts. A first hidden concept layer for determining an item evaluation weight corresponding to each concept, a second hidden concept layer for determining a lead vector based on the concept learning state and the item evaluation weight, and a second hidden concept layer for determining the item vector and the lead vector. The item-solving ability determination layer for determining item-solving ability and an output layer for calculating a correct answer probability of the user by inputting item difficulty and item-solving ability based on the item vector, the concept evaluation model comprising: A first hidden concept determining a concept evaluation weight corresponding to each of the plurality of latent concepts based on an input layer into which a concept vector corresponding to the determined concept is input and a degree of similarity between the concept vector and each of the plurality of vectorized latent concepts. A concept layer, a second hidden concept layer for determining a lead vector based on the concept learning state and the concept evaluation weight, the concept understanding capability determination layer for determining a concept understanding capability based on the concept vector and the lead vector, and the concept and an output layer that calculates a concept understanding probability of the user by inputting the concept difficulty and the concept understanding ability based on a vector.

According to a feature of the present invention, the concept learning state may be defined as a concept memory composed of absolute values of concept vectors obtained by vectorizing latent concepts of the user-corrected correct answer item.

According to another feature of the present invention, the step of receiving an answer to the question from the user device, and updating the concept memory based on whether the received answer is a correct answer and a latent concept associated with the question can do.

According to another feature of the present invention, after the providing step, determining an item with high comprehension associated with a concept having a level of understanding above a predetermined level among a plurality of additional items based on the predicted level of understanding of the concept, and The method may further include providing, to the user device, the remaining questions excluding the highly understandable questions among the plurality of additional questions.

In order to solve the above problems, a device for evaluating items according to an embodiment of the present invention is provided. The device includes a communication unit configured to receive an arbitrary question to be solved by a user, and a processor. At this time, the processor determines a latent concept for the item, and uses an item evaluation model composed of a plurality of latent concepts, a hidden concept layer including the user's concept learning state, and an item solving ability determination layer, Predicting the probability of correct answer of , sharing the hidden concept layer to a concept evaluation model for predicting the probability of understanding a user's concept, and using the concept evaluation model composed of the shared hidden concept layer and the concept understanding ability determination layer, the determined It is configured to predict a concept understanding probability of a user for a concept and provide the predicted correct answer probability and concept understanding probability to the user device, wherein the item evaluation model includes an input layer into which item vectors corresponding to the items are input, the a first hidden concept layer for determining an item evaluation weight corresponding to each of the plurality of latent concepts based on a similarity between an item vector and the plurality of vectorized latent concepts;

A second hidden concept layer for determining a lead vector based on the concept learning state and the item evaluation weight, the item solving ability determination layer for determining item solving ability based on the item vector and the lead vector, and the item vector and an output layer that calculates a probability of correct answer of the user by inputting an item difficulty level and the item solving ability based thereon, and the concept evaluation model includes an input layer to which a concept vector corresponding to the determined concept is input, the concept vector a first hidden concept layer for determining a concept evaluation weight corresponding to each of the plurality of latent concepts based on a degree of similarity with each of the plurality of latent concepts vectorized; and a read vector based on the concept learning state and the concept evaluation weight The second hidden concept layer for determining, the concept understanding ability determination layer for determining the concept understanding ability based on the concept vector and the lead vector, and the concept difficulty based on the concept vector and the concept understanding ability input to the user It is configured to include an output layer that calculates the concept understanding probability of .

Other embodiment specifics are included in the detailed description and drawings.

As the present invention shares a hidden concept layer including 'a plurality of latent concepts' constituting the item evaluation model and 'the user's concept learning state' with the concept evaluation model, the predictive performance of the two models can be mutually elevated.

In addition, the present invention predicts the probability of a user's correct answer to an item through an item evaluation model and predicts the probability of a user's concept understanding of a concept applied to an arbitrary item through a concept evaluation model, thereby providing accurate information about the learner's learning state. analysis can proceed.

Moreover, the present invention can have higher item prediction performance than when a single item evaluation model is applied by applying a concept evaluation model configured to interpret and evaluate a learner's understanding of a concept to the item evaluation system together.

More specifically, the present invention can share the learning state of the concept with the user, so the evaluation ability for items with low frequency of appearance is higher than that of a single model, and the diagnostic performance is maintained at a high level even if the evaluation of the item is repeated. It is possible to provide an item evaluation system based on an item evaluation model and a concept evaluation model.

Accordingly, the present invention can provide a customized e-learning environment for individual learners by providing an understanding of concepts related to questions and correct answers of questions output by each model by the item evaluation system.

Furthermore, by providing an item evaluation system configured to evaluate items to be presented based on the knowledge state of each learner, it is possible to provide learning efficiency similar to that of 1: 1 offline class that provides optimized learning for each individual learner. .

Effects according to the present invention are not limited by the contents exemplified above, and more various effects are included in the present invention.

1A is a schematic diagram illustrating an item evaluation system according to an embodiment of the present invention.
Figure 1b is a schematic diagram for explaining a device for evaluating items according to an embodiment of the present invention.
Fig. 1c is a schematic diagram of a user device according to an embodiment of the present invention.
1D and 1E exemplarily illustrate a display unit of a user device according to an embodiment of the present invention.
2 is a schematic flowchart for explaining a method for evaluating items according to an embodiment of the present invention.
3 illustratively illustrates the procedure of the item evaluation method according to an embodiment of the present invention.
4 is a schematic flowchart for explaining an update step of an item evaluation model applied to various embodiments of the present invention.
5 illustratively illustrates an update step of an item evaluation model applied to various embodiments of the present invention.
FIG. 6A exemplarily illustrates structures of an item evaluation model and a concept evaluation model applied to various embodiments of the present invention.
6B illustrates a configuration for updating an item evaluation model applied to various embodiments of the present invention by way of example.
7A and 7B show comparison of performances of a device for item evaluation based on an item evaluation model and a concept evaluation model applied to various embodiments of the present invention and a single item evaluation model.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various forms different from each other, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention pertains. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used for like elements.

In this document, expressions such as "has," "may have," "includes," or "may include" indicate the existence of a corresponding feature (eg, numerical value, function, operation, or component such as a part). , which does not preclude the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A and/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, Or (3) may refer to all cases including at least one A and at least one B.

Expressions such as “first,” “second,” “first,” or “second,” used in this document may modify various elements, regardless of order and/or importance, and refer to one element as It is used only to distinguish it from other components and does not limit the corresponding components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, without departing from the scope of rights described in this document, a first element may be named a second element, and similarly, the second element may also be renamed to the first element.

A component (e.g., a first component) is "(operatively or communicatively) coupled with/to" another component (e.g., a second component); When referred to as "connected to", it should be understood that the certain component may be directly connected to the other component or connected through another component (eg, a third component). On the other hand, when an element (eg, a first element) is referred to as being “directly connected” or “directly connected” to another element (eg, a second element), the element and the above It may be understood that other components (eg, a third component) do not exist between the other components.

As used in this document, the expression "configured to" means "suitable for," "having the capacity to," depending on the circumstances. ," "designed to," "adapted to," "made to," or "capable of." The term "configured (or set) to" may not necessarily mean only "specifically designed to" hardware. Instead, in some contexts, the phrase "device configured to" may mean that the device is "capable of" in conjunction with other devices or components. For example, the phrase "a processor configured (or configured) to perform A, B, and C" may include a dedicated processor (e.g., embedded processor) to perform those operations, or by executing one or more software programs stored in a memory device. , may mean a general-purpose processor (eg, CPU or application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe a specific embodiment, and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the art described in this document. Among the terms used in this document, terms defined in a general dictionary may be interpreted as having the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, an ideal or excessively formal meaning. not be interpreted as In some cases, even terms defined in this document cannot be interpreted to exclude the embodiments of this document.

Each feature of the various embodiments of the present invention can be partially or entirely combined or combined with each other, and as those skilled in the art can fully understand, various interlocking and driving operations are possible, and each embodiment can be implemented independently of each other. It may be possible to implement together in an association relationship.

For clarity of interpretation of this specification, terms used in this specification will be defined below.

The term “item evaluation model” used in the specification of the present application may be a model based on an artificial neural network algorithm configured to output a correct answer to an item, preferably a probability that a learner will answer the item correctly, by taking the item as an input.

Meanwhile, the item evaluation model may be constructed under different conditions according to the state of knowledge of each learner who is a user.

The item evaluation model may include hidden layers of “latent concepts” and “concept learning states”.

The term “latent concept” used in the present specification may refer to concepts associated with an item formed by model learning of an item evaluation model. More specifically, the size of the latent concept may be selected, and the value/size may be invariant when an appropriate embedding value is learned based on the selection. At this time, the latent concept may correspond to ' key memory' .

According to a feature of the present invention, the item evaluation model may be configured to determine, through latent concepts, a concept associated with an input item.

The term “concept learning state” used in the present specification may refer to a learner's learning state of concepts. That is, the concept learning state may be different according to the learner's knowledge state. At this time, the concept learning state may have various values depending on whether or not the questions presented to the learner are answered correctly and concepts inherent in the questions. More specifically, the concept learning state is variable according to the learner's knowledge state, and may include an absolute value for 'KEY'. At this time, the concept learning state may correspond to ' value memory ' or 'concept memory'.

According to a feature of the present invention, the concept learning state may be updated based on whether or not the learner has answered the questions correctly, and furthermore, based on the concepts of the correct questions.

According to another feature of the present invention, the item evaluation model may be configured to determine a concept that a learner associated with an item has from a concept memory through a concept learning state.

Finally, the item evaluation model may output the probability that the user answers the item correctly.

The term "concept evaluation model" used in the present specification may be a model based on an artificial neural network algorithm configured to probabilistically output a user's understanding of each concept by taking a concept determined for a question to be solved by a learner as an input. .

In this case, the concept evaluation model may be configured to share the 'latent concept' and 'concept learning state' of the item evaluation model. Accordingly, the concept evaluation model may output an understanding of the concept of the item based on the latent concept shared from the item evaluation model and the concept learning state. In other words, as the concept evaluation model shares the 'latent concept' and 'concept learning state' of the item evaluation model, it is possible to increase the interpretability of the hidden concept layer, thereby providing a detailed analysis of the learner's knowledge state. there is. Moreover, the evaluation ability for items with low frequency of appearance in the learning phase can be excellent, and even if the evaluation is repeated, the diagnostic performance can be maintained at a high level. Therefore, the item evaluation device according to various embodiments of the present invention may have better diagnostic performance than a single item evaluation model in item evaluation.

Meanwhile, the item evaluation model and the concept evaluation model disclosed in the present specification may be models based on deep learning algorithms. More specifically, the item evaluation model and concept evaluation model include dynamic key-value memory network (DKVMN), deep neural network (DNN), deep convolution neural network (DCNN), recurrent neural network (RNN), and restricted boltzmann machine (RBM). , Deep Belief Network (DBN), Single Shot Detector (SSD), and Support Vector Machine (SVM). However, it is not limited thereto.

Hereinafter, a device for evaluating items according to various embodiments of the present invention will be described in detail with reference to FIGS. 1A to 1E .

1A is a schematic diagram illustrating an item evaluation system according to an embodiment of the present invention. Figure 1b is a schematic diagram for explaining a device for evaluating items according to an embodiment of the present invention. Fig. 1c is a schematic diagram of a user device according to an embodiment of the present invention. 1d to 1e illustrate the display unit of the user device according to an embodiment of the present invention.

First, referring to FIG. 1A , an item evaluation system 1000 is a system that provides item evaluation results based on a learner's state of knowledge, and includes an item evaluation device 100, a user device 200, and an item to be evaluated. DB 300 may be included.

First, the item evaluation device 100 may include a general-purpose computer, a laptop, and/or a data server that receives evaluation target items from the evaluation target item DB 300 and performs calculations. Meanwhile, the user device 200 is configured to access a web server providing a web page related to a service for item evaluation or a mobile web server providing a mobile web site according to a user's request. It may be a device, but is not limited thereto.

More specifically, referring to FIG. 1B , the device for evaluating items 100 includes a storage unit 110 , a communication unit 120 , and a processor 130 .

First, the storage unit 110 may analyze items and store various data for providing item evaluation services to learners, for example, evaluation target items obtained from the evaluation target item DB 300 . In various embodiments, the storage unit 110 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory , a magnetic disk, and an optical disk may include at least one type of storage medium.

Next, the communication unit 120 connects the item evaluation device 100 to communicate with an external device. For example, the communication unit 120 may be connected to the user device 200 using wired/wireless communication to transmit/receive data on evaluation items. Specifically, the communication unit 120 may be connected to the user device 200 and further configured to transmit evaluation results.

The processor 130 is operatively connected to the storage unit 110 and the communication unit 120, and can perform various commands for analyzing items and providing a service for item evaluation.

Specifically, the processor 130 may receive an item through the communication unit 120 and evaluate the item based on the received item.

To this end, the processor 130 analyzes the received item to determine a concept associated with the item, predicts a concept associated with the item from a concept memory for the user, and outputs a probability of correcting the item, and an item evaluation model configured to output a probability associated with the item. A concept evaluation model configured to output an understanding of a concept is available. At this time, the item evaluation models are pre-learned models based on questions solved by the learner, and may be used to determine whether or not the evaluation item is correct according to the learner's state of knowledge.

According to aspects of the present invention, processor 130 may be configured to vectorize an item to generate an item vector, compare a similarity between the item vector and the latent concept, and determine a concept for the item based on the similarity. there is.

According to another feature of the present invention, the processor 130 determines a weight for each latent concept according to the degree of similarity, and determines whether or not the item is correct based on the concept learning state and the weight for the latent concept using an item evaluation model. It can be further configured to predict.

According to another feature of the present invention, the processor 130 determines a memory associated with an item among concept memories using an item evaluation model, determines an item-solving ability based on the associated memory, and determines an item-solving ability based on the item-solving ability. Thus, it may be further configured to predict whether the question is correct or not.

According to another feature of the present invention, the processor 130 determines the difficulty of the item, determines a memory associated with the item among the concept memories, determines the ability to solve the item based on the associated memory, and determines the difficulty of the item and the item solving ability. It may be further configured to predict whether or not to answer the question based on the ability.

According to another feature of the present invention, the processor 130 may be further configured to update the concept memory based on whether or not the answer is correct for the question received through the communication unit 120 and a concept associated with the question.

That is, the processor 130 may proceed with learner-customized item evaluation and reflect an update of the knowledge state according to the learner's learning, by using these item evaluation models and concept evaluation models. Furthermore, the processor 130 may determine and provide recommended items according to the learner's level of understanding of the concept.

In particular, as the processor 130 uses a concept evaluation model that shares a 'latent concept' in which embedding vectors for each of the concepts of the item evaluation model are stored and a 'concept learning state' of a concept memory possessed by each learner, It can provide an accurate analysis of the learner's state of knowledge.

In the following, the user device 200 will be described in detail with reference to FIG. 1C.

Referring to FIG. 1C together, the user device 200 includes a communication unit 210, a display unit 220, a storage unit 230, and a processor 240.

First, the communication unit 210 connects the user device 200 to enable communication with an external device. For example, the communication unit 210 may be connected to the item evaluation device 100 using wired/wireless communication to transmit/receive data on evaluation items. Specifically, the communication unit 210 may be further configured to connect with the item evaluation device 100 and receive evaluation results.

The display unit 220 may display various types of content (eg, text, image, video, icon, banner, or symbol) to the user. Furthermore, the display unit 220 may display various interface screens for displaying the user's item evaluation result data determined by the item evaluation device 100 . For example, the display unit 220 may be configured to display a user's correct answer to a question, an understanding of a concept, and a recommended question related to a concept lacking in understanding.

In various embodiments, the display unit 220 may include a touch screen, and for example, a touch using an electronic pen or a part of the user's body, a gesture, a proximity, a drag, or a swipe A swipe or hovering input may be received.

The storage unit 230 may store various data used to request item evaluation and provide a user interface for displaying result data for item evaluation. In various embodiments, the storage unit 230 may be a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (for example, SD or XD memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) , a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. Meanwhile, the item evaluation device 100 may operate in association with a web storage that performs the storage function of the storage unit 230 on the Internet.

The processor 240 is operatively connected to the communication unit 210, the display unit 220, and the storage unit 230, and can perform various commands for providing a user interface for displaying result data for item evaluation. .

Hereinafter, a display unit of a user device according to an embodiment of the present invention will be described in detail with reference to FIGS. 1D and 1E.

First, referring to FIG. 1D , the display unit 220 of the user device 200 recommends and outputs items associated with a concept in which the degree of understanding of the concept determined by the item evaluation device 100 is equal to or lower than a predetermined level. More specifically, the learner's understanding of concept a determined by the item evaluation device 100 is 10%, the learner's understanding of concept b is 95%, and the learner's understanding of concept c is 70%. At this time, as a recommended question, a question related to concept a (eg, calculation of a fraction), which is relatively poorly understood, may be determined and provided to the user through the display unit 220 . On the other hand, referring to FIG. 1E together, items related to concept b (eg, equation, multiplication of numbers) having a relatively high level of understanding may be excluded from the recommended items.

Accordingly, learners can be provided with a customized e-learning environment for each individual as intensive learning is possible for weak concepts.

Hereinafter, item evaluation methods according to various embodiments of the present invention will be described with reference to FIGS. 2 and 3 .

2 is a schematic flowchart for explaining a method for evaluating items according to an embodiment of the present invention. 3 illustratively illustrates the procedure of the item evaluation method according to an embodiment of the present invention.

First, referring to FIG. 2 , according to the item evaluation method according to an embodiment of the present invention, an item to be solved by the user is received (S210). Next, the concept of the item is determined by the item evaluation model (S220), and whether or not the item is correct is predicted (S230). Then, the level of understanding of the concept related to the item is predicted by the concept evaluation model (S240), and whether or not the item is answered correctly and the degree of understanding of the concept related to the item is provided (S250).

More specifically, in the step of receiving the item (S210), the item may be received from an evaluation item providing server such as a question bank, or the item may be input through an input unit of the item evaluation device.

According to a feature of the present invention, a step of inputting a selection for at least one of a plurality of questions may be further performed before the step of receiving the question ( S210 ) is performed.

Accordingly, in the step of receiving the question (S210), the question selected by the user may be received.

Next, in the step of determining the concept of the item (S220), the concept of the item may be determined.

According to a feature of the present invention, in the step of determining the concept of an item (S220), the item is vectorized so that an item vector is generated, the similarity between the item vector and the latent concept is compared, and based on the similarity, the item is vectorized. concept can be determined.

For example, referring to FIG. 3 together, an evaluation item 310 is received through the step of receiving the item (S210), and the evaluation item is evaluated by the item evaluation model 400 in the step of determining the concept of the item (S220). Question 310 is entered. Then, the evaluation items 310 are embedding vectorized to generate the item vectors 312 . Next, similarities with the embedding vectors for each of the concepts in the latent concept 410 of the item evaluation model 400 are compared.

Meanwhile, according to another feature of the present invention, a weight for each latent concept may be determined after the concept for the item is determined (S220).

For example, referring to FIG. 3 again, weights may be set to a predetermined level or higher for concepts related to evaluation items 310, and weights may be applied to concepts that have a relatively low correlation with evaluation items 310. may be set below a predetermined level.

Referring back to FIG. 2 , in the step of predicting whether or not the question is correct (S230), the probability of whether the user will answer the question correctly may be calculated.

According to a feature of the present invention, in the step of predicting whether or not the item is correct (S230), whether the item is correct or not may be predicted based on the concept learning state and the weight determined for the latent concept by the item evaluation model.

According to a feature of the present invention, in the step of predicting whether or not the question is correct (S230), a memory associated with the item is determined from among the concept memories by the item evaluation model, and the user's ability to solve the item is determined based on the previously associated memory. And, based on the ability to solve the question, whether or not the answer is correct for the question can be predicted.

According to another feature of the present invention, in the step of predicting whether the question is correct or not (S230), the difficulty of the item is determined by the item evaluation model, the memory associated with the item is determined among the concept memories, and based on the associated memory The user's problem-solving ability may be determined, and a correct answer to the item may be predicted based on the difficulty of the item and the item-solving ability.

For example, referring to FIG. 3 together, in step S230 of predicting whether or not the question is correct, based on the concept learning state 420 corresponding to the concept memory that is the user's knowledge state and the weight for each latent concept , a VALUE value corresponding to the KEY may be set. Next, the item solving ability 316 of the user may be determined by the item solving ability determination network 430 . At this time, the item evaluation model 400 may determine the item difficulty 314 of the evaluation item 310 based on the item vector 312 . Meanwhile, the item difficulty 314 may be determined by the item evaluation model 400 and an independent item difficulty prediction model (not shown) and transmitted to the item evaluation model 400 . Then, based on the item difficulty 314 and the user's item solving ability 316, a correct answer probability 318, which is a probability that the user corrects the evaluation item 310, may be determined to be 80%.

Referring back to FIG. 2 , in step S240 of predicting the level of understanding of a concept related to an item, a concept evaluation model configured to share at least one of a latent concept and a concept learning state and predict a level of understanding of a concept related to the item provides the user with a concept evaluation model. The level of understanding of the concept of can be predicted.

For example, referring back to FIG. 3 , concept evaluation model 500 can be configured to share latent concepts 510 and concept learning status 520 for a user with item evaluation model 400 . Accordingly, it is possible to increase the interpretability of the hidden concept layer, so that a detailed analysis of the learner's state of knowledge may be possible. Furthermore, according to the sharing of the hidden layer, the evaluation ability for items with low frequency of occurrence can be excellent in the learning stage, and even if the evaluation is repeated, the diagnostic performance can be maintained at a high level. Meanwhile, in the step of predicting the degree of understanding of the concept related to the item (S240), the concept 320 associated with the evaluation item 310 is input to the concept evaluation model 500, and the related concept 320 is embedded and vectorized to obtain a related concept ( 320) can obtain concept vectors 322 for each. Then, the similarity with the embedding vectors for each of the concepts in the latent concept 510 is compared, and based on the concept learning state 520 corresponding to the concept memory that is the user's knowledge state and the weight for each latent concept , a VALUE value corresponding to the KEY may be set. The user's concept understanding ability 326 can then be determined by the concept understanding ability determination network 530 . At this time, the concept evaluation model 500 may determine a concept difficulty 324 for the concept 320 associated with the evaluation item 310 based on the concept vector 322 . Meanwhile, the concept difficulty 324 may be determined by an item difficulty prediction model (not shown) independent of the concept evaluation model 500 and transmitted to the concept evaluation model 500 . Then, based on the concept difficulty 324 and the user's concept understanding ability 326, the concept understanding probabilities 328 of concepts a, b, and c for the evaluation item 310 are 60%, 90%, and 85%, respectively. can be determined by

On the other hand, the step of predicting the degree of understanding of the concept related to the item (S240) is not limited to the above, and is performed simultaneously with the step of determining the concept of the item (S220) or predicting whether the question is correct (S230) It could be.

Referring back to FIG. 2 , in step S250 in which whether an item is correct and the level of understanding of a concept related to the item is provided, whether or not the item is correct is predicted (S230) and the degree of understanding of the concept related to the item is predicted ( S240) Whether or not the answer is correct for the question determined in each step and the degree of understanding of the concept are provided.

For example, referring to FIG. 3 , in a step S250 in which whether an item is answered correctly and the level of understanding of a concept related to the item is provided, the probability of correct answer output by the item evaluation model 400 and the concept evaluation model 500, respectively 318 and conceptual understanding 328 may be provided.

On the other hand, according to the feature of the present invention, after the step (S250) of providing whether or not the question is correct and the degree of understanding of the concept related to the question, among a plurality of additional questions based on the predicted user's degree of understanding of the concept, a predetermined level or less is reached. A step of determining an item associated with a concept having an understanding of , and a step of providing a related item may be further performed.

Furthermore, based on the predicted level of understanding of the user's concept, determining a high-understanding item associated with a concept having a level of understanding above a predetermined level among a plurality of additional items, and selecting a high-understanding item among the plurality of additional items. A step of providing the remaining questions may be further performed.

That is, learners can be provided with questions related to concepts they lack, so that learners' learning efficiency can be increased.

Meanwhile, in the item evaluation method according to various embodiments of the present invention, analysis of the item may be performed based on Equation 1 below.

[Equation 1]

Here, BCE refers to 'binary cross entropy', q _t refers to an evaluation item corresponding to time t, a _t refers to the user's answer corresponding to time t, and Ω _t refers to a concept related to q _t refers to a set In addition, c indicates a concept, DKVMN can be defined as an item solution prediction part, and ExpM can be defined as a concept understanding level prediction part. At this time, Ω _t is 1, that is, the concept associated with the item may be one, but is not limited thereto.

According to another feature of the present invention, the update of the knowledge state may be reflected according to the user's correct answer to the question and the predicted level of understanding of the concept.

Hereinafter, updating of the item evaluation model will be described in detail with reference to FIGS. 4 and 5 .

4 is a schematic flowchart for explaining an update step of an item evaluation model applied to various embodiments of the present invention. 5 illustratively illustrates an update step of an item evaluation model applied to various embodiments of the present invention.

First, referring to FIG. 4 , in the item evaluation model, an answer to an item is received from a user (S410), and a concept learning state may be updated based on the received correct answer and a concept associated with the item (S420).

For example, referring to FIG. 5 together, in step S410 of receiving answers to questions, an evaluation item and a correct answer set 620 may be received. Next, in the step of updating the concept learning state (S420), the evaluation questions and the correct answer set 620 are vectorized into an embedding vector, and then an erase vector and an add vector are generated, respectively. Then, after the evaluation item 610 is input to the item evaluation model 400, the weight determined according to the degree of similarity with the latent concept 410 is multiplied for each of the deleted vector and the added vector. Then, after the existing concept learning state (t) 420 is reflected in the deleted vector, the two vectors are merged to obtain a newly updated concept learning state (t+1) 420'. The updated concept learning state 420 ′ may be applied to the evaluation of an item to be presented after the evaluation item 610 .

Hereinafter, with reference to FIGS. 6A and 6B , the structure of an item evaluation model and concept evaluation model used in various embodiments of the present invention and a configuration for updating the item evaluation model will be described in detail.

FIG. 6A exemplarily illustrates structures of an item evaluation model and a concept evaluation model applied to various embodiments of the present invention. 6B illustrates a configuration for updating an item evaluation model applied to various embodiments of the present invention by way of example.

_t ^q에 대응하는 학습자에 대한 문항 풀이 능력 (316) 이 결정된다. 그 다음, 문항 벡터 (312) 에 기초하여 결정된

_q에 대응하는 문항 난이도 (314) 및 문항 풀이 능력 (316) 에 기초하여 p_t에 대응하는 정답 확률 (318) 이 최종적으로 산출될 수 있다. 한편, 개념 평가 모델 (500) 및 문항 평가 모델 (400) 은, 싱크가 맞춰진 잠재 개념 (410, 510), 개념 학습 상태 (410, 520) 를 포함할 수 있다. First, referring to FIG. 6A , the item evaluation model 400 has a latent concept 410, which is a 'key memory' that stores embedding vectors for each concept, is variable according to the state of knowledge of the learner, and has a 'Key'. It can be composed of a concept learning state 420 corresponding to 'value memory (or concept memory)' including corresponding absolute values and a problem solving ability determination network 430 that determines the item solving ability of the user. At this time, the evaluation item 310 corresponding to q _t is converted into an embedding vector to generate an item vector 312 corresponding to k _t , and then a vector for each concept in the latent concept 410 corresponding to C and an item vector ( 312), a weight 412 corresponding to W _t ^q may be determined. Then, based on the concept learned state 420 and the weights 412, a read vector 422 corresponding to r _t ^q is generated. Then, based on the item vector 312 and the read vector 422, a network 430 predicting the learner's item solving ability corresponding to the f _t ^q feature vector is generated,

The item solving ability (316) of the learner corresponding to _t ^q is determined. Then, determined based on the item vector 312

Based on the item difficulty 314 and the item solving ability 316 corresponding to _q , a correct answer probability 318 corresponding to p _t may be finally calculated. Meanwhile, the concept evaluation model 500 and the item evaluation model 400 may include synchronized latent concepts 410 and 510 and concept learning states 410 and 520 .

_t ^c에 대응하는 학습자에 대한 사용자의 개념 이해 능력 (326) 이 결정된다. 그 다음, 개념 벡터 (322) 에 기초하여 결정된

_c에 대응하는 개념 난이도 (324) 및 개념 이해 능력 (326) 에 기초하여 m_t에 대응하는 개념 이해 확률 (318) 이 최종적으로 산출될 수 있다.That is, the concept evaluation model 500 may be composed of a latent concept 510 shared from the item evaluation model 400, a concept learning state 520, and a concept comprehension ability determination network 530. At this time, the concept 320 associated with the item corresponding to C _t , to which the concept is tagged by the expert, is embedding vectorized to generate a concept vector 322 corresponding to h, and then in the latent concept 510 corresponding to C A weight 512 corresponding to W _t ^c may be determined according to a degree of similarity between a vector for each concept and the concept vector 322 . Then, based on the concept learned state 520 and the weights 512, a read vector 522 corresponding to r _t ^c is generated. Then, based on the concept vector 322 and the read vector 522, a network 530 predicting the learner's concept comprehension ability corresponding to the f _t ^c feature vector is created;

The user's concept comprehension ability 326 for the learner corresponding to _t ^c is determined. Then, based on the concept vector 322,

Based on the concept difficulty 324 and the concept understanding ability 326 corresponding to _c , the concept understanding probability 318 corresponding to m _t can be finally calculated.

In this case, the item evaluation model 400 and the concept evaluation model 500 may be models based on a plurality of DKVMN algorithms sharing a hidden concept layer, but are not limited thereto.

Meanwhile, the concept learning state 420 of the item evaluation model 400 may be updated based on whether the user answered the question correctly and the concept related to the item, and the updated concept learning state 420' corresponds to the concept evaluation model (500).

More specifically, referring to FIG. 6B, for the update of the concept learning state 420, the previously learned qt embedding vector and a _t , which is the user's response to the question, are combined, corresponding to (q _t , a _t ) A set of evaluation questions and correct answers 620 are non-linearly mapped to generate a correct answer vector 622 corresponding to v _t . At this time, the item answer vector 622 may be generated by Equation 2 below.

[Equation 2]

v _t = tanh( FC ( A (q _t ),a _t ))

That is, the update does not generate and learn a new embedding matrix for the (q _t , a _t ) evaluation items and the correct answer set 620 , but instead uses the vectorized item vector 312 as the embedding of the existing evaluation items 310. It starts by combining with a _t . Accordingly, the item evaluation model is updated without unnecessary overhead, and more stable learning (for example, in the DKVMN model) can be performed. Next, sigma (σ) is applied to the correct item vector 622 to generate a deleted vector corresponding to et, and the hyperbolic tangent (tanh) is applied to the correct item vector 632 to generate an additional vector 634. . Then, each of the deleted vector 632 and the added vector 634 is multiplied by a weight 412 determined according to the degree of similarity with the latent concept 410 . Then, after the existing concept learning state 420 is reflected in the deletion vector 632, the values of the two vectors 632 and 634 are combined to obtain a newly updated concept learning state corresponding to S _t+1 ( 420') can be obtained. The updated concept learning state 420 ′ may be applied to the evaluation of an item to be presented after the evaluation item 310 . Furthermore, it can be shared with the concept evaluation model 500.

Comparative Example: Performance comparison of item evaluation model + concept evaluation model and single item evaluation model according to various embodiments of the present invention

Hereinafter, evaluation results of the item evaluation device according to various embodiments of the present invention will be described with reference to FIGS. 7A and 7B.

Referring to FIG. 7A , an evaluation result for a single item evaluation model based on the DKVMN algorithm is shown. Furthermore, referring to FIG. 7B , an evaluation result of an item evaluation device based on an item evaluation model and a concept evaluation model applied to various embodiments of the present invention is shown.

Referring to FIGS. 7A (a) and 7B (a), the accuracy of analysis of the item evaluation model + concept evaluation model and the single item evaluation model according to various embodiments of the present invention is shown to be at a similar level. However, further referring to FIGS. 7A (b) and 7B (B), the cross-entropy loss value of the item evaluation model + concept evaluation model is significantly lower than that of the single item evaluation model. appear. Moreover, referring to the AUC value corresponding to the performance of the models of FIGS. 7A (c) and 7B (c), in the case of a single item evaluation model, it is 0.8162, but decreases to 0.76 or less as the evaluation progresses. However, in the case of the item evaluation model + concept evaluation model, after a high AUC value of 0.8269 is measured, it appears to be maintained without falling below 0.8. That is, these results may mean that the item evaluation model + concept evaluation model has better predictive performance than a single item evaluation model, and that the performance is maintained even if item evaluation is repeated.

At this time, in the single item evaluation model, the ability to evaluate items for concepts with a low frequency of appearance may be low due to overfitting due to items having concepts that frequently appear in the learning stage. That is, the performance of a single item evaluation model may deteriorate depending on the balance of item items for each concept.

On the other hand, as the item evaluation model and the concept evaluation model share the learning state of the concept for the user, the evaluation ability for the item for a concept with a low frequency of appearance is higher than that of the single model, and the diagnostic performance is high even if the evaluation of the item is repeated. level can be maintained.

Therefore, the item evaluation device according to various embodiments of the present invention based on the two models of the item evaluation model and the concept evaluation model has an effect of performing a detailed analysis of the learner's learning state.

Accordingly, the present invention can provide a customized e-learning environment for individual learners by providing an understanding of concepts related to questions and correct answers of questions output by each model by the item evaluation system.

Although one embodiment of the present invention has been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and may be variously modified and implemented without departing from the technical spirit of the present invention. there is. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. The protection scope of the present invention should be construed according to the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: device for evaluating questions
110, 230: storage unit
120, 210: communication department
130, 240: processor
200: user device
220: display unit
300: Evaluation target item DB
310, 610: evaluation questions
312, 612: question vector
314, 614: Question difficulty
316, 616: Ability to solve questions
318, 618: probability of correct answer
322: concept vector
324: Concept Difficulty
326: Ability to understand concepts
328: concept understanding probability
410, 510: latent concept
412, 512: weight
420, 520: concept learning state
422, 522: lead vector
420': updated concept learning status
430: Item Solving Ability Determination Network
530: Concept Understanding Ability Determination Network
1000: item evaluation system

Claims (8)

As an item evaluation method implemented by a processor,
Receiving an item to be solved by a user from a user device;
determining a latent concept for the item;
predicting a user's correct answer probability for the item using an item evaluation model composed of a plurality of latent concepts, a hidden concept layer including a user's concept learning state, and an item-solving ability determination layer;
sharing the hidden concept layer to a concept evaluation model for predicting a concept understanding probability of a user;
predicting a concept understanding probability of a user for a determined concept using the concept evaluation model composed of a shared hidden concept layer and a concept comprehension capability determination layer; and
providing predicted correct answer probabilities and concept understanding probabilities to the user device; including,
The item evaluation model,
an input layer into which item vectors corresponding to the items are input;
a first hidden concept layer for determining an item evaluation weight corresponding to each of the plurality of latent concepts based on a similarity between the item vector and the plurality of vectorized latent concepts;
a second hidden concept layer for determining a lead vector based on the concept learning state and the item evaluation weight;
the item solving ability determination layer determining item solving ability based on the item vector and the read vector; and
an output layer that calculates a probability of correct answer of the user by inputting the item difficulty and the item solving ability based on the item vector; including,
The concept evaluation model,
an input layer into which a concept vector corresponding to the determined concept is input;
a first hidden concept layer for determining a concept evaluation weight corresponding to each of the plurality of latent concepts based on a degree of similarity between the concept vector and each of the plurality of vectorized latent concepts;
a second hidden concept layer for determining a read vector based on the concept learning state and the concept evaluation weight;
the concept comprehension capability determining layer for determining a concept comprehension capability based on the concept vector and the lead vector; and
an output layer for calculating a concept understanding probability of the user by inputting the concept difficulty and the concept understanding ability based on the concept vector; Including, item evaluation method. According to claim 1,
The concept learning state,
An item evaluation method, which is defined as a concept memory consisting of absolute values of concept vectors obtained by vectorizing latent concepts of the correct answer items corrected by the user. According to claim 2,
receiving an answer to the question from the user device; and
and updating the concept memory based on whether the received answer is correct and a latent concept associated with the item. According to claim 1,
After the providing step,
Determining an item with high comprehension associated with a concept having a level of understanding equal to or higher than a predetermined level, among a plurality of additional items, based on the predicted level of understanding of the concept; and
providing the remaining items except for the items with high level of understanding among the plurality of additional items to the user device; Further comprising, item evaluation method. A communication unit configured to receive an arbitrary question to be solved by a user from a user device; and
A processor operably connected to the communication unit;
the processor,
Determine the latent concept for the item, and predict the probability of the user's answer to the item using an item evaluation model composed of a plurality of latent concepts, a hidden concept layer including the user's concept learning state, and a question solving ability determination layer. and sharing the hidden concept layer to a concept evaluation model for predicting the user's concept understanding probability, and using the concept evaluation model composed of the shared hidden concept layer and the concept comprehension ability determination layer to determine the user's understanding of the determined concept. predict a concept understanding probability and provide the predicted correct answer probability and concept understanding probability to the user device;
The item evaluation model,
an input layer into which item vectors corresponding to the items are input;
a first hidden concept layer for determining an item evaluation weight corresponding to each of the plurality of latent concepts based on a degree of similarity between the item vector and the plurality of vectorized latent concepts;
a second hidden concept layer for determining a lead vector based on the concept learning state and the item evaluation weight;
the item solving ability determination layer determining item solving ability based on the item vector and the read vector; and
an output layer that calculates a probability of correct answer of the user by inputting the item difficulty and the item solving ability based on the item vector; including,
The concept evaluation model,
an input layer into which a concept vector corresponding to the determined concept is input;
a first hidden concept layer for determining a concept evaluation weight corresponding to each of the plurality of latent concepts based on a degree of similarity between the concept vector and each of the plurality of vectorized latent concepts;
a second hidden concept layer for determining a read vector based on the concept learning state and the concept evaluation weight;
the concept comprehension capability determining layer for determining a concept comprehension capability based on the concept vector and the lead vector; and
an output layer for calculating a concept understanding probability of the user by inputting the concept difficulty and the concept understanding ability based on the concept vector; Item evaluation device comprising a. According to claim 5,
The concept learning state,
A device for evaluating items, which is defined as a concept memory consisting of absolute values of concept vectors obtained by vectorizing the concepts of the correct answer items corrected by the user. According to claim 6,
The communication department,
Further configured to receive an answer to the question from the user device,
the processor,
The device for evaluating an item, further configured to update the concept memory based on whether the received answer is correct or not and a concept associated with the item. According to claim 5,
the processor,
Based on the predicted level of understanding of the concept, among a plurality of additional items, a high level of understanding related to a concept having a level of understanding above a predetermined level is determined;
A device for evaluating items further configured to provide the user device with the remaining items excluding the items with high understanding among the plurality of additional items.

Images