KR20210125250A - Method for questions assessment and device for questions assessment using the same - Google Patents

Abstract

The present invention provides a question evaluation method and a device using the same. The question evaluation method realized by a processor comprises: a step of receiving a question to be solved by a user; a step of using a question evaluation model constructed to determine the concept of a question based on latent concepts to determine the concept for the question; a step of using the question evaluation model further constructed to predict the correct answer status of a question based on a concept learning state of the user and a determined concept to predict the correct answer status of the question; a step of using a concept evaluation model constructed to share at least one between latent concepts and the concept learning state and predict a concept understanding degree for a question to predict the understanding degree of a concept associated with the question for the user; and a step of providing the correct answer status of the question and the understanding degree of the concept associated with the question.

Description

Item evaluation method and device using the same

The present invention relates to an item evaluation method and a device using the same, and more particularly, to an item evaluation method and item evaluation using the same, configured to evaluate whether the user has the correct answer to the item to be solved and the understanding of the concept of the item It's about devices.

In recent years, various learning methods have emerged that allow learners to actually face-to-face with the teacher, breaking away from the conventional learning method in which learning is performed under the guidance of the teacher, and allowing learning to proceed according to the learner's situation regardless of time and place.

In particular, the e-Learning learning method is an active learning method that allows learners to lead their own learning in the ubiquitous information and communication environment using the Internet, away from the classroom-centered learning environment. There is a trend that the proportion of the method is gradually expanding in the overall education, and various types of e-learning learning techniques have been suggested according to this trend.

However, these conventional techniques are focused only on simply delivering learning items to the learner, and in any order according to the learner's understanding changed by learning and the learning goal actively selected by the learner. There is no description of whether to proceed with learning, what kind of learning content to select, and what level of difficulty to provide to the learner.

In particular, as the conventional e-learning learning environment has limitations in performing efficient learning in consideration of the individual learner's knowledge ability, the learning efficiency may be lower than that of a 1:1 offline class that provides optimized learning for each learner. have.

The description underlying the invention has been prepared to facilitate understanding of the invention. It should not be construed as an admission that the matters described in the background technology of the invention exist as prior art.

On the other hand, the inventors of the present invention, as a way to overcome the limitations of the conventional learning system as described above, focused on the development of an educational data mining tool for predicting individual learner performance and advising the learner's learning. . In particular, the inventors of the present invention have noted that knowledge tracing may be an important factor in improving the learning ability of each student.

In this case, the knowledge tracking is a task of modeling the student's knowledge state, and may be expressed as a scalar value or a vector indicating a master level of knowledge components (KC), for example, a student ability level.

Therefore, the inventors of the present invention were able to recognize that in an e-learning learning environment based on the estimated learner's knowledge state, the learner's achievement level can be better understood and learning materials can be selected accordingly. Moreover, the inventors of the present invention could be expected to be able to prepare a better learning plan for maximizing learning efficiency by applying a learning analysis tool to an e-learning learning environment.

As a result, the inventors of the present invention have developed an item evaluation system configured to evaluate items provided in the e-learning learning environment based on the learner's knowledge status.

At this time, the inventors of the present invention tried to apply to the item evaluation system an item evaluation model configured to predict the user's correct answer to an item and a concept evaluation model configured to evaluate the degree of understanding of the concepts of the item.

More specifically, the inventors of the present invention determine a concept for an item based on latent concepts, and evaluate an item configured to predict whether the item is correct or not based on the determined concept and the concept learning state for the learner The model could be applied to the item evaluation system.

Furthermore, the inventors of the present invention intend to apply the item evaluation model to the item evaluation system, which is configured to share at least one of a latent concept and a concept learning state, and predict the concept understanding level for the item.

At this time, the inventors of the present invention share 'latent concept' and/or 'concept learning state', which is a hidden concept layer of the item evaluation model, to increase the interpretability of these layers. By applying it, it could be expected that an accurate analysis of the learner's learning status could be carried out.

Furthermore, the inventors of the present invention found that the item evaluation system had higher item prediction performance than when a single item evaluation model was applied by applying the concept evaluation model configured to interpret and evaluate the degree of understanding of the learner together with the item evaluation system. Could.

More specifically, the single item evaluation model may have poor evaluation ability of items for concepts with low frequency of appearance due to overfitting due to items with concepts frequently appearing in the learning stage. That is, a single item evaluation model may have poor performance depending on the balance of questions for each concept.

On the other hand, in the item evaluation system of the present invention, which consists of an item evaluation model and a concept evaluation model, as the concept learning state for users is shared, the evaluation ability for the items of the concept with low frequency of appearance is higher than that of the single model, and the Even if the evaluation is repeated, the diagnostic performance can be maintained at a high level.

Therefore, the inventors of the present invention can provide a level of understanding of concepts related to the correct answer of the question output by each model by the item evaluation system and the question, thereby providing a customized e-learning learning environment for each learner. could be expected more.

Accordingly, the problem to be solved by the present invention is to determine the concept of the received item using the item evaluation model, predict whether the item is correct or not, and predict the understanding of the concept related to the item using the concept evaluation model. It is to provide a constructed item evaluation method and a device using the same.

The problems of the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above problems, an item evaluation method according to an embodiment of the present invention is provided. The method is an item evaluation method implemented by a processor, using an item evaluation model configured to determine a concept for an item based on a step of receiving an item to be solved by a user, and latent concepts. Predicting whether the question is correct by using an item evaluation model further configured to predict whether the question is correct or not based on the determined concept and the concept learning state for the user, predicting whether the question is correct, potential concept and concept learning state predicting the degree of understanding of the concept related to the item for the user by using a concept evaluation model configured to share at least one of and predict the degree of understanding of the concept for the item, and whether the item is correct or not and the concept related to the item providing an understanding.

According to a feature of the present invention, the item evaluation model may be further configured to probabilistically predict whether an item is correct or not. In this case, the concept evaluation model may be further configured to probabilistically predict the degree of understanding of the concept related to the item. In addition, the providing may include providing a probability of correct answer to the question and a probability of understanding a concept.

According to another feature of the present invention, the determining of the concept for the item includes: vectorizing the item to generate an item vector, comparing the item vector and the similarity with the latent concept, and based on the similarity It may include determining a concept for the item.

According to another feature of the present invention, the item evaluation method may further include, after determining the concept, determining a weight for each potential concept according to the degree of similarity. In this case, predicting whether the item is correct may include predicting whether the item is correct or not based on the concept learning state and the weight for the latent concept using the item evaluation model.

According to another feature of the present invention, the concept learning state may be defined as a concept memory composed of an absolute value of a concept vector obtained by vectorizing a concept of a correct answer item matched by a user.

According to another feature of the present invention, predicting whether an item is correct may include determining a memory associated with an item from among conceptual memories using an item evaluation model, and determining an item solving ability based on the associated memory. It may include a step of predicting whether or not an answer to the question is correct based on the step and the question-solving ability.

According to another feature of the present invention, the item evaluation model may be further configured to predict whether the item is correct or not based on the difficulty of the item and the user's ability to solve the item. In this case, the step of predicting whether an item is correct may include determining the difficulty level of the item, determining the memory associated with the item from among concept memories, determining the item solving ability based on the associated memory, and the difficulty level of the item and predicting whether an answer to the question is correct based on the question-solving ability.

According to another feature of the present invention, the item evaluation method further includes: receiving an answer to the item from the user; can do.

According to another feature of the present invention, the item evaluation method determines, after the providing step, an item related to a concept having a level of understanding below a predetermined level among a plurality of additional items based on the predicted degree of understanding of the user's concept. It may further include the step of, and the step of providing a related item.

According to another feature of the present invention, the item evaluation method, after the step of providing, based on the predicted degree of understanding of the user's concept, among a plurality of additional items, the degree of understanding associated with the concept having a degree of understanding above a predetermined level is high The method may further include determining the items and providing the remaining items except for items with high understanding among the plurality of additional items.

In order to solve the above problems, there is provided a device for item evaluation according to an embodiment of the present invention. The device includes a communication unit configured to receive a question to be solved by a user, and a processor. At this time, the processor determines the concept of the item using an item evaluation model configured to determine the concept of the item based on latent concepts, and the concept of the item based on the determined concept and the concept learning state for the user Using the item evaluation model further configured to predict the correct answer, predict whether the item is correct or not, share at least one of the latent concept and the concept learning state, and use the concept evaluation model configured to predict the concept understanding of the item to the user It is configured to predict the degree of understanding of the concepts related to the question, and to provide the correct answer to the question and the degree of understanding of the concept related to the question.

According to a feature of the present invention, the processor may be further configured to vectorize an item to generate an item vector, compare the item vector and the degree of similarity with the latent concept, and determine a concept for the item based on the similarity. .

According to another feature of the present invention, the processor determines a weight for each potential concept according to the degree of similarity, and predicts whether the question is correct based on the concept learning state and the weight for the latent concept by using the item evaluation model. can be further configured.

According to another feature of the present invention, the concept learning state may be defined as a concept memory composed of an absolute value of a concept vector obtained by vectorizing a concept of a correct answer item matched by a user.

According to another feature of the present invention, the processor determines a memory associated with an item from among concept memories using the item evaluation model, determines the item solving ability based on the associated memory, and based on the item solving ability It may be further configured to predict whether the answer to the question is correct.

According to another feature of the present invention, the item evaluation model may be configured to predict whether the item is correct or not based on the difficulty of the item and the user's ability to solve the item. At this time, the processor determines the difficulty of the item, determines the memory associated with the item among the conceptual memories, determines the item solving ability based on the associated memory, and provides a correct answer to the item based on the difficulty level of the item and the item solving ability It may be further configured to predict whether or not

According to another feature of the present invention, the communication unit may be further configured to receive an answer to the question from the user. In this case, the processor may be further configured to update the concept memory based on whether the received answer is correct, and the concept associated with the question.

According to another feature of the present invention, the processor is further configured to determine an item related to a concept having a degree of understanding below a predetermined level from among a plurality of additional items based on the predicted degree of understanding of the user's concept, and to provide the related item can be configured.

The details of other embodiments are included in the detailed description and drawings.

The present invention can provide an item evaluation system with a concept evaluation model configured to share at least one of a latent concept and a concept learning state with the item evaluation model, and predict the degree of concept understanding of the item.

The present invention is an item evaluation system to which a concept evaluation model that increases the interpretability of these layers by sharing 'latent concept' and/or 'concept learning state', which is a hidden concept layer of the item evaluation model, is applied. This has the effect of enabling accurate analysis of the learner's learning state.

Furthermore, the present invention can have higher item prediction performance than when a single item evaluation model is applied by applying a concept evaluation model configured to interpret and evaluate the degree of understanding of a learner in the item evaluation system together.

More specifically, the present invention can share the concept learning state for users, so that the evaluation ability for the items of the concept with low frequency of appearance is higher than that of the single model, and the diagnosis performance is maintained at a high level even if the evaluation of the items is repeated It is possible to provide an item evaluation system based on an item evaluation model and a concept evaluation model.

Accordingly, the present invention can provide a level of understanding of concepts related to the correct answer to the question output by each model by the item evaluation system and the question, thereby providing a customized e-learning learning environment for each learner.

Furthermore, by providing an item evaluation system configured to evaluate the items to be presented based on the knowledge status of each learner, it is possible to provide a learning efficiency similar to that of a 1:1 offline class that provides optimized learning for each learner. .

The effect according to the present invention is not limited by the contents exemplified above, and more various effects are included in the present invention.

1A is a schematic diagram illustrating an item evaluation system according to an embodiment of the present invention.
1B is a schematic diagram illustrating a device for item evaluation according to an embodiment of the present invention.
1C is a schematic diagram of a user equipment according to an embodiment of the present invention;
1D and 1E exemplarily show a display unit of a user device according to an embodiment of the present invention.
2 is a schematic flowchart for explaining a method for evaluating an item according to an embodiment of the present invention.
3 exemplarily shows a procedure of an item evaluation method according to an embodiment of the present invention.
4 is a schematic flowchart for explaining an update step of an item evaluation model applied to various embodiments of the present invention.
5 exemplarily shows an update step of an item evaluation model applied to various embodiments of the present invention.
6A exemplarily illustrates the structure of an item evaluation model and a concept evaluation model applied to various embodiments of the present invention.
6B exemplarily illustrates a configuration for updating an item evaluation model applied to various embodiments of the present invention.
7A and 7B are diagrams illustrating comparison of the performance of a device for item evaluation based on an item evaluation model and a concept evaluation model applied to various embodiments of the present invention and a single item evaluation model.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims. In connection with the description of the drawings, like reference numerals may be used for like components.

In this document, expressions such as "has," "may have," "includes," or "may include" refer to the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features.

In this document, expressions such as "A or B," "at least one of A and/and B," or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used herein, expressions such as "first," "second," "first," or "second," may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components. For example, the first user equipment and the second user equipment may represent different user equipment regardless of order or importance. For example, without departing from the scope of the rights described in this document, the first component may be named as the second component, and similarly, the second component may also be renamed as the first component.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component) When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

As used herein, the expression "configured to (or configured to)" depends on the context, for example, "suitable for," "having the capacity to ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” in hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase “a processor configured (or configured to perform) A, B, and C” refers to a dedicated processor (eg, an embedded processor) for performing the operations, or by executing one or more software programs stored in a memory device. , may mean a generic-purpose processor (eg, a CPU or an application processor) capable of performing corresponding operations.

Terms used in this document are only used to describe specific embodiments, and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meanings as commonly understood by one of ordinary skill in the art described in this document. Among the terms used in this document, terms defined in a general dictionary may be interpreted with the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in this document, ideal or excessively formal meanings is not interpreted as In some cases, even terms defined in this document cannot be construed to exclude embodiments of this document.

Each feature of the various embodiments of the present invention may be partially or wholly combined or combined with each other, and technically various interlocking and driving are possible, as will be fully understood by those skilled in the art, and each embodiment may be independently implemented with respect to each other, It may be possible to implement together in a related relationship.

For clarity of interpretation of the present specification, terms used herein will be defined below.

The term “item evaluation model” used in the present specification may be an artificial neural network algorithm-based model configured to output whether the question is correct or not, preferably, the probability that the learner will correct the correct answer to the question by inputting the question as an input.

Meanwhile, the item evaluation model may be constructed under different conditions according to the knowledge state of each learner who is a user.

The item evaluation model may include hidden layers of “latent concepts” and “concept learning states”.

As used herein, the term “latent concept” may refer to concepts related to items formed by model learning of the item evaluation model. More specifically, it is possible to select the size of the latent concept, and if an appropriate embedding value is learned based on the selection, the value/size may be invariant. In this case, the latent concept may correspond to 'Key memory'.

According to a feature of the present invention, the item evaluation model may be configured to determine a concept associated with an input item through a latent concept.

As used herein, the term “concept learning state” may refer to a learning state of a learner's concepts. That is, the concept learning state may be different depending on the learner's knowledge state. In this case, the concept learning state may have various values depending on whether the questions presented to the learner are correct or not, and concepts embedded in the questions. More specifically, the concept learning state is variable according to the learner's knowledge state, and may include an absolute value for 'KEY'. In this case, the concept learning state may correspond to ' value memory ' or 'concept memory'.

According to a feature of the present invention, the concept learning state may be updated based on whether the learner has an answer to the question, and furthermore, the concept of the correct questions.

According to another feature of the present invention, the item evaluation model may be configured to determine the concept that the learner has associated with the item from the concept memory through the concept learning state.

Finally, the item evaluation model may output the probability that the user corrects the correct answer for the item.

The term “concept evaluation model” used in this specification may be an artificial neural network algorithm-based model configured to probabilistically output the user's understanding of each concept by inputting the concept determined for the question to be solved by the learner. .

In this case, the concept evaluation model may be configured to share the 'latent concept' and the 'concept learning state' of the item evaluation model. Accordingly, the concept evaluation model may output the degree of understanding of the concept for the item based on the latent concept and the concept learning state shared from the item evaluation model. In other words, the concept evaluation model can increase the interpretability of the hidden concept layer by sharing the 'latent concept' and 'concept learning state' of the item evaluation model, thereby providing a detailed analysis of the learner's knowledge state. have. Moreover, in the learning stage, the evaluation ability for the items of the concept with a low frequency of appearance may be excellent, and the diagnosis performance may be maintained at a high level even if the evaluation is repeated. Accordingly, the device for item evaluation according to various embodiments of the present disclosure may have better diagnostic performance than a single item evaluation model in item evaluation.

Meanwhile, the item evaluation model and the concept evaluation model disclosed in the present specification may be a model based on a deep learning algorithm. More specifically, the item evaluation model and the concept evaluation model are DKVMN (dynamic key-value memory network), DNN (Deep Neural Network), DCNN (Deep Convolution Neural Network), RNN (Recurrent Neural Network), RBM (Restricted Boltzmann Machine) , DBN (Deep Belief Network), SSD (Single Shot Detector), and SVM (Support Vector Machine) may be based on at least one algorithm. However, it is not limited thereto.

Hereinafter, a device for item evaluation according to various embodiments of the present invention will be described in detail with reference to FIGS. 1A to 1E .

1A is a schematic diagram illustrating an item evaluation system according to an embodiment of the present invention. 1B is a schematic diagram illustrating a device for item evaluation according to an embodiment of the present invention. 1C is a schematic diagram of a user equipment according to an embodiment of the present invention; 1d to 1e exemplarily show a display unit of a user device according to an embodiment of the present invention.

First, referring to FIG. 1A , the item evaluation system 1000 is a system that provides item evaluation results based on the knowledge state possessed by the learner. The item evaluation device 100 , the user device 200 and the item to be evaluated DB 300 may be included.

First, the item evaluation device 100 may include a general-purpose computer, a laptop, and/or a data server that receives an evaluation subject item from the evaluation subject item DB 300 and performs an operation. On the other hand, the user device 200 is for accessing a web server (web server) that provides a web page related to a service for item evaluation according to a user's request or a mobile web server (mobile web server) that provides a mobile website It may be a device, but is not limited thereto.

More specifically, referring to FIG. 1B , the device for item evaluation 100 includes a storage 110 , a communication unit 120 , and a processor 130 .

First, the storage 110 may store various data for providing an item evaluation service to a learner by analyzing the items, for example, evaluation subject items obtained from the evaluation subject item DB 300 . In various embodiments, the storage unit 110 is a flash memory type, hard disk type, multimedia card micro type, card type memory (eg, SD or XD memory, etc.), RAM, SRAM, ROM, EEPROM, PROM, magnetic memory. , a magnetic disk, and an optical disk may include at least one type of storage medium.

Next, the communication unit 120 connects the item evaluation device 100 to communicate with an external device. For example, the communication unit 120 may be connected to the user device 200 using wired/wireless communication to transmit/receive data for an evaluation item. Specifically, the communication unit 120 may be further configured to be connected to the user device 200 to transmit the evaluation result.

The processor 130 is operatively connected to the storage unit 110 and the communication unit 120 , and may perform various commands to analyze items and provide services for item evaluation.

Specifically, the processor 130 may receive the item through the communication unit 120 and evaluate the item based on the received item.

To this end, the processor 130 analyzes the received item to determine the concept associated with the item, predicts the concept associated with the item from the concept memory for the user, and outputs a probability of correcting the item by predicting the item evaluation model and the item associated with the item. A concept evaluation model configured to output a degree of understanding of a concept is available. In this case, the item evaluation models are pre-learned models based on the items solved by the learner, and may be used to determine whether the evaluation item is correct or not according to the learner's knowledge state.

According to a feature of the present invention, the processor 130 may be configured to vectorize an item to generate an item vector, compare the item vector and the degree of similarity with the latent concept, and determine a concept for the item based on the similarity. have.

According to another feature of the present invention, the processor 130 determines a weight for each potential concept according to the degree of similarity, and determines whether the question is correct based on the concept learning state and the weight for the latent concept using the item evaluation model. It may be further configured to predict.

According to another feature of the present invention, the processor 130 determines a memory associated with an item from among the concept memories using the item evaluation model, determines an item solving ability based on the associated memory, and based on the item solving ability Thus, it may be further configured to predict whether or not the answer to the question is correct.

According to another feature of the present invention, the processor 130 determines the difficulty of the item, determines the memory associated with the item in the concept memory, determines the item solving ability based on the associated memory, and determines the difficulty level of the item and the item solving It may be further configured to predict whether or not the correct answer to the question based on the ability.

According to another feature of the present invention, the processor 130 may be further configured to update the concept memory based on whether the question received through the communication unit 120 is correct or not, and the concept associated with the question.

That is, by using the item evaluation model and the concept evaluation models, the processor 130 may perform item evaluation tailored to the learner and reflect the update of the knowledge state according to the learner's learning. Furthermore, the processor 130 may determine and provide a recommendation item according to the learner's degree of understanding of the concept.

In particular, the processor 130 uses the concept evaluation model that shares the 'latent concept' in which embedding vectors for each of the concepts of the item evaluation model are stored and the 'concept learning state' of the concept memory that each learner has, It can provide an accurate analysis of the learner's knowledge state.

Hereinafter, the user device 200 will be described in detail with reference to FIG. 1C .

1C , the user device 200 includes a communication unit 210 , a display unit 220 , a storage unit 230 , and a processor 240 .

First, the communication unit 210 connects the user device 200 to enable communication with an external device. For example, the communication unit 210 may be connected to the item evaluation device 100 using wired/wireless communication to transmit/receive data on the evaluation item. Specifically, the communication unit 210 may be further configured to be connected to the item evaluation device 100 to receive an evaluation result.

The display unit 220 may display various contents (eg, text, image, video, icon, banner or symbol, etc.) to the user. Furthermore, the display unit 220 may display various interface screens for displaying the item evaluation result data of the user determined by the item evaluation device 100 . For example, the display unit 220 may be configured to display whether the user's answer to the question is correct, the degree of understanding of the concept, and the recommended question related to the concept of which the degree of understanding is insufficient.

In various embodiments, the display unit 220 may include a touch screen, for example, a touch, gesture, proximity, drag, swipe using an electronic pen or a part of the user's body. A swipe or hovering input may be received.

The storage unit 230 may store various data used to request item evaluation and provide a user interface for displaying result data for item evaluation. In various embodiments, the storage unit 230 may include a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg, SD or XD). memory, etc.), RAM (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) , a magnetic memory, a magnetic disk, and an optical disk may include at least one type of storage medium. Meanwhile, the item evaluation device 100 may operate in relation to a web storage that performs a storage function of the storage unit 230 on the Internet.

The processor 240 is operatively connected to the communication unit 210, the display unit 220, and the storage unit 230, and may perform various commands for providing a user interface for displaying result data for item evaluation. .

Hereinafter, a display unit of a user device according to an embodiment of the present invention will be described in detail with reference to FIGS. 1D and 1E .

First, referring to FIG. 1D , the display unit 220 of the user device 200 recommends and outputs an item related to a concept in which the degree of understanding of a concept determined by the item evaluation device 100 is below a predetermined level. More specifically, the degree of understanding of the learner of the concept a determined by the device 100 for item evaluation is 10%, the degree of understanding of the learner of the concept b is 95%, and the degree of understanding of the learner of the concept c is 70%. In this case, as a recommended item, an item related to a concept a (eg, calculation of a fraction) having a relatively low level of understanding may be determined and provided to the user through the display unit 220 . On the other hand, referring to FIG. 1E together, items related to concept b (eg, equations and multiplication of numbers), which had a relatively high degree of understanding, may be excluded from the recommended items.

Accordingly, the learner can be provided with a customized e-learning learning environment for each individual as intensive learning of weak concepts is possible.

Hereinafter, an item evaluation method according to various embodiments of the present invention will be described with reference to FIGS. 2 and 3 .

2 is a schematic flowchart for explaining a method for evaluating an item according to an embodiment of the present invention. 3 exemplarily shows a procedure of an item evaluation method according to an embodiment of the present invention.

First, referring to FIG. 2 , according to the item evaluation method according to an embodiment of the present invention, a question to be solved by the user is received ( S210 ). Then, the concept of the item is determined by the item evaluation model (S220), and whether the item is correct or not is predicted (S230). Then, the degree of understanding of the concept related to the question is predicted by the concept evaluation model (S240), and whether the question is correct or not and the degree of understanding of the concept related to the question are provided (S250).

More specifically, in step S210 of receiving the item, the item may be received from an evaluation item providing server such as a question bank, or the item may be input through an input unit of the item evaluation device.

According to a feature of the present invention, before the step of receiving the item ( S210 ) is performed, a step of inputting a selection for at least one of a plurality of items may be further performed.

Accordingly, in the step S210 of receiving the item, the item selected by the user may be received.

Next, in the step S220 in which the concept of the item is determined, the concept of the item may be determined.

According to a feature of the present invention, in the step (S220) in which the concept for the item is determined, the item is vectorized so that an item vector is generated, the item vector and the latent concept are compared, and based on the similarity, the item is concept can be determined.

For example, referring to FIG. 3 together, the evaluation item 310 is received through the step S210 of receiving the item, and the item evaluation model 400 is evaluated at the step S220 where the concept for the item is determined. Question 310 is entered. Then, the evaluation item 310 is embedded vectorized to generate an item vector 312 . Then, the degree of similarity with embedding vectors for each of the concepts in the latent concept 410 of the item evaluation model 400 is compared.

Meanwhile, according to another feature of the present invention, a weight for each potential concept may be determined after the step (S220) in which the concept for the item is determined.

For example, referring back to FIG. 3 , a weight for a concept associated with the evaluation item 310 item may be set to a predetermined level or higher, and a weight for a concept having a relatively low correlation with the evaluation item 310 item may be It may be set below a predetermined level.

Referring back to FIG. 2 , in the step S230 of predicting whether the question is correct or not, the probability of whether the user answers the question correctly may be calculated.

According to a feature of the present invention, in the step (S230) of predicting whether the question is correct, whether the question is correct or not may be predicted based on the weight determined for the concept learning state and the latent concept by the item evaluation model.

According to a feature of the present invention, in the step (S230) of predicting whether the question is correct or not, the memory associated with the question among the conceptual memories is determined by the item evaluation model, and the user's question-solving ability is determined based on the associated memory and the correct answer to the question can be predicted based on the question-solving ability.

According to another feature of the present invention, in the step (S230) of predicting whether the item is correct or not, the difficulty of the item is determined by the item evaluation model, the memory associated with the item is determined among the concept memories, and based on the associated memory The question-solving ability of the user may be determined, and whether an answer to the question is correct may be predicted based on the difficulty of the question and the question-solving ability.

For example, referring together with FIG. 3 , in the step S230 in which the correct answer to the question is predicted, based on the concept learning state 420 corresponding to the concept memory, which is the knowledge state of the user, and the weight for each of the latent concepts , a VALUE value corresponding to KEY may be set. Then, the question-solving ability 316 of the user may be determined by the item-solving ability determination network 430 . In this case, the item evaluation model 400 may determine the item difficulty 314 for the evaluation item 310 based on the item vector 312 . Meanwhile, the item difficulty 314 may be determined by an item difficulty prediction model (not shown) independent of the item evaluation model 400 and transmitted to the item evaluation model 400 . Next, based on the item difficulty 314 and the user's item solving ability 316 , a correct answer probability 318 , which is a probability that the user answers the evaluation item 310 , may be determined to be 80%.

Referring back to FIG. 2 , in the step S240 of predicting the degree of understanding of the concept related to the item, the user by the concept evaluation model configured to share at least one of the latent concept and the concept learning state and predict the degree of understanding of the concept for the item The degree of understanding of the concept can be predicted.

For example, referring back to FIG. 3 , the concept evaluation model 500 may be configured to share the item evaluation model 400 and the latent concept 510 and the concept learning state 520 for the user. Accordingly, it is possible to increase the interpretability of the hidden concept layer, so that a detailed analysis of the learner's knowledge state may be possible. Moreover, according to the sharing of the hidden layer, the evaluation ability for the items of the concept with a low frequency of appearance in the learning stage may be excellent, and the diagnosis performance may be maintained at a high level even if the evaluation is repeated. On the other hand, in the step (S240) in which the degree of understanding of the concept related to the item is predicted, the concept 320 related to the evaluation item 310 is input to the concept evaluation model 500, and the related concept 320 is embedded into the vectorized concept ( 320) obtain a concept vector 322 for each. Then, the degree of similarity with the embedding vectors for each of the concepts in the latent concept 510 is compared, and based on the concept learning state 520 corresponding to the concept memory, which is the knowledge state of the user, and a weight for each of the latent concepts , a VALUE value corresponding to KEY may be set. The concept understanding ability 326 of the user may then be determined by the concept understanding ability determination network 530 . In this case, the concept evaluation model 500 may determine the concept difficulty 324 for the concept 320 associated with the evaluation item 310 based on the concept vector 322 . On the other hand, the concept difficulty 324 may be determined by an item difficulty prediction model (not shown) independent of the concept evaluation model 500 and transmitted to the concept evaluation model 500 . Then, based on the concept difficulty 324 and the concept understanding ability 326 of the user, the concept understanding probabilities 328 of the concepts a, b, and c for the evaluation item 310 are 60%, 90%, and 85%, respectively. can be determined as

On the other hand, the step (S240) of predicting the degree of understanding of the concept related to the question is not limited to the above, and is performed simultaneously with the step (S220) of determining the concept for the question or the step (S230) of predicting whether the question is correct or not could be

Referring back to FIG. 2 , in the step (S250) in which whether the question is correct or not and the degree of understanding of the concept related to the question is provided, the step (S230) of whether the question is correct or not, the step of predicting the understanding of the concept related to the question ( S240), the correct answer to each determined question and the degree of understanding of the concept are provided.

For example, referring to FIG. 3 , in the step S250 , in which the question is correct and the degree of understanding of the concept related to the question is provided, the probability of correct answers output by each of the item evaluation model 400 and the concept evaluation model 500 . 318 and conceptual understanding 328 may be provided.

On the other hand, according to a feature of the present invention, after the step (S250) of whether the question is correct and the degree of understanding of the concept related to the question is provided, based on the predicted degree of understanding of the user's concept, among a plurality of additional questions, the predetermined level or less Determining an item related to a concept having an understanding of , and providing a related item may be further performed.

Further, on the basis of the predicted user's understanding of the concept, from among a plurality of additional questions, determining an item with a high degree of understanding related to a concept having an understanding level higher than or equal to a predetermined level, and selecting an item with a high level of understanding among the plurality of additional items The step of providing the remaining items except for may be further performed.

That is, the learner can be provided with items related to the insufficient concept, so that the learner's learning efficiency can be increased.

Meanwhile, in the item evaluation method according to various embodiments of the present disclosure, the item analysis may be performed based on Equation 1 below.

[Equation 1]

Here, BCE refers to 'binary cross entropy', q _t refers to an evaluation item corresponding to time t, a _t refers to a user's answer corresponding to time t, and Ω _t is a concept related to _{q t} refers to a set. In addition, c denotes a concept, item pool is DKVMN prediction part, and ExpM may be defined in part also predicted understood concept. In this case, Ω _t may be 1, that is, one concept related to the item, but is not limited thereto.

According to another feature of the present invention, it is possible to reflect the update of the knowledge state according to whether the user's correct answer to the question and the predicted degree of understanding of the concept.

Hereinafter, the update of the item evaluation model will be described in detail with reference to FIGS. 4 and 5 .

4 is a schematic flowchart for explaining an update step of an item evaluation model applied to various embodiments of the present invention. 5 exemplarily shows an update step of an item evaluation model applied to various embodiments of the present invention.

First, referring to FIG. 4 , in the item evaluation model, an answer to an item is received from a user (S410), and the concept learning state may be updated based on whether the received correct answer and a concept related to the item are received (S420).

For example, referring to FIG. 5 together, in step S410 of receiving an answer to an item, an evaluation item and a correct answer set 620 may be received. Next, in step S420 in which the concept learning state is updated, the evaluation item and the correct answer set 620 are embedded vectorized, and then an erase vector and an add vector are generated, respectively. Then, after the evaluation item 610 is input to the item evaluation model 400 , a weight determined according to the degree of similarity to the latent concept 410 is multiplied for each of the deletion vector and the addition vector. Then, after the existing concept learning state (t) 420 is reflected in the deletion vector, the two vectors are merged to obtain a newly updated concept learning state (t+1) 420 ′. The updated concept learning state 420 ′ may be applied to the evaluation of the item to be presented next to the evaluation item 610 .

Hereinafter, the structure of the item evaluation model and the concept evaluation model used in various embodiments of the present invention, and a configuration for updating the item evaluation model, will be described in detail with reference to FIGS. 6A and 6B .

6A exemplarily illustrates the structure of an item evaluation model and a concept evaluation model applied to various embodiments of the present invention. 6B exemplarily illustrates a configuration for updating an item evaluation model applied to various embodiments of the present invention.

_t ^q에 대응하는 학습자에 대한 문항 풀이 능력 (316) 이 결정된다. 그 다음, 문항 벡터 (312) 에 기초하여 결정된

_q에 대응하는 문항 난이도 (314) 및 문항 풀이 능력 (316) 에 기초하여 p_t에 대응하는 정답 확률 (318) 이 최종적으로 산출될 수 있다. 한편, 개념 평가 모델 (500) 및 문항 평가 모델 (400) 은, 싱크가 맞춰진 잠재 개념 (410, 510), 개념 학습 상태 (410, 520) 를 포함할 수 있다. First, referring to FIG. 6A , the item evaluation model 400 is a latent concept 410 that is a 'Key memory' that stores embedding vectors for each of the concepts, and is variable depending on the knowledge state of the learner and depends on the 'Key'. The concept learning state 420 corresponding to the 'Value memory (or concept memory)' including the corresponding absolute value and the item solving ability determination network 430 for determining the item solving ability for the user may be configured. At this time, the evaluation item 310 corresponding to _{q t is embedded vectorized to generate an item vector 312 corresponding to k t} , and then a vector and an item vector for each concept in the latent concept 410 corresponding to C ( 312), a weight 412 corresponding to _{W t} ^{q may be determined.} Then, based on the concept learning state 420 and the weights 412 , a read vector 422 corresponding to _{r t} ^{q is generated.} Then, based on the item vector 312 and the read vector 422, a network 430 for predicting the item solving ability of the learner corresponding to _{the f t} ^{q feature vector is generated,}

An item solving ability 316 for the learner corresponding to _t ^{q is determined.} Then, based on the item vector (312)

A correct answer probability 318 corresponding _{to p t} may be finally calculated based on the item difficulty 314 corresponding to _{q and the item solving ability 316 .} Meanwhile, the concept evaluation model 500 and the item evaluation model 400 may include synchronized latent concepts 410 and 510 and concept learning states 410 and 520 .

_t ^c에 대응하는 학습자에 대한 사용자의 개념 이해 능력 (326) 이 결정된다. 그 다음, 개념 벡터 (322) 에 기초하여 결정된

_c에 대응하는 개념 난이도 (324) 및 개념 이해 능력 (326) 에 기초하여 m_t에 대응하는 개념 이해 확률 (318) 이 최종적으로 산출될 수 있다.That is, the concept evaluation model 500 may include a latent concept 510 and a concept learning state 520 shared from the item evaluation model 400 , and a concept understanding ability determination network 530 . _{At this time, the concept 320 associated with the item corresponding to C t} , which is tagged by an expert, is embedded vectorized to generate a concept vector 322 corresponding to h, and then a latent concept 510 corresponding to C. According to the similarity between the vector for each concept and the concept vector 322 , a weight 512 corresponding to _{W t} ^{c may be determined.} Then, based on the concept learning state 520 and the weights 512 , a read vector 522 corresponding to _{r t} ^{c is generated.} Then, based on the concept vector 322 and the read vector 522, a network 530 that predicts the learner's ability to understand the concept corresponding to _{the f t} ^{c feature vector is generated,}

_The user's ability to understand concepts 326 for the learner corresponding to ^{t c is determined.} Then, based on the concept vector 322

A concept understanding probability 318 corresponding _{to m t} may be finally calculated based on the concept difficulty 324 corresponding to _{c and the concept understanding ability 326 .}

In this case, the item evaluation model 400 and the concept evaluation model 500 may be models based on a plurality of DKVMN algorithms sharing a hidden concept layer, but are not limited thereto.

On the other hand, the concept learning state 420 of the item evaluation model 400 may be updated based on whether the user corrects the item and the concept associated with the item, and the updated concept learning state 420' is the concept evaluation model 500 can be shared.

More specifically, referring to FIG. 6B , in order to update the concept learning state 420, the previously learned qt embedding vector and a _{t, which} is the user's response to the question, are combined, and correspond to _{(q t} , a _{t ).} The evaluation item and the correct answer set 620 are non-linearly mapped to generate the item correct answer vector 622 corresponding _{to v t .} In this case, the item correct vector 622 may be generated by Equation 2 below.

[Equation 2]

v _t = tanh( FC ( A (q _t ),a _t ))

That is, the update does not generate and learn a new embedding matrix for the evaluation item and the correct answer set 620 of _{(q t} , a _{t ), and the existing evaluation item 310 uses the embedding vectorized item vector 312 .} It is initiated by binding with _{a t .} Accordingly, the item evaluation model is updated without unnecessary overhead, and more stable learning (eg, in the DKVMN model) may proceed. Then, a sigma (σ) is applied to the item correct vector 622 to generate a deletion vector corresponding to et, and a hyperbolic tangent (tanh) is applied to the item correct vector 632 to generate an additional vector 634 . . Then, a weight 412 determined according to the degree of similarity to the latent concept 410 is multiplied for each of the deletion vector 632 and the addition vector 634 . Then, after the existing concept learning state 420 is reflected in the deletion vector 632, the values of the two vectors 632 and 634 are combined, so that the newly updated concept learning state corresponding to _{S t+1 (} 420') can be obtained. The updated concept learning state 420 ′ may be applied to the evaluation of an item to be presented next to the evaluation item 310 . Moreover, it can be shared in the concept evaluation model 500 .

Comparative Example: Comparison of performance of item evaluation model + concept evaluation model, and single item evaluation model according to various embodiments of the present invention

Hereinafter, evaluation results of the item evaluation device according to various embodiments of the present invention will be described with reference to FIGS. 7A and 7B .

Referring to FIG. 7A , an evaluation result for a single item evaluation model based on the DKVMN algorithm is shown. Furthermore, referring to FIG. 7B , evaluation results of the item evaluation device based on the item evaluation model and the concept evaluation model applied to various embodiments of the present invention are shown.

Referring to FIGS. 7A and 7B , the accuracy of analysis of the item evaluation model + concept evaluation model and the single item evaluation model according to various embodiments of the present invention is shown at a similar level. However, referring further to (b) and 7b (b) of FIGS. 7A and 7B, the loss value of the cross-entropy is significantly lower in the item evaluation model + the concept evaluation model than the item evaluation model alone. appear. Furthermore, referring to the AUC values corresponding to the performance of the models of FIGS. 7A (c) and 7B (c), the single item evaluation model showed 0.8162, but as the evaluation progressed, it was shown to decrease to 0.76 or less. However, in the case of the item evaluation model + concept evaluation model, after a high AUC value of 0.8269 is measured, it appears to be maintained without falling below 0.8. That is, these results may mean that the item evaluation model + the concept evaluation model has better predictive performance than the single item evaluation model, and that the performance is maintained even if the item evaluation is repeated.

In this case, the single item evaluation model may have poor evaluation ability of items for concepts with low frequency of appearance due to overfitting due to items having concepts frequently appearing in the learning stage. That is, a single item evaluation model may have poor performance depending on the balance of questions for each concept.

On the other hand, as the item evaluation model and the concept evaluation model share the concept learning state for the user, the evaluation ability for the items of the concept with low frequency of appearance is higher than that of the single model, and the diagnosis performance is high even if the evaluation of the item is repeated. level can be maintained.

Accordingly, the device for item evaluation according to various embodiments of the present disclosure based on two models of the item evaluation model and the concept evaluation model has the effect of performing a detailed analysis of the learner's learning state.

Accordingly, the present invention can provide a level of understanding of concepts related to the correct answer to the question output by each model by the item evaluation system and the question, thereby providing a customized e-learning learning environment for each learner.

Although one embodiment of the present invention has been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made within the scope without departing from the technical spirit of the present invention. have. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the equivalent range should be construed as being included in the scope of the present invention.

100: device for item evaluation
110, 230: storage unit
120, 210: communication unit
130, 240: processor
200: user device
220: display unit
300: DB of items to be evaluated
310, 610: evaluation questions
312, 612: question vector
314, 614: Item difficulty
316, 616: question-solving ability
318, 618: Probability of correct answer
322: Concept Vector
324: Concept Difficulty
326: concept understanding ability
328: concept understanding probability
410, 510: latent concept
412, 512: weight
420, 520: concept learning state
422, 522: lead vector
420': updated concept learning state
430: question-solving ability determination network
530: concept understanding ability determination network
1000: item evaluation system

Claims (18)

An item evaluation method implemented by a processor, comprising:
receiving a question to be solved by a user;
determining concepts for the items by using an item evaluation model configured to determine concepts for items based on latent concepts;
predicting whether the item is correct or not, using the item evaluation model further configured to predict whether the item is correct based on the determined concept and the concept learning state for the user;
predicting the degree of understanding of the concept related to the item for the user by using a concept evaluation model configured to share at least one of the latent concept and the concept learning state and predict the degree of understanding of the concept for the item, and
An item evaluation method comprising the step of providing a correct answer to the item and a degree of understanding of a concept related to the item. According to claim 1,
The item evaluation model is
Further configured to probabilistically predict whether the question is correct,
The concept evaluation model is,
It is further configured to probabilistically predict the degree of understanding of the concept related to the item,
The providing step is
An item evaluation method comprising the step of providing a probability of correct answer for the item and a probability of understanding the concept. According to claim 1,
The step of determining the concept for the question is,
vectorizing the items to generate an item vector;
comparing the degree of similarity between the item vector and the latent concept; and
and determining a concept for the item based on the degree of similarity. 4. The method of claim 3,
After determining the concept,
Further comprising the step of determining a weight for each of the latent concepts according to the degree of similarity,
Predicting whether the question is correct or not,
Using the item evaluation model, the item evaluation method comprising the step of predicting whether the item is correct based on the weight for the concept learning state and the latent concept. According to claim 1,
The concept learning state is,
An item evaluation method, which is defined as a concept memory consisting of the absolute value of a concept vector obtained by vectorizing the concept of the correct answer item matched by the user. 6. The method of claim 5,
Predicting whether the question is correct or not,
Using the item evaluation model,
determining a memory associated with the item from among the concept memories;
determining the item-solving ability based on the associated memory; and
An item evaluation method comprising the step of predicting whether the item is correct or not based on the item solving ability. 6. The method of claim 5,
The item evaluation model is
Further configured to predict whether the question is correct or not based on the difficulty of the question and the user's ability to solve the question,
Predicting whether the question is correct or not,
determining the difficulty level of the question;
determining a memory associated with the item from among the concept memories;
determining the question-solving ability based on the associated memory; and
Further comprising the step of predicting whether the correct answer to the question based on the difficulty of the question and the question-solving ability, the item evaluation method. 6. The method of claim 5,
receiving an answer to the question from the user; and
Including the step of updating the concept memory based on whether the received answer is correct, and the concept associated with the item, the item evaluation method. According to claim 1,
After the step of providing,
determining, from among a plurality of additional items, an item related to a concept having a level of understanding less than or equal to a predetermined level, from among a plurality of additional items based on the predicted level of understanding of the concept; and
Further comprising the step of providing the associated item, item evaluation method. According to claim 1,
After the step of providing,
predicted recall Determining an item with a high level of understanding related to a concept having a level of understanding higher than or equal to a predetermined level among a plurality of additional items based on the level of understanding of the concept, and
Among the plurality of additional questions, the method further comprising the step of providing the remaining questions except for the high comprehension questions. a communication unit configured to receive questions to be solved by the user, and
including a processor;
The processor is
Using the item evaluation model configured to determine the concept of the item based on latent concepts, the concept of the item is determined, and whether the item is correct based on the determined concept and the concept learning state for the user Using the item evaluation model further configured to predict A device for item evaluation, configured to predict a degree of understanding of a concept related to the item for the user by using it, and to provide whether the item is correct or not and a degree of understanding of a concept related to the item. 12. The method of claim 11,
The processor is
The device for item evaluation, further configured to vectorize the item to generate an item vector, compare the degree of similarity between the item vector and the latent concept, and determine a concept for the item based on the degree of similarity. 13. The method of claim 12,
The processor determines a weight for each of the latent concepts according to the degree of similarity,
The device for item evaluation, further configured to predict whether the item is correct or not based on the weight for the concept learning state and the latent concept by using the item evaluation model. 12. The method of claim 11,
The concept learning state is,
A device for item evaluation, which is defined as a concept memory consisting of the absolute value of a concept vector obtained by vectorizing the concept of the correct answer item matched by the user. 15. The method of claim 14,
The processor uses the item evaluation model to determine a memory associated with the item from among the conceptual memories, determines the item solving ability based on the associated memory, and applies the item to the item based on the item solving ability. A device for item evaluation, further configured to predict whether the answer is correct or not. 15. The method of claim 14,
The item evaluation model is
It is configured to predict whether the question is correct or not based on the difficulty of the question and the user's ability to solve the question,
The processor is
Determining the difficulty of the item, determining the memory associated with the item from among the conceptual memories, determining the item solving ability based on the associated memory, and determining the item difficulty and the item solving ability based on the item difficulty and the item solving ability A device for item evaluation, further configured to predict whether the answer is correct or not. 15. The method of claim 14,
The communication unit,
further configured to receive an answer to the question from the user,
The processor is
The device for item evaluation, further configured to update the concept memory based on whether the received answer is correct or not, and a concept associated with the item. The processor is further configured to determine an item related to a concept having a degree of understanding below a predetermined level from among a plurality of additional items, and to provide the related item.

Images