RU2640709C1 - Method of student's knowledge evaluation at computer-based testing - Google Patents

Abstract

FIELD: education.

SUBSTANCE: invention relates to the educational medium in which the student chooses the response to the posed question from the set of answers at the time of registration for event-related potentials, and can be used for automated evaluation of knowledge. By the presence of component P300, the type of decision is classified using a set of decision rules for matching random guessing, copying, partial knowledge, uncertain knowledge or confident knowledge.

EFFECT: reliability of the student's knowledge evaluation is increased.

5 cl, 1 tbl, 5 dwg

Description

Technical field

The invention relates to teaching aids in which a student selects an answer to a question from a set of answers simultaneously with the registration of event-related potentials (BSC), and can be used for automated knowledge assessment.

State of the art

There is a known method of testing knowledge in which students evaluate each of the proposed options for answering a test task and choose only one. This is a traditional assessment method using multiple-choice tests. If the option chosen by the student matches the one marked by the developer of the test task as correct, then the student gets 1 point, otherwise, if the option is incorrect or the question is missed, the student does not receive any points. This standard method of assessing knowledge is popular due to its easy implementation in automated environments and objectivity (the probability of obtaining a high total score due to guessing quickly decreases with an increase in the number of tasks in the test).

The disadvantage of this method of testing knowledge is the high probability of error in the unfair performance of the task: with the use of extraneous sources (cheating) or random guessing (Lau PNK, Lau SH, Hong KS, Usop H. 2011. "Guessing, Partial Knowledge, and Misconceptions in Multiple- choice Tests. "Journal of Educational Technology & Society 14 (4): 99-110. http://www.jstor.org/stable/jeductechsoci.14.4.99) / 1 /. Due to the high probability of error, the result of one test task is not reliable. To obtain a reliable assessment, it is required to present a test of many (ten or more) similar test tasks, the estimates for which are averaged.

The effectiveness of the traditional method of testing knowledge can be enhanced by:

- detection of cases of unfair performance of tasks;

- a more accurate assessment of knowledge on a specific issue with the introduction of intermediate assessments such as “uncertain knowledge”, “residual knowledge”, “knowledge of terms, but ignorance of mechanisms” and other levels of knowledge and skills specific to different disciplines.

Known testing methods with the introduction of a clarifying question about confidence (Certainty-Based Marking (CBM), https://docs.moodle.org/30/en/Using_certainty-based_marking) / 2 /; excluding incorrect options (Chang SH, Lin PC, Lin ZC. Measures of partial knowledge and unexpected responses in multiple-choice tests. Educational Technology & Society. 2007 Oct 1; 10 (4): 95-109, http: // www .jstor.org / stable / jeductechsoci.10.4.95) / 3 /. The disadvantage of these methods is to increase the time spent on passing the test, as well as the fact that they do not allow to distinguish between uncertainty in knowledge of this issue and the general psychological uncertainty of the student.

A known method of improving the accuracy of the assessment, based on measuring the time taken to complete the test task, and calculating the index of efforts to pass the test (Chang SR, Plake BS, Kramer GA, Lien SM. Development and Application of Detection Indices for Measuring Guessing Behaviors and Test- Taking Effort in Computerized Adaptive Testing. Educational and Psychological Measurement. 2011; 71 (3): 437-459. Http://dx.doi.org/10.1177/0013164410385110) / 4 /. The method is based on the assumption that a conscientious student spends more time on the wrong answer (he doubts, remembers) than the right one. However, if the test task requires procedural thinking, then the decision time will be proportional to the number of required mental operations, that is, the procedural complexity of the task. There is also an inverse relationship with a random guessing strategy - a lot of wrong answers without time. To solve this problem, the authors propose an individualized approach, however, a student who has passed one test with one strategy can change it when passing another test, for example, if this time he is better prepared.

Thus, the known methods for increasing the accuracy of knowledge assessment use the time taken to complete the test task, but do not use information about the pattern of the task itself, namely, the sequence of execution of its components.

The standard pattern for completing a multiple choice test task begins by reading a question. The question contains key information, which, with the appropriate knowledge, allows you to extract the correct answer from memory for comparison with the proposed options. In other cases, the question contains a description of the criterion that must be applied to each answer option. Further, when viewing the next answer option, the student discovers the answer that matches the one retrieved from the memory, selects it, and finishes solving the task. If not one of the proposed options does not coincide with the created image inside the student’s brain, then the student can proceed to a more attentive reading of the text of the question in order to check his decision in the hope of finding an error in it. If the student discovers an error, he proceeds to the corresponding answer option, marks it as correct, and finishes solving the task. If the error cannot be found, then the student can refuse a full-fledged solution to this task and proceed to a guessing strategy in which he chooses the most plausible option. Thus, the pattern of completing the test task is determined by the number and sequence in time of the operations of viewing the text of the question and viewing the options for the answer, and the critical moment determining the connection between the availability of knowledge and the correct answer occurs when viewing the correct answer when there is an image of this answer inside the student’s brain.

When a person’s internal image coincides with the external one, then approximately 300 ms after the presentation of the external stimulus, a positive component of the SSP P300, also called P3, develops in the parietal parts of the cerebral cortex. Thus, the registration of SSP in the parietal parts of the cerebral cortex can be used to detect the coincidence of the internal image (endogenous) with the external (exogenous) image.

Registration of BSC with subsequent isolation of component P300 is used to diagnose the functional state of the brain (RU 2521345, A61B 5/0484, publ. 2013-03-22) / 5 /, functional interhemispheric asymmetry (RU 2528658, A61B 5/0484, publ. 2014 -09-20) / 6 /, detection of guilt or false information (US 5113870 (A), A61B 5/0484, A61B 5/16, publ. 1992-05-19) / 7 /, the presence of consciousness (WO 02100267 (A9 ), A61B 5/00, A61B 5/0205, A61B 5/0476, publ. 2004-02-19) / 8 /.

However, the use of electroencephalographic indicators of the brain to detect events that match the internal image with the external stimulus presented to assess student knowledge is not known.

In recent years, there has been a significant reduction in the cost of technology for recording and analyzing the electrical activity of the brain, which has made methods based on it available for widespread use in education. Thus, the invention was announced about the invention of a portable device in the form of a rim for registration P300 (KR 20160041748, A61B 5/0476, publ. 2016-04-18) / 9 /.

Closest to the technical nature of the claimed invention is the "Regulating device and method for identifying brain waves" (Brain wave identification method adjusting device and method, WO 2008056492, A61B 5/0476, publ. 2008-05-15) / 10 /, taken as prototype of the present invention. In the prototype, the system includes an interface module that registers the user's electroencephalogram (EEG), recognizes the component P300 of the visual BSC contained in the EEG after presenting the manipulation menu, and operates the device based on the recognized component P300. In the description of the prototype, a television is assumed to be a controlled device, and intended operations with the device include switching channels, selecting the desired program genre for viewing and adjusting the sound volume level. In an example implementation, a menu of several items is presented to the user on the screen, which, in turn, with a fixed time step set by software, are highlighted in the form of a frame around the current item. If, in response to highlighting the current item, the potential of P300 is recorded in the brain, then this menu item is activated. The development of the potential of P300 in the cerebral cortex is accompanied by an event of coincidence of the internal image (the item that the user wants to go to) and the external stimulus (the item that is highlighted by a frame) and does not require volitional efforts, therefore the prototype method allows you to switch menu items without hands.

However, in the prototype method, the viewing period for each of the proposed options is fixed and is set programmatically, which, if the method for assessing knowledge is applied, does not allow the user to choose an answer option based on thinking about the meaning of the question for an arbitrary time interval and comparing the answer options with each other.

The problem to which the invention is directed, is to develop an improved method for reliable assessment of student knowledge in computer testing.

The problem was solved due to a new technical result - separation of the correct answer based on the student’s knowledge from a randomly guessed answer or an answer obtained using outside sources, as well as separation of the incorrect answer when trying to use partial knowledge from a randomly selected incorrect answer.

Unlike the prototype, in the claimed invention, the user actively navigates through text elements with answer options, being late for viewing each of them as much time as he needs to think about the task. Moreover, the sequence of viewing the answer options is not determined in advance, which gives the student the opportunity to compare alternative answer options with each other. The student uses the manipulator (mouse) to mark the selected answer option, and not the electroencephalographic interface. By the presence of the P300 component in the SSP, after presenting the answer option, the student is judged that the student has the correct answer ready that matches the one presented on the screen.

The use of behavioral and electroencephalographic indicators of a student’s work to classify types of solutions to test tasks does not follow from the prior art, which confirms the inventive step of the proposed method.

The specified technical result is achieved by the fact that the method of evaluating a student’s knowledge in computer testing consists in presenting a test task, consisting of a question and answer options, on a computer screen, registering the time when the text of the question began to be viewed and each answer option, the time when the text of the question is finished and of each answer option, the time it takes to decide on the answer option using the manipulator, the calculation of the integral indicators of the test task, the character They show the sequence and speed of viewing the question and answer options, registering the SSP while viewing the text of the answer option, highlighting the SSP P300 component, if any, conclude that the endogenous image corresponding to the correct answer to the question coincides with the exogenous image that occurs when reading text on the screen computer classification of the type of solution using a set of decision rules for matching random guessing, cheating, partial knowledge, uncertain knowledge or confident knowledge.

In a preferred embodiment:

- the question and answer options that make up the test task are presented on the computer screen in the form of visually marked areas of the screen, and the text is hidden;

- the text of the question and the answer options are presented when you hover over the corresponding area of the screen;

- registration of the time point for starting to view the text of the question and each answer option is carried out by hovering over the area of the screen intended for the presentation of the corresponding text;

- registration of the time point for viewing the text of the question and each answer option is carried out at the time the cursor is moved outside the screen area intended for the presentation of the corresponding text.

Knowledge of the time point for starting viewing the text of the answer option is necessary for registering the BSC. The appearance of the text in the focus of the student’s attention is a significant visual verbal stimulus, in response to which the BSC develops in the brain. The P300 component develops in the parietal zones of the brain after 300-400 ms from the moment the stimulus is presented. By the presence of the component of the SSP R300, it is possible to separate the correct answer based on the student’s confident knowledge from a randomly guessed answer based on partial knowledge.

Registration of the moments of the beginning and end of viewing the text of each answer option allows you to calculate the number of text views and their order. This allows you to highlight long pauses by which you can separate the correct answer based on the student’s existing knowledge from cheating using extraneous sources.

Recording the time of making a decision about choosing an answer option using the manipulator allows you to calculate the reaction time by which you can separate spontaneous randomly guessed correct answers from consciously made decisions about the correctness of the text read.

The calculation of the number of views of the text of the answer options before and after the decision makes it possible to highlight the characteristic patterns of performing the test task associated with uncertainty in one’s own knowledge, according to which one can separate the correct answer based on confident knowledge from the answer obtained on the basis of uncertain knowledge.

List of Figures

FIG. 1. The connection scheme of the modules to implement the method of assessing student knowledge in computer testing.

FIG. 2. The block diagram of the implementation of the method of assessing student knowledge in computer testing.

FIG. 3. The block diagram of the algorithm for selecting the P300 component.

FIG. 4. The flowchart of the registration algorithm of the time points of the beginning and end of viewing the text element of the test task presented on the screen.

FIG. 5. Patterns of the deployment of events viewing text elements in time for five types of solutions to the test task.

Table. Comparison of the reliability of assessments of student knowledge by standard and claimed methods obtained on a probabilistic model of passing the test of 20 tasks.

The implementation of the method includes the following operations: Prepare student 1 (Fig. 1) to register an EEG, for which an electric signal sensor 2 is connected to the surface of the head, connected to an EEG unit 3, which streams the EEG through a communication unit 4 to computer 5. Check the quality of signal registration in accordance with the instructions for the electroencephalograph. EEG is recorded on computer 5 in real time.

Prepare the workplace of the student 1 undergoing testing, so that he can see the screen of the stimulation unit 6 and it is convenient to use the manipulator 7.

The screen presents test tasks one at a time. A flowchart of a method for assessing student knowledge in relation to the i-th task is shown in FIG. 2.

During the test, the time is taken to start viewing the text of the question, finish viewing the text of the question, start viewing the text of each answer, time to decide on the choice of the answer using the manipulator when performing task 9 in protocol 10. At the same time registration of the SSP 12 is carried out, for which, as incentive marks, the time points for starting to view the response options 11 stored in the protocol 10 are used.

In the BSC registered after the start of viewing the current answer option, the P300 component is isolated. Information on the presence of the P300 component in the registered MTP and information from the protocol 10 are used to calculate M complex indicators x ₁ ... x _M , which are input to the classifier 17, in which using a set of decision rules establish the correspondence of the type of solution q _{i to} random guessing Q _G , charge-off Q _C , partial knowledge of Q _P , uncertain knowledge of Q _I or confident knowledge of Q _K. Information on the type of solution q _{i is} used to compute the estimate G _i 18 using the transfer function ƒ for completing the ith test task.

The algorithm for isolating the P300 component in the registered SSP is shown in FIG. 3.

From the registered multichannel EEG 19, a fragment of 1 s is extracted, the next one after the start of the j-th viewing of the text t _j 20, which is an SSP 21 in response to a visual stimulus. Since the visual stimulus contains verbal information, the perception of the stimulus leads to the formation of a complex image that includes visual and verbal components. However, electrical brain activity, reflecting the perception and analysis of the stimulus, is hidden among the background activity of the brain, constantly present in the recorded EEG. To extract the relevant signal from the total EEG, spatial 22 and frequency 23 filtering is performed.

(Hjorth, В., 1975. An on-line transformation of EEG scalp potentials into orthogonal source derivations. Electroenceph. Clin. Neurophysiol. Suppl. 39 (5), 526-530) /11/ в отведении Pz по сйстеме 10-20. Для этого используют 5 каналов многоканальной ЭЭГ, записанной по монополярной схеме с активными отведениями в точках по стандарту 10-20: Cz, Oz, Р3, Р4, Pz.Spatial filtering 22 is carried out, calculating the locality

(Hjorth, B., 1975. An on-line transformation of EEG scalp potentials into orthogonal source derivations. Electroenceph. Clin. Neurophysiol. Suppl. 39 (5), 526-530) / 11 / in lead Pz according to system 10-20 . To do this, use 5 channels of a multi-channel EEG recorded according to a monopolar scheme with active leads at points according to the standard 10-20: Cz, Oz, P3, P4, Pz.

The frequency filtering 23 is carried out by a Butterworth filter (https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.butter.html) / 12 / with a passband of 1-8 Hz.

After that, the amplitude 24 of the maximum deviation of the signal in the positive direction from the zero line in the range of 300-400 ms from the beginning of the fragment is measured. If the obtained value of A _j exceeds the threshold A *, then the presence of the P300 component (z _j = 1) 26 is noted, otherwise its absence (z _j = 0) 27 is noted. Exceeding the threshold value is interpreted as the coincidence of the endogenous image extracted from memory with the exogenous way formed when reading the text of the answer option.

The threshold value A * is found empirically, since the amplitude of the signal can vary depending on the characteristics of the filter. In contrast to the prototype method, calibrating the threshold value A * for detecting the P300 component in the present method is not a significant operation. Calibration can be done by presenting obviously simple tasks or tasks containing a hint. In this case, the amplitude of the P300 component in the BSC when viewing the correct answer should be sufficient to detect the correspondence of endogenous and exogenous images, and when viewing the wrong variants, it should not. Since the inventive method does not impose restrictions on the length and complexity of the text in the elements of the test task, the calibration should be carried out taking into account the subjective complexity of the presented test tasks for this student. An important requirement for successful calibration is short text answer options with a length of not more than 3 words, as well as the correspondence of the text length in calibration and work test tasks.

Events associated with the student’s actions are recorded in the form of an array of pair values, where the first value is the time from the start of the presentation of the test task t _i0 , s, and the second is the integer code of this event. In a particular example of the method, the events when writing to the protocol 10 were encoded according to the scheme:

1 - start of a mouse click,

-1 - click the mouse button

1000 - start viewing the question,

-1000 - end of viewing the question,

1001 - start viewing the 1st answer option,

-1001 - end of viewing the 1st answer option,

1002 - start viewing the 2nd answer option,

-1002 - end of viewing the 2nd answer option,

etc. by the number of answer options.

The protocol 10 also records the duration of the task, s, equal to the time from t ₀ to pressing the "Next" button, the content of text elements, the host address, user code, question code in the question bank, serial number of the test task and other metadata that are not related to the invention.

An algorithm for recording the start and end times of viewing text elements is presented in FIG. four.

According to the invention, the text elements of the test task, including the question and answer options, are presented in the form of visually highlighted areas of the screen occupied by text elements. When you hover the mouse cursor 43 over the area occupied by the element e _k, the text of the element T _{k is} made visible 44. In this case, the time moment of the start of the jth viewing of the text element t _{onj is recorded} . The text remains visible, and therefore readable until the students move the cursor outside the text element e _k 46, when the text T _{k is} again replaced by visually highlighted areas of the screen 47. At the same time, the end time of the jth viewing of the text element t _{offj is recorded} . The time _instants t _onj and t _offj are stored in the protocol 10.

There are other ways to register the start and end times of viewing each text element, for example, using a device for tracking gaze (eye-tracker, https://en.wikipedia.org/wiki/Eye_tracking) / 13 /, the disadvantages of which are high cost and necessity calibration.

Information about the time points of the start and end of viewing text elements is used to calculate the M performance indicators of the test task, characterizing the sequence and speed of viewing the question and answer options. The set of indicators x ₁ ... x _{M is} selected in such a way as to characterize the typical patterns observed when testing students' knowledge.

In a particular example of the method for the operation of the classifier 17 (Fig. 2) calculate the following complex indicators:

x ₁ = y - sign of the correct answer, if the number of the selected answer corresponds to the correct one.

x ₂ = z is a sign of the presence of the P300 component in the BSC to the selected response option.

x ₃ = extra - a sign of excessive views of the answer options before making a decision, corresponding to V _extra > 1, where V _extra - the number of extra views of the answer options above the minimum necessary for the optimal solution.

x ₄ = check - the check flag corresponding to V _check > 1, where V _check is the number of views of the answer options after the final mouse click.

x ₅ = scan is the check flag corresponding to n _scan > 0, where n _scan is the number of scans before the first click, where scanning is understood as sequential scans of all answer options in the direct or reverse order without deciding whether to select one of them.

x ₆ = spont - a sign of a spontaneous reaction, when the reaction time from the start of viewing the answer option is r <0.5 s, which is too short to make an informed decision about the correctness of the text read;

, где Р - интервалы между последовательными действиями, превышающие 5 с. x ₇ = pau - pause sign corresponding

where P - intervals between successive actions exceeding 5 s.

x ₈ = out is a sign of switching to another application window, corresponding to p _out > 0.05, where p _out is the fraction of the task’s execution time spent outside the current application window.

The critical values for the indicators were selected by the cross-validation method when training the automatic classifier. 10-fold cross-validation was carried out using the random forest method (Random Forest https://en.wikipedia.org/wiki/Random_forest) / 14 / when training 300 random trees on partial training samples from 500 solved tasks marked out by an expert on five strategies. The critical values for the decision rules were obtained on the basis of the analysis of the constructed graphs of the partial dependence of the determination function of this strategy on the values of this indicator (Partial Dependence Plots http://scikit-learn.org/stable/auto_ examples / ensemble / plot_partial_dependence.html) / 15 /.

To classify 17 types of solution of test tasks for classes, a set of decision rules is used:

where Q _K is confident knowledge, Q _I is uncertain knowledge, Q _P is partial knowledge, an attempt to guess by comparing the question and answer options, Q _G is random guessing, Q _C is cheating. If the rule does not work, then use the following in the order shown.

The calculation of the assessment 18 is carried out according to the formula

- передаточная функция, так что в частном случае осуществления способа получают оценку на шкале от 0 до 1 со значениями функции

,

.where q _i is the type of solution of the i-test task,

- transfer function, so that in the particular case of the method, an estimate is obtained on a scale from 0 to 1 with the values of the function

,

.

Execution example.

The method was used in assessing the knowledge of students in the discipline "Human and Animal Physiology" at the Department of Physiology of the Academy of Biology and Biotechnology of the Southern Federal University.

For testing, we used the Moodle learning management system (https://moodle.org/) / 16 /, which is a web application and includes a standard module for testing knowledge with the ability to edit test tasks. During the test, test tasks were presented for one randomly selected question from the bank on this topic. The test task for assessing students' knowledge consisted of the text of the question and 2-6 answer options. Text elements of the test task were covered with opaque panels of gray color, which became transparent when the mouse cursor was over. If the answer options were accompanied by a round indicator (radio input), then only one answer could be selected. If the answer options were accompanied by square indicators (checkbox input), then it was possible to select several answer options. In the latter case, it is mandatory to look at all the answers, since each of them requires a separate decision. During the testing process, the student purposefully moves the mouse cursor to the text element that he wants to read. Usually, when passing the test, the student reads the question, then proceeds to reading the answer options. If the answer he read seems plausible to him, then he marks it as correct, for which he clicks on it with the mouse (clicks on the button when the cursor is over the answer text). The indicator that accompanies this answer is switched to the “selected” state, and a dot or a bird is displayed on the indicator. If you can select only one answer, then after choosing the correct answer, the student may not read the remaining options and finish the test task by clicking on the “Next” button. At the same time, he proceeds to the next test task or to the results viewing page. If it is possible to select several answer options, the student looks through each answer option and clicks on it if he considers it correct.

SSP was recorded using a mobile electroencephalograph 3 (Fig. 1) Encephalan EEGR-19/25 (Medikom-MTD, Russia), the signal from which was transmitted via wireless communication channel 4 using the bluetooth standard. EEG was recorded in 19 leads according to the scheme 10-20. Signals of EOG, ECG, GHG (respiration curve), and posture changes were also recorded.

Patterns of sequences of events that are characteristic of different types of test solution with the choice of a single answer option out of 4 proposed are shown in FIG. 5 where

a - decision pattern with confident knowledge;

b - decision pattern with uncertain knowledge;

c - solution pattern with partial knowledge;

g - solution pattern for random guessing;

d - the pattern of the decision to write off.

Patterns are presented in the form of a time scan, where each viewing of a text element corresponds to a rectangle, the left side of which corresponds to the beginning of the viewing, and the right side - to the end, which is usually associated with moving the cursor to another element. The ordinate axis is the time counted from the start of the presentation of the test task. The vertical bold line shows the moment you click on the mouse button. The selected segments r is the reaction time, P is the pause time. Conventional symbols are plotted along the ordinate axis - q - question, light fill of the rectangle, a1 ... a4 - answer options, dark fill of the rectangle, ERP - schematic representation of the general dynamics of the SSP recorded within 1 s when viewing answer options lasting more than 300 ms. For all patterns, the correct option is number 3, and option 2 is as close as possible to the correct answer, that is, it is the best distractor. The right side shows the output of the classifier of the type of solution q _i and the indicator of the presence of the P300 component in the BSC to the selected answer option z.

Example 1 (Fig. 5, a)

The solution of the test task with reliable knowledge is characterized by an average reading time of question 28, sufficient to understand the meaning of the question. In the process of conscious perception of the question text, an endogenous image of the correct answer is formed. When viewing the next answer option, the student forms an exogenous image that corresponds to the perceived verbal stimulus. With the coincidence of exogenous and endogenous images, a pronounced positive oscillation develops (positivity down in the figure), known as the P300 component. For purposes of illustration, a deviation crossing the zero line is considered sufficient to detect P300. Following this, the student makes the right choice 29 and completes the decision. For example, the question text says “How much will 2 × 9 be?”. The student forms an endogenous image of "eighteen" or "18". He looks at the first answer - “5”. An exogenous “five” image is formed, which does not correspond to the endogenous image. Then he looks at the second answer - “19”. This answer seems to be correct, but there is still a mismatch between "eighteen" and "nineteen." Then he looks at the third answer - “18”. Exogenous and endogenous images coincide and the student clicks with relief on this option.

Example 2 (Fig. 5, b)

After a long reading of question 30, the student begins to look at the answer options. He finds the correct answer and marks it with a click of 31. However, the uncertainty in the choice made makes the student look at other options 32 to compare the proposed options with each other. In this case, exogenous images overlap each other and confusion may occur in the head. Uncertain of their knowledge, the student re-reads the text of question 33. After which he rechecks the correctness of the choice 34 already made and completes the task. Thus, a solution with uncertain knowledge is characterized by unnecessary time expenditures compared to confident, as well as a greater likelihood of error if the alternative answers are very similar to the correct one.

Example 3 (Fig. 5, c)

The student reduces the reading of question 35 to a time insufficient to form a ready-made answer. The student manages to grab a couple of key terms and in this state goes on to view the answer options. To complete the picture, he quickly gets acquainted with the content of all the answer options, looking through them in a row, not choosing any of them. This is a characteristic pattern of scanning 36. Now the student knows from which options he has to choose, and then he again reads the text of the question in order to more accurately understand what is being asked in the question. Residual knowledge or general erudition and logic help to eliminate the obviously wrong answer options, 1st and 4th. Thus, the probability of correct guessing increases from 0.25 (1 option out of 4) to 0.5 (1 option out of two). Since the student’s knowledge is only partial, he is limited by the choice of the two most plausible options. He chooses one option 37, then another 38, relying on luck. When viewing plausible options, he does not have an endogenous image of the finished answer, so P300 is not detected. Partial knowledge of the solution pattern is characteristic in a hurry during testing and inadequate preparation.

Example 4 (Fig. 5, g)

After a short review of the text of question 39, the student realized that he did not have the knowledge necessary to answer the question, so he decided to click at random on any of the answer options. The motivation to click on any of the options is that with the standard method of assessing knowledge, random guessing in the test configuration under consideration gives 25% of the correct answers. The key features of the random guessing pattern are a short reaction time of r 40 and a small number of views. The student does not try to guess the correct question based on the analysis of semantic content, therefore, does not read the text. Even if he looks through several text elements, these will be short unsystematic scans, insufficient for the formation of endogenous and exogenous images.

Example 5 (Fig. 5, d)

The student quickly reads question 41, and then conducts a search in an external source - in a smartphone with access to a social network, where screen shots with the completed tasks are laid out. As external sources, a document with answers (a textbook), a database, a search engine on the Internet, and cheat sheets can act. If the student does not have ready-made answers for this test, then he has to read several paragraphs of text, which takes a relatively long time. A characteristic feature of cheating is a long pause 42. After finding the answer, the student with the correct answer formed in the endogenous way proceeds to viewing the proposed options. At this stage, he actually does not differ from a student with confident knowledge of the material. When viewing the correct answer, the P300 component develops in the brain, followed by a mouse click 43.

Thus, the type of solution that the student has resorted to to complete this test task is reflected in the order and speed of events for viewing text elements and clicking the mouse button.

Assessment of the increase in reliability was carried out by simulating probabilistic modeling of the distribution of estimates obtained under various testing scenarios. The table shows the results of the simulation of the test results with 20 test items from the selection of one answer out of 4 possible, and the corresponding assessment of the reliability change in the transition from the standard method of assessing knowledge to the claimed method of assessing student knowledge in computer testing. The probability of a correct answer with the standard method of assessment corresponded to the types of solutions Q _K - 1.0, Q _I - 0.9, Q _P - 0.5, Q _G - 0.25, Q _C - 1.0. During the simulation, twenty correct answers were created programmatically. It should be noted that the likelihood of an error also exists with reliable knowledge - taking into account typical exam stress, random errors by type of typos can amount to 5%. With a probability of a random error of 0.05, the response number shifted to a neighboring position. The simulation was repeated 10,000 times. Modeled the situation:

1. Strong knowledge of the material.

2. Uncertain knowledge of answers to half of the questions, the rest is trying to guess based on residual knowledge.

3. Confident knowledge of half the material, uncertain knowledge of the rest

4. Residual knowledge insufficient to pass the test

5. Random guessing

6. Cheating using extraneous sources

7. Cheating off half of the answers with the use of extraneous sources, the rest is random guessing in the absence of time.

Information sources

1. Lau PNK, Lau SH, Hong KS, Usop H. 2011. "Guessing, Partial Knowledge, and Misconceptions in Multiple-choice Tests". Journal of Educational Technology & Society 14 (4): 99-110. http://www.jstor.org/stable/jeductechsoci.14.4.99.

2. Certainty-Based Marking (CBM), https://docs.moodle.org/30/en/Using_certainty-based_marking.

3. Chang SH, Lin PC, Lin ZC. Measures of partial knowledge and unexpected responses in multiple-choice tests. Educational Technology & Society. 2007 Oct 1; 10 (4): 95-109. http://www.jstor.org/stable/jeductechsoci.10.4.95.

4. Chang SR, Plake BS, Kramer GA, Lien SM. Development and Application of Detection Indices for Measuring Guessing Behaviors and Test-Taking Effort in Computerized Adaptive Testing. Educational and Psychological Measurement. 2011; 71 (3): 437-459. http://dx.doi.org/10.1177/0013164410385110.

5. RU 2521345, A61B 5/0484, publ. 2013-03-22.

6. RU 2528658, A61B 5/0484, publ. 2014-09-20.

7. US 5113870 (A), A61B 5/0484, A61B 5/16, publ. 1992-05-19.

8. WO 02100267 (A9), A61B 5/00, A61B 5/0205, A61B 5/0476, publ. 2004-02-19.

9. KR 20160041748, A61B 5/0476, publ. 2016-04-18.

10. WO 2008056492, A61B 5/0476, publ. 2008-05-15.

11. Hjorth, B., 1975. An on-line transformation of EEG scalp potentials into orthogonal source derivations. Electroenceph. Clin. Neurophysiol. Suppl. 39 (5), 526-530.

12. Butterworth Filter https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.signal.butter.html.

13. Eye-tracker, https://en.wikipedia.org/wiki/Eye_tracking.

14. Random Forest https://en.wikipedia.org/wiki/Random_forest.

15. Partial Dependence Plots http://scikit-learn.org/stable/auto_examples/ensemble/plot_partial_dependence.html.

16. Moodle Learning Management System (https://moodle.org/).

Claims (5)

1. A method of assessing a student’s knowledge in computer testing, characterized by presenting a test task consisting of a question and answer options on a computer screen, registering the time point for starting to view the text of the question and each answer option, the time for the end of viewing the question text and each answer option, time deciding on the choice of the answer option using the manipulator, calculating the integral indicators of the test task, characterizing the sequence and speed of viewing the question and answer options, registering the SSP while viewing the text of the answer option, highlighting the SSP P300 component, if any, concludes that the endogenous image corresponding to the correct answer to the question coincides with the exogenous image that occurs when reading text on a computer screen, classifying the view decisions using a set of decision rules for matching random guessing, cheating, partial knowledge, uncertain knowledge or confident knowledge. 2. The method according to p. 1, in which the question and answer options that make up the test task are presented on the computer screen in the form of visually marked areas of the screen, and the text is hidden. 3. The method according to p. 1, in which the text of the question and answer options are presented when you hover over the corresponding area of the screen. 4. The method according to p. 1, in which the registration of the time point to start viewing the text of the question and each answer option is carried out by hovering over the area of the screen intended for the presentation of the corresponding text. 5. The method according to p. 1, in which the registration of the time point for viewing the text of the question and each answer option is carried out at the moment the cursor is moved outside the screen area intended for the presentation of the corresponding text.

Images