KR102248714B1 - Method and appratus to measure cognitive load based on a time-series analysis of psychophysiological responses in video-based learning - Google Patents

Abstract

Disclosed is an apparatus and method for diagnosing cognitive load according to a learner's prior knowledge and task synthesis in a video learning environment. The cognitive load diagnosis method includes: evaluating a learner's prior knowledge level with respect to the learning content and a task complexity of the learning content; Measuring a basal response to the learner's physiological and psychological responses generated before learning the learning content; Collecting the learner's physiological and psychological responses generated in the process of learning the learning content; And diagnosing the learner's cognitive load by analyzing time-series characteristics of the collected learner's physiological and psychological responses based on the evaluated learner's prior knowledge level and task complexity of the learning content.

Description

Cognitive load diagnosis device and method according to learner's prior knowledge and task combination in video learning environment {METHOD AND APPRATUS TO MEASURE COGNITIVE LOAD BASED ON A TIME-SERIES ANALYSIS OF PSYCHOPHYSIOLOGICAL RESPONSES IN VIDEO-BASED LEARNING}

The present invention relates to an apparatus and method for diagnosing a learner's cognitive load by analyzing a learner's physiological and psychological responses collected in a video learning environment using a time series analysis technique, and providing an appropriate prescription to the learner based on this.

Video learning, which was initially spread in the form of e-learning through desktops and networks, spreads more rapidly with the spread of mobile devices and the emergence of new learning environments such as Massive Open Online Courses (MOOCs) and flip learning. have. Video is a multimedia content that utilizes audio-visual stimuli, and has a media characteristic that teachers and learners can learn'anytime, anywhere' even in a physically separated space. This is a major driver that promoted the spread of the video learning environment, but paradoxically, it brought a limitation in that it was difficult to grasp the cognitive and emotional state of the learner compared to traditional face-to-face learning.

In most video learning environments, since the instructor and the learner are separated spatio-temporally, it is not easy to design content by immediately grasping the learning process of each individual, and it is difficult to expect a smooth interaction between the instructor and the learner. For this reason, it is difficult to apply'adaptive instructional design', which provides individualized support by understanding the learner's unique learning experience in a video learning environment. This externalized to various educational problem phenomena such as learner's negligence and dropout, and as a result, it caused difficulties in achieving the learning goals in relation to the learner's knowledge, skills, and attitudes.

Accordingly, the need for research to understand the inner process experienced by learners in a video learning environment is emerging. In other words, in order to optimize the video learning environment, 1) capture the learner's trait and state in more detail, 2) analyze it in a scientific way, and 3) provide appropriate teaching and learning support. Just-in-time, it should be provided in a suitable way.

In previous studies, post-mortem surveys have been mainly used to grasp the inner process of learners. This is a method of retrospectively measuring learners' subjective thoughts and feelings at the end of learning, and it is possible to obtain summary and subjective information about learners' experiences in the entire learning process. However, it is difficult to obtain rich and objective feedback on what kind of experience the learner has in specific scenes in such a post questionnaire. Accordingly, the teaching design in the past has also been made in a way to meet the universal learning needs of average learners, rather than an adaptive method that reflects individual learners' characteristics and differences in the learning process.

However, with the development of big data collection and analysis technology in recent years, it has become possible to utilize various data accumulated in the learning context, and through this, learning analytics to optimize learning by improving the teaching-learning environment has become a pivotal field.

Much of the research that has attempted a learning analytic approach has been mainly done using clickstream data. Clickstream data refers to behavioral data such as playback and pause during video learning. The basis for analyzing these behavioral data is the assumption that the data can be used as a surrogate variable for the level of immersion in the learning context or for psychological recruitment experiencing confusion. Analyzing the behavioral data left in the video learning environment can be a clue to the overall trend of learning, but only provides limited information on the learner's cognitive and emotional experiences.

The recently developed physiological and psychological data collection technology provides the possibility to grasp the inner process experienced by learners in a learning situation through non-interference and continuous data collection.

The present invention can provide an apparatus and method for analyzing physiological and psychological responses collected from learners during a video learning process using a time series analysis technique.

In addition, the present invention can provide an apparatus and method for providing an appropriate prescription according to the cause of cognitive load by interpreting the analyzed learner's physiological and psychological responses according to the learner's prior knowledge level and the task complexity of the learning content.

A method for diagnosing cognitive load according to an embodiment of the present invention includes the steps of: evaluating a learner's prior knowledge level with respect to learning content and task complexity of the learning content; Measuring a basal response to the learner's physiological and psychological responses generated before learning the learning content; Collecting the learner's physiological and psychological responses generated in the process of learning the learning content; And diagnosing the learner's cognitive load by analyzing time-series characteristics of the collected learner's physiological and psychological responses based on the evaluated learner's prior knowledge level and task complexity of the learning content.

In the evaluating step, the level of prior knowledge of the learner may be evaluated using the achievement level of the prior knowledge test related to the learning content and the previous achievement level of the learner with respect to the learning content among data collected in the learning system.

In the evaluating step, the task composition may be evaluated according to an analysis unit of the learning content by using the number of elements constituting the learning task included in the learning content.

The collecting may include removing outliers based on a normal range of the collected learners' physiological and psychological responses; Linearly interpolating missing values included in the learner's physiological and psychological responses generated by removing the outliers; Converting the linear interpolated learner's physiological and psychological responses into time series data; And applying a data smoothing technique to the learner's physiological and psychological responses converted into the time series data.

The psychophysiological response may include at least one of pupil size, brain waves, and heart rate variability.

The diagnosing may include matching the collected physiological and psychological responses of the learners with the content and time point of the learning content; Dividing the matched learner's physiological and psychological responses according to a learning unit of the learning content to extract a specific pattern; And measuring the learner's cognitive load by analyzing the extracted specific pattern based on the evaluated learner's prior knowledge level and task complexity of the learning content.

It may further include providing a guide for the learning content to the learner based on the diagnosed learner's cognitive load.

The cognitive load diagnosis apparatus according to an embodiment of the present invention includes a processor, wherein the processor evaluates a learner's prior knowledge level with respect to the learning content and a task complexity of the learning content, and is generated before learning the learning content. Measure the base response to the learner's physiological and psychological responses, collect the learner's physiological and psychological responses generated in the process of learning the learning content, and based on the evaluated learner's prior knowledge level and task complexity of the learning content Thus, it is possible to diagnose the learner's cognitive load by analyzing the time-series characteristics of the collected learner's physiological and psychological responses.

The processor may evaluate the level of prior knowledge of the learner using the achievement level of the prior knowledge test related to the learning content and the previous achievement level of the learner with respect to the learning content among data collected in the learning system.

The processor may evaluate the task complexity according to the analysis unit of the learning content by using the number of elements constituting the learning task included in the learning content.

The processor removes outliers based on the collected normal range of the learner's physiological and psychological responses, and linearly interpolates the missing values included in the learner's physiological and psychological responses by removing the outliers, and the linearly interpolated physiological psychology of the learner The response may be converted into time series data, and a data smoothing technique may be applied to the learner's physiological and psychological responses converted into the time series data.

The psychophysiological response may include at least one of pupil size, brain waves, and heart rate variability.

The processor matches the collected physiological and psychological responses of the learners with respect to the content and time point of the learning content, divides the matched physiological and psychological responses of the learners according to the learning unit of the learning content to extract a specific pattern, and the The cognitive load of the learner can be measured by analyzing the extracted specific pattern based on the evaluated learner's prior knowledge level and task complexity of the learning content.

The processor may provide a guide to the learning content to the learner based on the diagnosed learner's cognitive load.

The present invention can more objectively measure the cognitive load based on the physiological and psychological reactions collected in the learning process, and can measure the variation of the cognitive load that appears during the process by analyzing the time-series characteristics of the continuously collected reactions.

In addition, the present invention analyzes the variation of the cognitive load based on the learner's prior knowledge level and task composition, thereby more clearly identifying the cause of the cognitive load, and accordingly, can be used as a basis for providing appropriate instructional design support. .

1 is a diagram illustrating an apparatus for diagnosing cognitive load according to an embodiment of the present invention.
2 is a flowchart illustrating a data preprocessing process according to an embodiment of the present invention.
3 is an example showing a method of extracting a specific pattern for a learner's physiological and psychological responses according to an embodiment of the present invention.
4 to 5 are diagrams illustrating a cognitive load search between learning units according to an embodiment of the present invention.
6 to 9 are diagrams illustrating a cognitive load search in an intermediate learning section according to an embodiment of the present invention.
10 to 16 are diagrams illustrating a cognitive load search in a learning sub-section according to an embodiment of the present invention.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, or substitutes to the embodiments are included in the scope of the rights.

The terms used in the examples are used for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the embodiments, the detailed description thereof will be omitted.

1 is a diagram illustrating an apparatus for diagnosing cognitive load according to an embodiment of the present invention.

Referring to FIG. 1, the apparatus 100 for diagnosing cognitive load of the present invention may include a processor 110. First, as an example, the cognitive load diagnosis apparatus 100 can diagnose a learner's cognitive load for various learning contents, but in the present invention, the predecessor/existence proposition (image 1) and the positive term proposition (image 2) in a mathematics subject. Two videos on the subject were used as learning contents.

The processor 110 may evaluate 111 a level of prior knowledge of a learner with respect to the learning content and a task complexity of the learning content in the video learning environment. More specifically, the processor 110 may evaluate the level of prior knowledge of the learner using the achievement level of the prior knowledge test related to the learning content and the previous achievement level of the learner with respect to the learning content among data collected in the learning system.

The level of prior knowledge can be determined through a pre-test produced by an expert who wants to grasp the level of prior knowledge of the learner. More specifically, the processor 110 may (1) classify the level of prior knowledge of the learner through the learner's test score for the pre-test in which the difficulty level for each question is considered.

In addition, the processor 110 may (2) relatively classify a learner's prior knowledge level based on a distribution of test scores for a pre-test performed by a plurality of learners.

At this time, the processor 110 calculates each learner's pre-test score based on the score for each item of the pre-test according to the difficulty given by the expert in consideration of both methods (1) and (2), and then calculates their distribution. The level of prior knowledge of each learner can be classified as a standard.

In addition, the processor 110 may evaluate the task combination according to the analysis unit of the learning content by using the number of elements constituting the learning task included in the learning content. As an example, the processor 110 combines tasks for each section of the learning image based on three elements of'proposition structure','mathematical sign', and'negative concept' derived through content analysis of image 1 and image 2, which are learning contents. Can be classified.

The task composition is the sum of all the elements that compose the task. As the number of elements increases, the interaction between the elements increases, which can increase the task complexity. Therefore, the processor 110 classifies the elements of the learning content into 1) proposition structure, 2) mathematical sign, and 3) negative concept as shown in Table 1 below, and accordingly, the task combination for image 1 and image 2 Sections can be defined.

<Table 1>

More specifically, the processor 110 may classify the task combination upper and lower according to the learning section of the learning content to examine the difference in cognitive load in the corresponding scene. The learning section was divided in consideration of the content and playback time, and can be divided into three criteria (large section, medium section, and sub section) as shown in Table 2 below.

The learning section and the learning middle section may be classified according to the content of the learning content, and may correspond to the division by subject (video 1/video 2) and the division by content (section 13), respectively. The learning sub-interval corresponds to a finer learning unit and may be generated by dividing each of the images 1 and 2 into 30 second units based on the playback time.

<Table 2>

On the other hand, when the learning content corresponds to the "test question", the processor 110 calculates the difficulty level for the corresponding learning content by using the difficulty level of the test question set in advance by the expert and the response pattern data of learners to the test question. And, the calculated difficulty level can be used for the task combination evaluation.

Next, the processor 110 may measure (112) a basal response to the physiological and psychological response of the learner generated before learning the learning content. Such a baseline response is intended to minimize the physiological and psychological responses due to individual differences between learners, and the measured base response can be used to standardize the learners' physiological and psychological responses that occur during the subsequent learning process.

As an example, the basal response may be calculated as an average of pupil size data collected while a learner views a specific display displayed in the center of a screen made with the same illuminance as the experiment screen. This method of measuring the basal response has the advantage of minimizing the effect that changes in contrast and brightness may have on the pupil size.

Thereafter, the processor 110 may collect 113 the physiological and psychological reactions of the learner generated in the process of learning the learning content. At this time, the collected physiological and psychological responses may include at least one of pupil size, brain wave, and heart rate variability, and the present invention provides a method of diagnosing a learner's cognitive load using the pupil size. However, the physiological psychological response used to diagnose the learner's cognitive load is not limited to the pupil size, and various physiological psychological reactions can be used. The processor 110 may perform pre-processing on the collected learner's physiological and psychological response data, and a detailed pre-processing method will be described with reference to 3 below.

Further, the processor 110 may diagnose 114 the cognitive load of the learner by analyzing the time-series characteristics of the physiological and psychological responses of the learners collected based on the evaluated learner's prior knowledge level and the task complexity of the learning content. .

More specifically, the processor 110 may extract a specific pattern by matching the physiological and psychological responses of the learners collected with respect to the content and time point of the learning content, and dividing the physiological and psychological responses of the matched learners according to the learning unit of the learning content. have. In addition, the processor 110 may measure the learner's cognitive load by analyzing the extracted specific pattern based on the evaluated learner's prior knowledge level and the task complexity of the learning content.

Finally, the processor 110 may provide 115 a guide for learning content to the learner based on the diagnosed learner's cognitive load. More specifically, the processor 110 may provide a guide for identifying problems of learners and feeding back the identified problems by comparing and analyzing the cognitive load of the diagnosed learner with the cognitive load of a group of learners having excellent learning outcomes.

Specifically, the processor 110 of the present invention can diagnose the type of cognitive load of the learner through a time series cluster analysis on the psychophysiological response of the learner, and support an appropriate learning method according to the diagnosed cognitive load type.

To this end, the processor 110 may store an intrinsic cognitive load and a representative pattern in an extrinsic cognitive load situation acquired through a preliminary study. In addition, the processor 110 compares the specific pattern extracted from the physiological and psychological response of the learner at the current point with the representative pattern stored through the prior research, so that each learner experiences a certain cognitive load in the process of learning the learning content. You can determine if there is.

For example, if it is determined that the intrinsic cognitive load is excessively generated on the learner due to high task complexity or difficulty of the learning content, the processor 110 presents learning content similar to the learning content and with low task complexity or difficulty. It can help learners to solve the learning content by allowing them to become proficient in the relevant content.

Alternatively, if it is determined that there is a section in which the external cognitive load commonly experienced by learners is excessively generated in relation to the learning content, the processor 110 matches the gaze movement information among the learners' physiological and psychological responses to teach the learning content. It can lead to modifications to design elements.

2 is a flowchart illustrating a data preprocessing process according to an embodiment of the present invention.

In step 210, the cognitive load diagnosis apparatus 100 may remove the outlier based on the collected normal range of the physiological and psychological responses of the learner. The cognitive load diagnosis device 100 is a value that is outside the general pupil size category (2.0-7.0mm) or outside the 2.5 SD cut-off (2.5 standard deviation point from the average) within the data for each participant (learner) is a missing value. Can be processed with

In addition, the cognitive load diagnosis apparatus 100 may treat a value outside the normal range of the pupil response according to the cognitive processing demand as an outlier value and treat it as a missing value. In the present invention, the pupil response was calculated by subtracting the individual baseline response from the observed value, not the collected pupil size itself, and the degree of pupil dilation. It is known that the pupil response does not exceed 1.0mm when this method is used. Accordingly, the cognitive load diagnosis apparatus 100 may treat the observed value as a missing value when the degree of change exceeds 1.0mm when the individual's baseline response is subtracted from the observed value.

In addition, the cognitive load diagnosis apparatus 100 may classify outliers based on the distance between the measuring device and the eyes. In the case of using a fixed eye tracker as in the present invention, it is known that accurate measurement is difficult if the distance between the measuring device and the eye is too close or far. The device used in the experiment has an algorithm that can correct the problem of pupil size distortion that may occur due to distance. However, the cognitive load diagnosis apparatus 100 may remove some data in which the distance between the eyes and the device is out of the recommended range (35-95cm) due to changes in the posture of the participant (learner) or the like.

In step 220, the cognitive load diagnosis apparatus 100 may linearly interpolate the missing values included in the learner's physiological and psychological responses generated by removing the outliers. In order to reduce unnecessary noise and integrate seamlessly with other data, the cognitive load diagnosis apparatus 100 may sum the observations of both pupil sizes excluding missing values among data collected at 1/30 second intervals while learning an image as an average per second. . Subsequently, the missing value determined as an outlier and removed may be corrected through linear interpolation.

In step 230, the cognitive load diagnosis apparatus 100 may convert the linearly interpolated learner's physiological and psychological responses into time series data. More specifically, the cognitive load diagnosis apparatus 100 may convert the pupil response data collected by performing time series analysis into time series data.

This can convert the existing data, in which the pupil response for each time point is expressed as a point, into a continuous time series object. In addition, the time information expressed in the collection time (eg, 2017-03-07 10:58:00 AM) is converted into a timestamp-type index (eg 1, 2, 3, 4…) so that the learners can video Facilitates comparison between pupil reactions seen in specific areas of.

In step 240, the cognitive load diagnosis apparatus 100 may apply a data smoothing technique to the learner's physiological and psychological responses converted into time series data. More specifically, the cognitive load diagnosis apparatus 100 may smooth the time series data using a Locally weighted scatterplot smoothing (Lowess) technique in order to reduce unnecessary noise and identify changes in pupil response mainly on important trends.

3 is an example showing a method of extracting a specific pattern for a learner's physiological and psychological responses according to an embodiment of the present invention.

The present invention provides a method of analyzing pupil response using a time-series approach to understand the cognitive load experienced by learners in detailed learning scenes of learning content. Time series analysis aims to identify changes in data over time and extract specific values. To this end, in the present invention, cluster analysis may be used in order to first grasp the aspect of data through visualization and to analyze a reaction occurring in a more detailed unit.

Clustering is a statistical technique that categorizes individuals in close order by measuring the degree of similarity or dissimilarity between individuals using the characteristics of observations. Cluster analysis is an analysis technique that can be used without prior assumptions about the distribution or pattern of data.

An object of the present invention is to understand a learner's cognitive load through a change in pupil response that appears during a video learning process. Research to understand human cognitive processes through changes in pupil response over time is increasing, but there are no known models or criteria for categorization in this regard. Therefore, cluster analysis to determine'which data and which data are more similar' may correspond to an appropriate exploratory technique that can be used in this case.

Time-series clustering differs from commonly known cluster analysis in that individuals grouped into groups based on similarity are not'people' but'time series data'. The time series clustering method is largely whole time-series clustering, which derives clusters based on the similarity of all data, and subsequence clustering, which derives clusters based on the similarity between parts extracted from time series data. , And time-point clustering in which clusters are derived based on the similarity of peculiar points such as peaks.

The total time series clustering has a weakness that it does not reflect a significant pattern that appears locally when the total time series data is long. On the other hand, because viewpoint clustering is sensitive to specific outliers, it has a limitation that it is difficult to derive meaningful information. Accordingly, the cognitive load diagnosis apparatus 100 of the present invention utilizes sub-sequence clustering to derive sub-sequence data in a meaningful minimum unit and categorize the data based on similarities between them.

Referring to FIG. 3, a cluster analysis process performed by the cognitive load diagnosis apparatus 100 of the present invention is shown. First, the cognitive load diagnosis apparatus 100 may divide the pupil response data of all learners collected while learning each image into learning sub-section units (30 second intervals). This may correspond to a step of generating sub-sequence data, that is, an object to be a candidate for the cluster.

Next, the cognitive load diagnosis apparatus 100 may measure similarity between sub-sequence data using the CORT method. In terms of grouping individuals based on similarity or dissimilarity of observations, cluster analysis can be a very important decision-making factor in the analysis.

The cognitive load diagnosis apparatus 100 of the present invention needs to consider both the magnitude and direction of the pupil reactions in order to determine the similarity between the pupil reactions. This is because the degree of pupil dilation at each point in time is directly related to the degree of cognitive load, and the directionality shows the pattern of changes occurring within the section.

The CORT method can be a similarity measurement method that takes full advantage of the fact that it reflects the signal size, which is the advantage of the Euclidean distance measurement method, and that it reflects the dynamics of the data, which is the advantage of the COR method that uses a correlation coefficient. Accordingly, the cognitive load diagnosis apparatus 100 may measure a distance between sub-sequence data using the CORT method.

In addition, the cognitive load diagnosis apparatus 100 may form a cluster using a ward connection method. The ward connection method determines a group by considering the sum of squared distances between all variables between groups, and minimizes the sum of squared distances within the group.

The cognitive load diagnosis apparatus 100 may measure a learner's cognitive load by analyzing the clustered result in consideration of the learner's prior knowledge level and the task complexity level of the learning content.

4 to 5 are diagrams illustrating a cognitive load search between learning units according to an embodiment of the present invention.

To examine the overall cognitive load experienced by learners in the learning zones classified into image 1 (formerly existential proposition) and image 2 (both term proposition) according to the subject 1) Verifying the difference in pupil response between the upper and lower groups of prior knowledge and 2 ) The difference in pupil response was verified by the upper and lower sections of the task combination.

<1. Differences in pupil response between groups above and below prior knowledge>

Since the sample size of each group was not large and the normality was not satisfied, the difference in the mean pupil response between the two groups was verified using the Wilcoxon rank sum test, a nonparametric method. As a result of the analysis, the difference in pupil response between the upper and lower groups was not statistically significant in both image 1 (p=.49) and image 2 (p=.44). However, it can be seen from FIG. 4 that the difference in pupil response between the two groups in image 2 (both anti-proposition) was greater than that in image 1 (formerly existent proposition).

<2. Differences in pupil response by the upper and lower sections of the task combination>

Whether there was a difference in pupil response between image 1 (pre-existing proposition) and high image 2 (both term proposition) with low task complexity was confirmed by the prior knowledge upper-lower group through the Wilcoxon code ranking test, a non-parametric method. As a result of the analysis, there was no statistically significant difference between the pupil response in image 1 (pre-existing proposition) and pupil response in image 2 (double antithesis) in both upper group (p=.97) and lower group (p=.10). Did. The distribution of data is visually shown in FIG. 5.

As for the box size on the boxplot, which means between the first and third quartiles of the data distribution, there was no significant difference according to the level of task complexity in both the upper and lower groups. However, in the lower group, it was confirmed that the median value of the pupil response data increased significantly as the task complexity increased. In the upper group, it was confirmed that the median value was rather lowered.

6 to 9 are diagrams illustrating a cognitive load search in an intermediate learning section according to an embodiment of the present invention.

6 to 9, a more detailed analysis unit was set to understand the cognitive load of the learner that changes during the learning process. To this end, the pupil response was not summarized in the form of a single value, but a visualization was performed to express the data by utilizing the continuity. This was carried out in two ways: 1) visualization of the pupil response of all participants and 2) visualization of changes in pupil response by group in the middle learning section classified according to the learning content.

<1. Visualization of pupil response of all participants>

In order to confirm the overall trend of the data, the pupil responses of all participants collected in Image 1 (formerly existent proposition) and Phase 2 (both term proposition) were displayed in the form of a continuous line. 6 is a result of visualizing the pupil response of a sieve learner collected from image 1 (formerly known as the existence proposition). The total data is distributed between about -0.27mm and about 0.95mm.

7 is a result of visualizing the pupil response of all learners collected in Image 2 (Banghang Myungje). The total data is distributed between about -0.40mm and 1.00mm. As can be seen in FIGS. 6 and 7, even though basic pre-processing was performed, it was difficult to find any features or trends simply by visualizing the entire time series data.

Therefore, the change in pupil response for each group above and below the prior knowledge was represented by a line and then divided into units of learning intervals. (See Figs. 8 to 9) A time index indicating the playback time of the learning image is displayed on the x-axis of the graph. , on the y-axis, the pupil response at the time point was indicated. The red and green colors of the line represent the upper and lower groups, respectively, and the gray shaded part surrounding the line represents the 95% confidence interval. The combination of tasks for each section is presented at the bottom of each section.

<2. Visualization of pupil response by group>

The change in pupil response observed in the learning process is described in terms of signal amplitude and trend. First, looking at the change in image 1 (formerly known as the existence proposition) with low task complexity, the pupil response of the upper and lower groups from sections 1-1 to 1-4 showed similar patterns in terms of size and direction of increase and decrease. In intervals 1-5 and 1-6, both groups showed a downward and upward trend, so they were similar in terms of tendency, but the difference began to be noticeable in terms of size.

In section 1-5, the upper group declined sharply and then rebounded again, while the lower group remained relatively modest. The size difference was observed to range from 0.5mm to 1.0mm. In section 1-6, the response of the upper group increased at the beginning of the section, then declined in the middle, and then rebounded at the end of learning. On the other hand, in the case of the lower group, the trend continued from 1 to 5, and then gradually decreased from the middle to the end of learning. The difference in pupil response between the two groups was up to 0.7 mm.

The pupil response in image 2 (positive proposition) with high task complexity showed greater variation among groups than in image 1 (formerly existent proposition). In section 2-1, which belongs to the starting point of learning, both groups showed a decreasing pattern as observed in section 1-1. The size of the pupil response was also similar. In section 2-2, the response of the upper group continued to decline, but the lower group showed a gentle upward curve. The difference in the size of the pupil response was also observed.

On the other hand, in section 2-3, the upper group showed a rising pattern and the lower group showed a gentle decline. The difference in size was reduced, and after the middle, the difference was within the margin of error. In section 2-4, the response of the upper group declined and then rose again steeply, and the lower group gradually increased.

The magnitude of the response maintained the difference between the two groups until the middle and then decreased. In section 2-5, the upper group showed a brief rebound at the mid point and then showed a tendency to decline overall, whereas the lower group showed a steep rise from the middle. The difference in the size of the response was not significant until the middle of the day, but the pattern was observed to take place rapidly afterwards.

In section 2-6, the upper group continued to show a modest upward trend, while the lower group showed a downward trend until the beginning and then increased. The difference in the size of the pupil response between the two groups was not significant. In the last section, 2-7, the upper group fell sharply, while the lower group remained modest. The difference in response magnitude between the two groups widened greatly from the start point and was similar at the end point.

10 to 16 are diagrams illustrating a cognitive load search in a learning sub-section according to an embodiment of the present invention.

Referring to FIGS. 10 to 16, the pupil reactions shown for each learning sub-section divided based on the playback time were examined in order to search in a more detailed unit. The analysis consists of 1) clustering using the sub-sequence of pupil response collected during the learning process, 2) verifying the difference in pupil response between the upper and lower groups of prior knowledge based on the ratio of the pattern corresponding to the clustering result, and 3) the pattern ratio. The difference in pupil response in the upper and lower sections of the used task combination was verified in order.

<1. Cluster analysis results using sub-sequences of pupil responses collected during the learning process>

Gap statistic was checked to determine the number of clusters prior to analysis. Referring to FIG. 10, it was confirmed that the appropriate number of clusters for distinguishing sub-sequences in image 1 (formerly existent proposition) and image 2 (both term proposition) is seven.

Based on this, the sub-sequence data of image 1 (formerly known as existential proposition) was classified into 7 clusters, and the results were defined as seven patterns (pattern 1 to pattern 7). Data belonging to the same cluster were similar in two aspects: 1) the magnitude of the response and 2) the slope of the graph, as shown in FIG. 11.

In order to understand the overall trend of each pattern, a representative line of the subsequence clustering result was visualized as shown in FIG. 12. When patterns are classified according to the slope, patterns 1 and 5 take the form of monotonic decrease, and patterns 2 and 7 take the form of monotonic increase. Pattern 3, Pattern 5, and Pattern 6 showed a complex shape, with Pattern 3 rising and falling, and Pattern 4 and Pattern 6 decreasing and then rising. The width of change in pupil response was observed between about 0.05mm and 0.15mm in all patterns.

The result of classifying the sub-sequence data of Image 2 (Bang Hang Title) into 7 clusters is shown in FIG. 13. As in image 1, the data classified into the same cluster were similar in two aspects: 1) the magnitude of the response and 2) the slope of the graph. In order to understand the overall trend of each pattern, the result of subsequence clustering of image 2 (positive title) was also confirmed by drawing a representative line as shown in FIG. 14.

When the patterns are classified by direction, Pattern 1, Pattern 2, and Pattern 5 can be divided into monotonically decreasing forms, and Pattern 3, Pattern 4, Pattern 6, and Pattern 7 can be divided into monotonically increasing forms. Pattern 2, Pattern 3, and Pattern 4 had fluctuations not exceeding 0.1 mm, whereas Pattern 1, Pattern 5, Pattern 6, and Pattern 7 had a large fluctuation range from 0.2 mm to 0.3 mm.

<2. Difference in cognitive load in image 1>

end. Verification of differences in the ratio of pupil response patterns between groups above and below prior knowledge

Prior to examining whether there was a difference in cognitive load between groups above and below prior knowledge in the learning sub-section, the result of sub-sequence clustering was matched with the information of each participant and section. This is to interpret the results by matching the clustering results with the level of prior knowledge and task complexity. 15 is a summary of the results in a matrix form.

The information presented in FIG. 15 will be described in detail as follows. The first and second columns refer to the participant's information, and each refers to an ID, which is an identifier for distinguishing between the prior knowledge group and the participant. The data was organized in the order of prior knowledge upper and lower groups, and in ascending order of ID. The letters marked from the third column indicate the name of the learning subsection. Since the learning sub-sections were created at 30 second intervals, section 1-A means from 1 to 30 seconds and section 1-B means from 31 to 60 seconds based on the playback time. The numbers from 1 to 7 placed under the section name mean the result of cluster analysis, that is, the result of classifying the pupil response in the section into a pattern.

In order to examine whether there is a difference in cognitive load experienced by learners according to the level of prior knowledge in the unit of learning subsection of image 1 (formerly known as existential proposition), the difference in the distribution of pupil response patterns was verified using a chi-square test. As a result of the analysis in the entire section including the upper and lower sections of the task complex, there was a difference in the pupil response pattern ratio between the prior knowledge upper and lower groups with a significance probability of .01 as shown in Table 3 below.

<Table 3>

In the prior knowledge group, pattern 7 was the most common at 25.7%, and pattern 5 and pattern 1 accounted for 21.9% and 19.3%, respectively. The least appeared pattern was pattern 2, which appeared 11 times (5.9%).

In the pupil response of the group under prior knowledge, pattern 7 was the most common with 21.3%, followed by pattern 3 and pattern 5. The least appeared pattern was Pattern 2 (6 times, 4.4%), similar to the prior knowledge group.

As a result of checking whether there was a difference in the pupil response pattern between the upper and lower groups of prior knowledge according to the level of task complexity, the significance probability in the upper section was .23, and the difference between the groups was not significant.

On the other hand, in the section under the task combination, the difference in the pattern between groups was significant as shown in Table 4 below with a significance probability of .02. Looking at the results, pattern 7 was the most common and pattern 2 was the least in the pupil response of the prior knowledge group in the task complex phase.

<Table 4>

In the pupil response of the group under prior knowledge, Pattern 1, Pattern 3, Pattern 5, and Pattern 7 appeared most at the same frequency, and Pattern 2 accounted for the smallest percentage. On the other hand, pattern 1 appeared most frequently in the pupil response of the group based on prior knowledge in the lower part of the task combination, but pattern 7 appeared most frequently in the lower group. The least appeared pattern was the same as pattern 2.

I. Verification of differences in the ratio of pupil response patterns for each upper and lower section of the task synthesis

In order to understand the change in cognitive load according to the level of task combination, the difference in the ratio of the pupil response pattern that appeared in the upper and lower sections of the task combination was verified as shown in Table 5 below. First, as a result of verifying the difference in the proportion of pupil response patterns in the upper and lower sections of the task combination without classifying the upper and lower groups of prior knowledge, there was no difference with a significance probability of. In the task combination phase, pattern 5 was the most common at 21.1%, and pattern 1 and pattern 3 accounted for 16.4% and 12.9%, respectively.

<Table 5>

On the other hand, in the section under the combination of tasks, pattern 7 was the most common at 22.4%, and pattern 5 and pattern 1 accounted for 21.7% and 18.4%, respectively.

In image 1 (formerly the existence proposition), it was verified whether there was a difference in the cognitive load of each task combination upper and lower section of the prior knowledge group. To this end, as a result of verifying whether there is a difference in the ratio of the pupil response pattern by the task combination upper and lower sections for each group, prior knowledge upper group (p=.53) and lower group (p=.70) both pupils according to the level of task complexity. It can be seen from Table 6 below that the difference is not significant in the response pattern ratio.

<Table 6>

<3. Difference in cognitive load in video 2

end. Verification of differences in the ratio of pupil response patterns between groups above and below prior knowledge

Sub-sequence clustering was performed as shown in Fig. 16 in order to examine whether there was a difference in cognitive load between the upper and lower groups of prior knowledge in the unit of the learning sub-section of Image 2 (Banghang Proposition). 16 shows the pupil response pattern matched with the prior knowledge level of each learner and the level of task complexity in the section, as in Image 1 (formerly known as the existence proposition).

The proportion of pupil response patterns between the upper and lower groups in the entire interval including the upper and lower sections of the task combination was significantly different, with a significance probability of .01. The detailed frequency table is shown in Table 7 below.

<Table 7>

Looking at the results, in the prior knowledge group, Pattern 2 was the most common with 35.2%, and Pattern 3 and Pattern 6 accounted for 24.1% and 13.0%, respectively. The least appeared pattern was pattern 7 and appeared 6 times (2.4%). In the group under prior knowledge, pattern 3 appeared most at 25.0%, followed by pattern 2 and pattern 1, in that order. The least appeared pattern was Pattern 7 (3 times, 1.6%) as in the prior knowledge group.

As a result of checking whether there was a difference in the pupil response pattern between the prior knowledge upper and lower groups in the upper and lower sections of the task combination, the difference between the groups was statistically significant in the upper section (p=.01).

On the other hand, in the section under the task combination, the difference between groups was not significant as the significance probability was. Table 8 below shows the results of dividing the pupil response patterns of the prior knowledge upper and lower groups for each task combination section.

<Table 8>

Looking at the results, in the pupil response of the prior knowledge phase group in the task complex phase, pattern 2 was observed most frequently, and pattern 7 was the least. In the group under prior knowledge, pattern 2 was the most common and pattern 7 was the least.

On the other hand, when looking at the pupil response of the group based on prior knowledge in the section under task combination, pattern 2 was the most, but pattern 3 was the most in the lower group. The least appeared pattern was pattern 7 in both the upper and lower groups.

I. Verification of differences in the ratio of pupil response patterns for each upper and lower section of the task synthesis

To understand the change in cognitive load according to the level of task complexity, the difference in pupil response patterns in the upper and lower sections of the task combination was analyzed. First, as a result of analyzing the difference for all without classification according to the level of prior knowledge, it can be seen from Table 9 below that there is a statistically significant difference in the proportion of the pupil response pattern with a significance probability of .02.

<Table 9>

In Image 2 (Banghang Proposition), it was verified whether there was a difference in cognitive load for each upper and lower section of the task combination of the prior knowledge group. To this end, as a result of verifying whether there is a difference in the proportion of pupil response patterns in the upper and lower sections of the task combination for each group, the difference was statistically statistically significant in both the upper group (p=.08) and the lower group (p=.11) of prior knowledge. It can be seen through Table 10 below that it is not significant.

<Table 10>

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to operate as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the following claims.

100: cognitive load diagnosis device
110: processor

Claims (14)

Evaluating, by a processor, a learner's prior knowledge level with respect to the learning content and a task complexity of the learning content;
Measuring, by the processor, a baseline response to the learner's physiological and psychological responses generated before learning the learning content;
Collecting, by the processor, data on physiological and psychological responses of the learner generated in the process of learning the learning content;
Generating, by the processor, sub-sequence data by dividing the collected physiological and psychological response data of the learner at predetermined intervals;
Measuring, by the processor, a distance between the generated sub-sequence data by using the size and direction of the physiological and psychological response data of the learner constituting the generated sub-sequence data;
Forming, by the processor, a cluster that minimizes a sum of squares of distances between the sub-sequence data by using the measured distances between the sub-sequence data; And
Diagnosing the learner's cognitive load by analyzing, by the processor, the clustered result based on the learner's prior knowledge level and the task complexity of the learning content.
Cognitive load diagnosis method comprising a. The method of claim 1,
The evaluating step,
A cognitive load diagnosis method for evaluating the level of prior knowledge of the learner by using the achievement level of a prior knowledge test related to the learning content and the previous achievement level of the learner for the learning content among data collected in the learning system. The method of claim 1,
The evaluating step,
A cognitive load diagnosis method for evaluating a task complexity according to an analysis unit of the learning content by using the number of elements constituting the learning task included in the learning content. The method of claim 1,
The collecting step,
Removing outliers based on the normal range of the collected learner's physiological and psychological response data;
Linearly interpolating missing values included in the learner's physiological and psychological response data generated by removing the outliers;
Converting the linearly interpolated learner's physiological and psychological response data into time series data; And
Applying a data smoothing technique to the learner's physiological and psychological response data converted into the time series data
Cognitive load diagnosis method comprising a. The method of claim 1,
The physiological and psychological reaction,
A method for diagnosing cognitive load comprising at least one of pupil size, brain waves, and heart rate variability. The method of claim 1,
The step of diagnosing,
Matching the clustered learner's physiological and psychological response data with the content and time point of the learning content;
Dividing the matched learner's physiological and psychological response data according to a learning unit of the learning content to extract a specific pattern; And
Measuring the learner's cognitive load by analyzing the extracted specific pattern based on the evaluated learner's prior knowledge level and task complexity of the learning content
Cognitive load diagnosis method comprising a. The method of claim 1,
Providing, by the processor, a guide to the learning content to the learner based on the diagnosed learner's cognitive load
Cognitive load diagnosis method further comprising a. In the cognitive load diagnosis device,
Including a processor,
The processor,
The process of evaluating the learner's prior knowledge level with respect to the learning content and the task complexity of the learning content, measuring the base response to the learner's physiological and psychological responses that occurred before learning the learning content, and learning the learning content Collects the learner's physiological and psychological response data generated in, divides the collected learner's physiological and psychological response data at regular intervals to generate sub-sequence data, and the learner’s physiological psychology constituting the generated sub-sequence data The distance between the generated sub-sequence data is measured by using the size and direction of the response data, and a cluster is formed to minimize the sum of the distance squares between the sub-sequence data by using the measured distance between the sub-sequence data, and the clustering Cognitive load diagnosis device for diagnosing the learner's cognitive load by analyzing the result based on the learner's prior knowledge level and task complexity of the learning content. The method of claim 8,
The processor,
A cognitive load diagnosis apparatus for evaluating the level of prior knowledge of the learner using the achievement level of a prior knowledge test related to the learning content and the previous achievement level of the learner for the learning content among data collected in the learning system. The method of claim 8,
The processor,
A cognitive load diagnosis apparatus for evaluating a task complexity according to an analysis unit of the learning content by using the number of elements constituting the learning task included in the learning content. The method of claim 8,
The processor,
Based on the normal range of the collected learner's physiological and psychological response data, outliers are removed, and the missing values included in the learner's physiological and psychological response data are linearly interpolated, and the linearly interpolated learner's physiological and psychological responses A cognitive load diagnosis apparatus for converting data into time series data and applying a data smoothing technique to the learner's physiological and psychological response data converted to the time series data. The method of claim 8,
The physiological and psychological reaction,
Cognitive load diagnosis apparatus including at least one of pupil size, brain waves, and heart rate variability. The method of claim 8,
The processor,
Matching the clustered learner's physiological and psychological response data with respect to the content and time point of the learning content, dividing the matched learner's physiological and psychological response data according to the learning unit of the learning content to extract a specific pattern, and the evaluation Cognitive load diagnosis apparatus for measuring the cognitive load of the learner by analyzing the extracted specific pattern based on the prior knowledge level of the learned learner and the task complexity of the learning content. The method of claim 8,
The processor,
Cognitive load diagnosis apparatus for providing a guide to the learning content to the learner based on the diagnosed cognitive load of the learner.

Images