KR20220135846A - Learner analysis and care system using emotional analysis technology - Google Patents

Abstract

The present invention relates to a learner analysis and care system using emotional analysis technology. The learner analysis and care system using emotional analysis technology comprises: an application program for learners; an application program for teachers; and an operation server which is connected with the application programs for learners and teachers and takes charge of operating a self-directed learning providing service. The operation server monitors and analyzes a learning state of a learner based on analyzed emotional analysis data when emotional analysis of the learner based on collected face recognition data is performed, and provides a personalized learning prescription that can enhance a learning activity for each learner based on a learning state analysis result.

Description

Learner analysis and care system using emotional analysis technology {LEARNER ANALYSIS AND CARE SYSTEM USING EMOTIONAL ANALYSIS TECHNOLOGY}

The present invention uses the emotional analysis data of the learner obtained through the facial recognition technology based on the face image to comprehensively monitor and analyze the learning state and activity of the learner, and accordingly, a learning prescription that can strengthen the learning activity of the learner It is about the learner analysis and care system that provides.

According to various recent reports, the paradigm of the future education market is foreshadowing a change to smart learning. According to the report of the Korea Institute for Advancement of Technology (KIAT) (“U.S. Key Industrial Technology Policy Analysis Report_Changes in the Education Market due to the Distribution of Smart Devices”), the future education market will change from the offline education method prior to the 1990s to online education due to the spread of the Internet. Beyond the e-learning method that takes place online, it is predicted that it will evolve into an offline-online integrated blended type smart learning that combines online/mobile technology-based education and offline face-to-face education. .

However, until now, many online education services and mobile apps that have been commercialized and provided under the guise of so-called smart learning remain at the level of simple e-learning that implements the offline education method online as it is, or Core properties-that is, self-directed (S: Self-directed), interest-induced (M: Motivated), personalized level (A: Adaptive), rich content (R: Resource free), various IT technologies (mobile technology, cloud Computing technology, big data processing technology, etc.) and technology convergence (T: Technology embedded) were clearly limited. In addition, smart learning technologies so far have not been able to implement a true offline-online integrated (that is, blended type) education model that combines online/mobile technology-based education and offline face-to-face education.

To overcome the above-mentioned limitations, domestic and foreign education-related companies are continuously upgrading various services to provide personalized and customized learning and to improve learning effects. This service advancement is evolving into a service that grafts AI (artificial intelligence) technology into education or uses multi-modal data such as voice, gaze, and facial expression for education in addition to text-based learning data. Multi-modal data refers to various data that can be acquired from a user, such as voice, gaze, facial expression, bio-signals, and behavior.

However, despite these service advancement efforts, the learner's learning status and activities are still used using the analysis results of various types of multi-modal data that can be acquired from the learner (especially, the emotional analysis results obtained through facial recognition among them). A system that comprehensively diagnoses and provides a personalized learning prescription that can reinforce learning activities for each learner has not yet been specifically proposed.

The present invention was derived to solve the above-described problems, and comprehensively monitors and analyzes the learning state and activity of the learner by using the emotional analysis data of the learner obtained through the facial recognition technology based on the face image, and, accordingly, the learner The purpose of this study is to provide a method for implementing a learner analysis and care system that provides personalized learning prescriptions that can enhance learning activities for each individual.

According to one aspect of the present invention, there is provided a learner analysis and care system using emotional analysis technology, comprising: an application program for learners installed in a learner terminal to provide self-directed learning; A teacher application that enables communication with learners in self-directed learning or coaching and management for the learners through web access or app access; It interlocks with the learner and teacher application programs and includes an operation server responsible for service operation for the provision of self-directed learning,

When the emotional analysis of the learner based on the collected facial recognition data is performed, the operation server monitors and analyzes the learning state of the learner based on the analyzed emotional analysis data, and learning activities for each learner based on the learning state analysis result A learner analysis and care system using emotion analysis technology can be provided, characterized in that it provides a personalized learning prescription that can strengthen

According to the learner analysis and care system using the emotion analysis technology according to the embodiment of the present invention, the learner's learning state and activities are comprehensively monitored using the learner's emotional analysis data obtained through the facial recognition technology based on the face image. and analysis, there is an effect that it is possible to provide a personalized learning prescription that can strengthen the learning activity for each learner.

1 is a diagram conceptually illustrating an overall system configuration example of a learner analysis and care system using emotion analysis technology according to an embodiment of the present invention.
2 and 3 are diagrams for explaining a detailed implementation flow of a learner analysis and care system using emotion analysis technology according to an embodiment of the present invention.
4 is a diagram illustrating an example of a learner care method before learning among the detailed implementation flow shown in FIG. 3 .
5 is a diagram illustrating an example of a learner care method during and after learning among the detailed implementation flow shown in FIG. 3 .

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

In describing the present invention, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, numbers (eg, first, second, etc.) used in the description process of the present specification are merely identification symbols for distinguishing one component from other components.

In addition, throughout the specification, when an element is referred to as "connected" or "connected" with another element, the one element may be directly connected or directly connected to the other element, but in particular It should be understood that, unless there is a description to the contrary, it may be connected or connected through another element in the middle. In addition, throughout the specification, when a part "includes" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated. In addition, terms such as "unit" and "module" described in the specification mean a unit that processes at least one function or operation, which means that it can be implemented as one or more hardware or software or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[Entire system configuration]

1 is a diagram conceptually illustrating an overall system configuration example of a learner analysis and care system using emotion analysis technology according to an embodiment of the present invention. Hereinafter, with reference to FIG. 1, an overall system configuration of the present invention will be described.

Referring to Figure 1, the learning analysis system according to an embodiment of the present invention, a plurality of learner terminals (100-1, 100-2, ... 100-n, hereinafter referred to as reference number 100); at least one teacher terminal 200; The learner terminal 100 and the teacher terminal 200 are network-linked, and to implement the learning analysis system according to the embodiment of the present invention, the member DB (Database), the learning content DB, etc. are included or driven in conjunction with the corresponding DBs. It may be implemented including the operation server 300 to be.

Here, various terminals (eg, desktop, notebook, tablet PC, etc.) may be used as the learner terminal 100 , and the learner terminal 100 in which the learner application program according to an embodiment of the present invention is installed. can The learner application program may be implemented as a web version or as a mobile app (App). The learner application program is used by the learner for self-directed learning using predetermined digital learning content, and the main functions related to the present invention will be described later.

The teacher terminal 200 may also be used in various terminals, and the teacher terminal 200 may be used for communication with a learner conducting self-directed learning or for the learner through a Web connection or an App connection. A teacher application may be installed to enable coaching and administration.

The operation server 300 stores various data collected through the learner application program (eg, each student's progress-specific learning data, learning time data, app use/registration data, etc.), and It links and visualizes the teacher application program and the data to enable monitoring, and plays a role in providing various learning contents and managing members. In particular, the operation server 300 analyzes and prescribes various learning activities and states based on emotion analysis using facial recognition technology in relation to the present invention, and main functions related to the present invention will be described later.

In the conceptual diagram of FIG. 1, it is shown that only one operation server is operated for service operation, but this is only for the convenience of the drawings, and it is obvious that various modifications are possible depending on the system design method. For example, if necessary according to the role and function of the operation server 300, it may be subdivided and operated such as a member DB server, a learning content DB server, a learning analysis server, a management server, and the like.

Hereinafter, with reference to FIGS. 2 to 5 , a detailed implementation method of a learner analysis and care system using emotion analysis technology according to an embodiment of the present invention will be described.

Here, FIGS. 2 and 3 are diagrams for explaining a detailed implementation flow of a learner analysis and care system using emotion analysis technology according to an embodiment of the present invention, and FIG. 4 is a learning among the detailed implementation flows shown in FIG. 3 . It is a diagram illustrating an example of a learner care method before, and FIG. 5 is a diagram illustrating an example of a learner care method during and after learning among the detailed implementation flows shown in FIG. 3 .

From the perspective of cognitive science (brain science), emotion is a state of mind related to the nervous system and corresponds to a decisive factor in rational decision-making. In addition, the learner's emotions become the basic background of thoughts, actions, and learning processes, and serve as an on-off switch to learning. Therefore, minimizing negative emotions and maximizing positive emotions helps to activate the brain's cognitive processing (higher-order thinking) in the learning process.

According to a number of research results, from the above perspective, students with low academic achievement tend to have lower positive emotions and higher negative emotions than general students. This is because the learner's negative emotions and emotions interfere with the operation for processing cognitive information, which ultimately negatively affects academic achievement. As such, emotions have a great influence on the learner's attention, motivation, and use of learning strategies, so positive emotions and emotions are essential for the reinforcement of learning and memory.

As described above, in order to increase the learner's learning effect, the learner's emotion regulation strategy is very important, and the present invention provides the learner's multi-modal data (especially, the learner's facial expression and/or heartbeat acquired through the learner's face recognition). data) to monitor and analyze the learner's emotions and status, and based on the analysis result, in the pre-learning stage, the learner's negative emotions are converted into positive emotions, so that the learner can learn in a positive emotional state. In the stage of learning, care is taken so that the learner can concentrate on the corresponding learning, and in the stage after learning is completed, by deriving 5 state indices related to the emotions felt by the learner and providing a personalized learning prescription, The purpose is to enhance the effectiveness of the learner's learning activities.

Accordingly, according to the system of the present invention, as shown in FIG. 2, emotion analysis and state index analysis are performed through the operation server 300 and the emotion recognition engine of FIG. 1 described above, and the analyzed result is It is prepared as an emotional analysis result report (ie, the Emotional A.I result analysis report of FIG. 2 ), and the report is shared among learners, teachers, and parents.

Hereinafter, care and prescription methods at each stage (ie, pre-learning stage, learning stage, post-learning stage) through the system of the present invention will be described in turn with reference to FIGS. 3 to 5 .

[How to care for learners before learning]

3 and 4, in the pre-learning stage, the system of the present invention motivates learning through diagnosis and care of the learner's emotional state and induces the learner to start learning with a more positive mind To do this, we analyze the emotional state of the learner before learning, and execute different messages and emotional reinforcement measures according to the analyzed emotional state.

As a preliminary step of the above-described emotional state analysis, when the learner logs in to the corresponding app (APP) for digital learning, the learner application program acquires the learner's face image in conjunction with the camera.

As an example, when the digital learning app (APP) is executed through the learner login process, the learner application is processed so that the operation of the camera mounted on or connected to the learner terminal is automatically executed in conjunction with the execution of the app, and the camera It is possible to acquire the face image of the learner taken through the In this case, the acquired face image of the learner is used as the image for analyzing the above-described emotional state, and may be stored in the memory.

As another example, when the above-described learner login process itself is implemented as a login process through facial recognition instead of by a conventional ID/Password, the learner application program records the learner's face image obtained in the facial recognition login process as described above. It can be used as it is as an image for emotional state analysis.

A plurality of images captured in real time by a camera may be used for the above-described learner's face image obtained as a pre-step of analyzing the emotional state. In this case, one representative image from among the plurality of face images may be extracted and used as an image for analyzing the emotional state, and all of the plurality of face images may be used as the image for analyzing the emotional state. This is because even if all of the plurality of face images are used as images for analyzing the emotional state, it is okay to select the most predominantly derived emotion according to the application of a predetermined facial recognition technique as the learner's emotion.

In addition, in order to ensure the accuracy of the facial recognition results in the future, the learner application program, in the process of acquiring the learner's face image, allows the learner's face to be displayed in a predetermined guide window (Fig. 4) Reference) can be guided to be located within.

As described above, when the learner's face image is obtained, the learner application program transmits the corresponding face image to the operation server 300, and the operation server 300 transmits the received face image to the emotion recognition engine accordingly It enables the recognition engine to perform face recognition on the corresponding face image (refer to FIG. 3 ). However, depending on the system implementation method, unlike in the process of FIG. 3, the learner application program directly transmits the learner's face image to the emotion recognition engine, so that the face recognition on the corresponding face image through the emotion recognition engine may be executed. is of course Thereafter, information on the emotional state of the learner according to the facial recognition result executed by the emotional recognition engine is transmitted back to the operation server 300 .

In FIGS. 2 and 3 , the operation server 300 and the emotion recognition engine are expressed and illustrated as independent subjects, but the function of the emotion recognition engine may be integrated and implemented in the operation server 300 . In addition, if the computational processing burden according to the function of the emotion recognition engine can be covered by the APP unit, the function of the emotion recognition engine may be integrated and implemented in the above-described student application program. As another example, some of the functions of the emotion recognition engine (e.g., extracting feature point data for facial recognition from the corresponding face image) are directly in charge of the above-described student application, and the remaining functions (e.g., feature point data for face recognition) Deriving the learner's emotional state using The present invention includes all of the various embodiments described above, but for convenience and concentration of description, the present invention will be described hereinafter only with reference to the embodiment of FIG. 3 .

In the present invention, as a technique for deriving a related emotion through facial recognition of a face image, a known technique may be used as it is or a technique partially modified based on the known technique may be used.

For example, according to the well-known technology, Korean Patent No. 10-1823611 (patent holder: Emotional Science Research Center / Patent name: AU-based emotional expression information extraction method), a total of 7 Distinguish between fear, surprise, disgust, sad, anger, happy, and contempt as emotional expressions (emotional states) of branches. is becoming

As another example, in Korea Patent Application Laid-Open No. 2019-0123371 (Applicant: Triss Co., Ltd. / Invention Name: Emotion recognition method and artificial intelligence learning method based on facial image), which is a known technology, by analyzing and tracking the expression of the facial image, the facial image As the final emotional state of , a technology for classifying a total of six emotions (anger, disgust, fear, happiness, sadness, surprise) in contrast to expressionlessness is being disclosed.

However, the core of the present invention is not in the facial recognition technique itself or in the technique itself to classify/extract emotions through facial recognition, but in the applicant's own learner care and prescription methodology, which will be described later. That is, as the final result of facial recognition, the classified (classified) emotional information is only used as a predetermined reference element in the applicant's own learner care and prescription methodology, and any commercial or well-known technique (similar or partially modified) It will be said that regardless of whether the technique is applied to the emotion recognition engine described above).

Instead, in the present invention, the operation server 300 classifies the finally derived (classified) emotion through the emotion recognition engine into any one of a positive emotion and a negative emotion.

In an embodiment of the present invention, the positive emotions may include happiness, surprise, and expressionlessness (general normal facial expressions), and the negative emotions include sadness, anger (angry), fear (fear), and disgust (disgust). may apply here. However, depending on the system design method, an expressionless expression may be mapped to a negative emotion differently from the above, or it may be treated as not applicable to any of them.

Accordingly, the operation server 300, when it is determined that the emotional state of the learner in the pre-learning stage corresponds to the above-described positive emotion, a message ( Hereinafter, an encouragement message) is processed to be displayed on the learner's learning screen (the screen before the main learning starts) (refer to FIGS. 3 and 4 ).

On the other hand, when it is determined that the emotional state of the learner in the pre-learning stage corresponds to the above-described negative emotions, the operation server 300 is a message having content to evoke or induce such negative emotions to be converted into positive ones (hereinafter , evoking message) can be processed so that the message is displayed on the learner's learning screen (the screen before the main learning starts). Alternatively, as shown in FIGS. 3 and 4 separately, an image for stabilizing or reinforcing the emotion of the learner so that he or she can prepare for learning with a more comfortable emotion by alleviating negative emotions (hereinafter, the video for ventilation, the It is also possible to transmit a video so that the mind reinforcement video) can be played through the screen before the learner starts learning.

In the above, the method of message expression or video transmission has been mainly described as a method for sustaining positive emotions or converting negative emotions into positive emotions. It goes without saying that various audiovisual methods such as facial expression emoticons representing the corresponding emotion may be used. In the present specification, for convenience and concentration of description, they are collectively referred to as an audiovisual alert.

In addition, the encouragement message and the message for evoking as described above, rather than being displayed on the screen as the same message according to the positive and negative emotional states, detailed emotions (ie, happiness, surprise, expressionlessness, sadness, anger, fear, disgust, etc.) ), it is desirable to have a customized message content. It can be designed and implemented based on artificial intelligence chatbot.

In addition, in FIGS. 3 and 4 , only the case where the ventilation image is transmitted only in the case of a negative emotional state, but when it is determined that the corresponding emotional state is excessively continued even in the case of a positive emotional state (for example, surprise Even if it is determined that the emotion of the person is very strong), the system may be designed so that the image for ventilation is transmitted.

[How to care for learners during learning]

Referring to FIGS. 3 and 5 , according to the system of the present invention, the learner's learning state can be maintained by determining the learner's concentration drop or screen deviation during learning.

To this end, the operation server 300, when it is confirmed that the learner's concentration falls below a pre-specified reference value based on the learning state determination result when the face recognition result through the learner's face image is reflected (also When such a non-concentration state lasts for more than a predetermined time (eg, 20 seconds, etc.) can be processed to be provided. At this time, the method of determining whether the learner is in a state of concentration or not can be clearly understood through the description of the “concentration index” among the five state indices to be described later. Detailed description will be omitted.

Similarly to the above, the operation server 300 may provide the above-described audio-visual alert even in a screen deviation case (more precisely, learning deviation due to seat departure). For example, the operation server 300 may process to provide an audiovisual alert as shown in FIG. 5 when the face of the learner is not recognized in the image of the learner, and, along with this, according to the learning content You can handle the learning screen to stop.

In the above, with reference to FIGS. 3 and 5, only the cases in which the students are in an unconcentrated state during learning or have departed from learning have been described. However, the care method in the learning stage according to an embodiment of the present invention is based on the emotion analysis result through facial recognition. Accordingly, a case in which the learner has negative emotions as described above during learning (eg, had positive emotions or switched to positive emotional states before learning, and negative emotions (ex) during learning It goes without saying that the same can be applied to a case in which it is switched back to fear, etc.) and the state continues for a predetermined time or more). In addition, the above-described learning care process may be similarly applied even when it is determined that the drowsy state is drowsy according to the facial recognition analysis result.

In addition, according to the embodiment of the present invention, when the learning care as described above is required in the learning stage, the operation server 300 transmits (alarm) the corresponding fact to the teacher application program, so that the teacher learns the learning process of the learner It may be possible to intervene in or manage it (see FIG. 3 ).

Derivation of status index

In addition, according to an embodiment of the present invention, the operation server 300 may derive a plurality of state indices for determining the learning state of the learner based on the reaction that appears to the learner during learning. In this regard, in the embodiment of FIG. 3 , as the state index, a happiness index, an anxiety index, a tension index, a concentration index, and a immersion index are exemplified.

In order to derive the above state index, the operation server 300 uses the information analyzed from the emotion recognition engine. At this time, the emotional state information described above and heart rate information to be described later are utilized as the analysis information that is the basis for deriving the state index. can be In some cases, in addition to emotional state information and heart rate information, facial movement degree, expression change degree, eye blink degree, eye movement information according to pupil tracking, etc. analyzed based on the face recognition result may be further utilized.

As biometric information, heart rate information may be acquired through an electrocardiogram sensor, a microwave Doppler sensor, a UWB radar sensor, or the like. However, in an embodiment of the present invention, heart rate information extracted (more precisely estimated) through a facial recognition technology based on a learner's face image is used. In this case, as a technique for acquiring a heart rate through facial recognition of a face image, a known technique may be used as it is or a technique partially modified based on the known technique may be used.

For example, Korea Patent No. 10-2176001 (Patent holder: Korea Electronics and Telecommunications Research Institute / Patent name: Biometric information measuring device and measurement method) and Korean Patent Registration No. 10-1787828 (Patent holder: Jeron Healthcare Co., Ltd. / According to the patent title: Heart rate measurement system using skin color filter), a technology for extracting the heart rate using the pixel value of the skin color of the face by applying the facial recognition technology is disclosed.

As another example, in Korea Patent Registration No. 10-2215557, which is a well-known technology (patent holder: Emotional Science Research Center / Patent name: camera-based heart rate measurement method and system using face color and tremor), the color change of facial image and facial tremor A technique for estimating the heart rate based on the disclosed technology.

However, the core of the present invention is not in the facial recognition technique itself or in the technique of extracting (estimating) the heart rate through the facial image, but in the applicant's unique learner care and As it is in the prescription methodology, heart rate information estimated as a final result through facial image analysis is only used as a predetermined reference element in the applicant's own learner care and prescription methodology, and any commercial or known technique (similar to this or including some modified techniques) whether the technique is applied to the above-described emotion recognition engine.

Accordingly, the operation server 300 may calculate each state index based on the following criteria. However, it should be clarified first that the standard for deriving the status index to be described later is an example, and other standards may be introduced. In addition, some of the criteria to be described later for deriving the status index may be commonly applied concurrently to a plurality of status indices. In this case, the learner's status at the same time is expressed as two or more status indices (for example, tension It is also possible that the expression is expressed as a state of concentration while being in a state of concentration).

Among the state indices, the happiness index calculates the degree to which the above-mentioned positive emotional state persists (eg, the frequency or ratio of the positive emotional state appearing during corresponding learning) based on the emotional state analyzed through the emotional recognition engine. It can be calculated by As another example, a method of calculating the happiness index based on only the extent to which an emotion corresponding to joy is maintained among positive emotional states may be applied.

The anxiety index among the state indices is based on the emotional state analyzed through the emotional recognition engine, and calculates the degree to which the aforementioned negative emotional state persists (eg, the frequency or ratio of the negative emotional state appearing during the corresponding learning) It can be calculated by As another example, a case in which facial expression changes corresponding to each of negative emotional states such as sadness, anger, disgust, and fear are irregularly repeated may be determined as an anxious state, and an anxiety index may be calculated according to the frequency or ratio. In this case, the degree of pupil shaking or eye blinking according to the pupil tracking result may also be reflected in calculating the anxiety index.

The tension index among the state indices can be calculated by calculating the degree (frequency or rate, hereinafter the same) that the heart rate rises above the average heart rate of the learner based on the heart rate analyzed through the emotional recognition engine. However, since the heart rate rise can be caused by various stimuli, it is also possible to calculate the tension index by reflecting the emotional state information of the learner together (for example, when the heart rate rises while the emotion corresponding to fear). Of course.

The concentration index among the state indices may be calculated when it is determined that the learner is in an aroused state based on the heart rate analyzed through the emotional recognition engine. In this case, the arousal state may be determined as follows. As an example, when the arousal state and the relaxed state are distinguished based on the heart rate, when the heart rate is low and falls below the cast-specified reference value (reference heart rate for determining the relaxed state) as the relaxed state, if the heart rate is not in the relaxed state (that is, A heart rate exceeding the reference heart rate for determining the relaxed state) or a heart rate greater than or equal to a predetermined reference heart rate for determining the awakening state may be determined as the awakened state. Also at this time, by reflecting the facial expression change according to the facial recognition result, it can be determined that the learner is in a state of concentration only when the change in facial expression is maintained within the pre-specified allowable range and is in the awakened state. .

A case in which the concentration state according to the above-described criteria continues for a predetermined time or more may be determined as the immersion state. Accordingly, the immersion index among the state indices may be calculated according to the continuation degree (frequency, ratio, or duration, etc.) of the concentration state.

As described above, each calculated state index is included in the emotional analysis result report (ie, the Emotional A.I result analysis report) as shown in FIGS. 2, 3 and 5, and information is shared between learners, teachers, and parents in the future. . In addition, the calculated status index is used for analyzing the learner's learning activity and personalized customized learning prescription.

Deduction of sensible difficulty

In addition, according to an embodiment of the present invention, the operation server 300 may derive the sensible difficulty experienced by the learner in the learning stage. To this end, the operation server 300 analyzes the emotional state analysis and state determination results as described above for each step of the learner's learning process (eg, by learning level, by learning lesson, by learning topic, by learning question). By statistically recording it, it can be utilized to calculate the perceived difficulty. In this case, the perceived difficulty may be divided into high (difficult)/medium (normal)/low (easy).

For example, based on the learner's heart rate in a specific learning stage, if it is determined as an awakened state, the perceived difficulty of learning at that time is determined as easy, and if it is determined as a relaxed state, the perceived difficulty of learning at that time is can be identified with difficulty. This is a method of matching the arousal state as a positive learning state concentrating on learning with low perceptual difficulty, and matching the relaxed state with a high perceptual difficulty level as a negative learning state (for example, boredom) not concentrating on learning. will be. Based on these criteria, in the case of an intermediate heart rate that does not correspond to any of the awake state and the relaxed state, it is possible to match the normal sensible difficulty.

As another example, based on the learner's emotional state in a specific learning stage, a case in which a positive emotional state continues is determined as easy to feel, and a case in which a negative emotional state continues is determined as difficult to feel, and in other cases ( That is, if the emotional state does not persist in either positive or negative direction), it is possible to determine the difficulty of feeling as moderate.

As another example, the sensible difficulty may be calculated based on the consideration of the heart rate and the emotional state together. For example, the arousal state and the relaxation state according to the heart rate criterion correspond to the upper and lower ends of the vertical axis, respectively, and the positive emotional state and the negative emotional state according to the emotion criterion are mapped on the quadrant to correspond to the right and left ends of the horizontal axis, respectively. have. Therefore, as an example, when the arousal state and the positive emotional state (quadrant 1 of the four quadrants) are determined to be normal, the case of the relaxed state and the positive emotional state (the fourth quadrant of the four quadrants) or the relaxed state and the negative emotional state In the case of (the third quadrant of the four quadrants), the sensory difficulty level is determined as low, and the arousal state and the negative emotional state (the second quadrant of the four quadrants) is determined as the sensory difficulty difficulty level, etc. may be used.

However, the above-described sensible difficulty calculation criteria are only an example, and may be variously modified by a system designer. Therefore, only the criteria described through the above example may not be applied to the present invention, and various modifications may exist. Of course you can. In addition, along with the above-mentioned criteria, additional information according to the learner's face recognition result (for example, showing a grimacing expression, no expression change, pupil shaking, etc.) may also be used to calculate the perceived difficulty. .

According to an embodiment of the present invention, the operation server 300 calculates the perceived difficulty according to the selected criteria among the above-described criteria, but relates to the sensory difficulty actually felt by the learner when the learner completes the learning in the future. By writing a questionnaire or processing to respond to a question, at this time, it is possible to change or update the perceived difficulty calculation criterion in a way of calculating the matching accuracy between the answered actual perceived difficulty and the calculated perceived difficulty. For this purpose, artificial intelligence deep learning techniques can be applied. In addition, the learning achievement information of the learner may be referred to in the process of changing or updating the calculation criteria. In this way, it is possible not only to increase the accuracy of the calculation of the sensory difficulty, but also to calculate the sensory difficulty that is personalized according to the characteristics and learning pattern of the learner (that is, the derivation of the calculation standard for the sensory difficulty specific to the learner).

The operation server 300 provides a personalized learning prescription to the learner in a way that adjusts or redesigns the learning curriculum, learning level, learning lesson, and learning questions of the learner to the corresponding difficulty by utilizing the sensory difficulty information. can

In addition, the above-described emotional state analysis result, state index analysis result, sensory difficulty analysis result, etc. are transmitted to the teacher, so that the teacher can perform customized coaching for the learner.

[How to prescribe learner after learning]

According to the system of the present invention, after the learning of the learner is completed, the operation server 300 is a learner face image (for example, For example, a facial image related to the most frequent facial expression among a plurality of facial expressions corresponding to the representative emotion) may be displayed on the screen (refer to FIGS. 3 and 5 ).

In this case, the operation server 300 may display the representative emotional image on the screen and an encouragement message suitable for the learning state of the learner. To this end, the operation server 300 compares the emotional state before learning of the learner with the representative emotion during learning, and displays a learning encouraging message corresponding to the change in the emotional state before and after learning on the screen. For example, if the emotional state before learning is a negative emotional state and the representative emotion during learning is happiness, a learning encouraging message such as "The learning today was really good!" If your emotion is fear, you can display a message to encourage learning, such as "Are you under a lot of stress from studying today? You're doing well!"

In addition, the operation server 300, the analysis result including the above-described status index is collected as a report to be provided to learners, teachers, and parents. In this case, as shown in FIG. 5 , the status index may be visualized in the form of a temporal trend change graph, a relative occupancy graph, and the like for five status indices and included in the report.

In addition, the operation server 300 may provide individualized learner analysis and customized learning prescription to the learner by individually or collectively reflecting the above-described emotional state analysis, state index analysis, and sensory difficulty analysis results. Specific examples of learner analysis and customized learning prescription are as follows.

The operation server 300 may extract and analyze information about the learner's preferred activity (ie, a preferred learning area and a non-preferred learning area). As an example, the operation server 300 extracts, as a preferred learning area, a learning area in which the emotional state of the learner appeared as a positive emotion (especially happiness), as a preferred learning area, based on the emotional state analysis result, and a learning area in which a negative emotion appeared. can be extracted as a non-preferred learning area. As another example, the operation server 300 prefers learning 'a learning area with a long concentration time, high immersion, and positive emotions' in the daily learning process, based on the emotional state analysis result and the state index analysis result. It can be extracted as an area, and a 'learning area with a short concentration time, low immersion, and negative emotions expressed' can be extracted as a non-preferred learning area.

In addition, the operation server 300 may diagnose emotions such as academic stress and academic anxiety of the learner based on the tension index and/or anxiety index among the state index analysis results.

In addition, the operation server 300, as described above, based on the sensory difficulty information that the learner feels for each learning item, each learning area, and each learning process, the learner's learning curriculum, learning level, learning lesson, learning question can be tailored or redesigned to suit their level of difficulty, providing a personalized learning regimen for the learner.

In addition, the operation server 300 may provide statistical data on each learner's individual concentration and immersion time by day/week/month/year. In addition, it is possible to provide statistical data on concentration/immersion and emotional state by age group/progress/learning time/learning time.

In addition, the operation server 300 tracks the temporal change of the emotional state of the learner in units such as weekly/monthly based on the representative emotion, so that the strength of the learner's learning activity can be strengthened and the weaknesses can be supplemented. You can categorize the feedback and provide it to the learners, parents, and teachers.

In addition, the operation server 300, based on the learner's emotional state, state index, sensible difficulty, etc., compares the pattern with the comparison target group (eg, the same age group, the same level, or a group of learners of high achievement, etc.) Through this, it is also possible to derive supplementary/reinforcing factors necessary for the learner.

Although the above has been described with reference to the embodiments of the present invention, those of ordinary skill in the art may variously modify the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. and can be changed.

Claims (12)

As a learner analysis and care system using emotion analysis technology,
An application program for learners that is installed on the learner terminal and provides self-directed learning; A teacher application that enables communication with learners in self-directed learning or coaching and management for the learners through web access or app access; It interlocks with the learner and the teacher application program and includes an operation server responsible for service operation for providing self-directed learning,
When the emotional analysis of the learner based on the collected facial recognition data is performed, the operation server monitors and analyzes the learning state of the learner based on the analyzed emotional analysis data, and learning activities for each learner based on the learning state analysis result A learner analysis and care system using emotion analysis technology, characterized in that it provides a personalized learning prescription that can strengthen
According to claim 1,
The sentiment analysis data includes a plurality of predefined emotion data,
The predefined plurality of emotional data includes at least one of emotion data of happiness, surprise, and expressionlessness, and includes at least one of emotion data of sadness, anger, fear, and disgust,
The operation server 300,
Classifying the plurality of predefined emotional data into a positive emotional state and a negative emotional state, classifying the happiness, the surprise, and the expressionlessness as the positive emotional state, and the sadness, the anger, the fear, and the disgust A learner analysis and care system using emotion analysis technology, characterized in that it is classified as a negative emotional state.
3. The method of claim 2,
The operation server 300,
According to the emotional state according to the face recognition result before learning of the learner,
When the emotional state corresponds to a negative emotional state, it is characterized in that processing is performed so that an audiovisual alert including an evoking message or an evoking image for inducing a negative emotion to be converted into a positive emotion can be provided through the learning screen Learner analysis and care system using emotion analysis technology.
4. The method of claim 3,
The operation server 300,
According to the face recognition result during the learner's learning,
When it is confirmed that the learner's concentration falls below a pre-specified reference value, or the learner's face is not recognized, an audiovisual alert to draw the learner's attention is provided through the learning screen, characterized in that , learner analysis and care system using emotion analysis technology.
4. The method of claim 3,
The operation server 300,
Calculating a predefined state index for the learning state of the learner based on the emotional state and the heart rate information obtained according to the result of face recognition during learning of the learner,
The state index, the happiness index, the anxiety index, the tension index, the concentration index, characterized in that it comprises at least one of the immersion index, a learner analysis and care system using emotion analysis technology.
6. The method of claim 5,
The happiness index is calculated by calculating the frequency or rate at which the positive emotional state is maintained during the learner's learning,
The anxiety index is calculated by calculating the frequency or rate at which the negative emotional state persists during the learner's learning, or by calculating the frequency or rate at which facial expression changes corresponding to the negative emotional state are irregularly repeated. A learner analysis and care system using emotion analysis technology, characterized in that.
7. The method of claim 6,
The tension index is calculated by calculating the frequency or rate at which the heart rate rises above the learner's average heart rate during learning of the learner,
The concentration index is calculated by calculating the frequency or rate of having a heart rate in an arousal state in which the corresponding heart rate is higher than or equal to a predefined reference value based on the heart rate during learning of the learner,
The immersion index is, during the learning of the learner, emotional analysis, characterized in that it is calculated by calculating the frequency, ratio or duration of the immersion state based on when the concentration state corresponding to the concentration index continues for a predetermined time or more Learner analytics and care systems using technology.
8. The method of claim 7,
The operation server 300,
Based on the heart rate information, the learner derives the sensible difficulty felt while performing the corresponding learning,
Based on the learner's heart rate, when it is determined that the learner is in an awakened state, the perceived difficulty of learning at the time is determined as easy, and when it is determined that the learner is in a relaxed state, the perceived difficulty of learning at the time is determined as difficult. Learner analysis and care system using emotion analysis technology.
8. The method of claim 7,
The operation server 300,
Based on the emotional state and the heart rate information, a sensible difficulty felt by the learner while performing the corresponding learning is derived,
The arousal state and the positive emotional state are determined to be normal, and the relaxed state and the positive emotional state or the relaxed state and the negative emotional state are determined to be low in sensibility difficulty, , A learner analysis and care system using emotion analysis technology, characterized in that the case of the arousal state and the negative emotional state is determined as the difficulty of feeling difficulty.
10. The method according to claim 8 or 9,
The operation server 300,
After the learner's learning is completed, information about the learner's preferred activity is extracted,
Based on the analysis result of the emotional state, the learning region that appeared in the positive emotional state is extracted as a preferred learning region, and the learning region that appears in the negative emotional state is extracted as a non-preferred learning region, or,
Based on the emotional state analysis result and the state index analysis result, a learning region in which a long concentration time, high immersion, and positive emotions are expressed is extracted as a preferred learning region, and a short concentration time and low immersion , A learner analysis and care system using emotion analysis technology, characterized in that the learning area in which negative emotions are expressed is extracted as a non-preferred learning area.
11. The method of claim 10,
The operation server 300,
For each learning item, each learning area, and each learning process, based on the sensory difficulty information felt by the learner, the learning curriculum, learning level, learning lesson, and learning question of the learner are adjusted to the corresponding difficulty, characterized in that, Learner analysis and care system using emotion analysis technology.
12. The method of claim 11,
The operation server 300,
Data on individual learner's concentration and immersion time are provided by statistical analysis by day/week/month/year, or data on concentration/immersion and emotional state by age group/progress/learning time are also provided as statistics. A learner analysis and care system using emotion analysis technology, characterized in that.

Images