CN115966003A - System for evaluating online learning efficiency of learner based on emotion recognition - Google Patents

Abstract

The invention discloses an emotion recognition-based learner online learning efficiency evaluation system, which comprises a real-time camera monitoring module, an image collection module, an image processing module and a statistical analysis module, wherein the real-time camera monitoring module is used for acquiring key data; the image processing module is used for analyzing and processing data; and the statistical analysis module is used for carrying out statistical analysis on the constructed model data. Compared with the prior art, the invention has the advantages that: the learning efficiency of the learner is comprehensively obtained through scientific weight calculation by combining the facial expression characteristics, the head posture and the hand skeleton behaviors through the high-definition camera carried by the learner in the online teaching process, so that the error rate of emotion recognition only from non-physiological signals is effectively reduced, and the fault tolerance rate is improved.

Description

An online learning efficiency evaluation system for learners based on emotion recognition

technical field

The invention relates to the field of education, in particular to an emotion recognition-based online learning efficiency evaluation system for learners.

Background technique

Emotion recognition originally refers to the individual's recognition of other people's emotions. Now it mostly refers to the automatic identification of the individual's emotional state by AI by obtaining the individual's physiological or non-physiological signals. It is an important part of emotional computing. It integrates human feelings, A state of thought and behavior that plays an important role in human-to-human communication. Common emotion recognition methods are mainly divided into two categories: recognition based on non-physiological signals and recognition based on physiological signals. And the application based on emotion recognition is becoming more and more extensive. For example, in the medical field, emotion recognition can provide a basis for the diagnosis and treatment of mental illness; in terms of user experience, when researching products, obtaining the user's emotional changes is conducive to discovering products. Existing problems; In the field of transportation, timely detection of the emotional state of those staff who need a high concentration of attention to operate is an effective means to avoid accidents.

At present, technology based on emotion recognition is also gradually applied in the field of education. As the basis of non-physiological signal recognition - in addition to being used to detect cheating in exams and correct irregular class behaviors, monitoring technology has also been combined in deeper aspects. For example, the cooperation between Canzao.com and Microsoft Asia Research Institute can Conduct long-term and comprehensive analysis of learners to evaluate teaching efficiency. However, there are still many shortcomings. The basic reliability of emotion recognition based on non-physiological signals is low. In the above cooperation, only the judgment of facial expressions is used to determine the learning efficiency of learners, which has poor reliability and low accuracy. This solution mainly proposes a method based on non-physiological emotion recognition, which is an evaluation system for judging learners' online learning efficiency through head posture detection, emotion recognition and bone detection, which greatly improves the detection accuracy.

Contents of the invention

The purpose of the present invention is to provide a learner's online learning efficiency evaluation system based on emotion recognition.

In order to solve the above technical problems, the technical solution provided by the present invention is: an online learning efficiency evaluation system for learners based on emotion recognition, including a real-time camera monitoring module based on the system architecture, an image collection module, an image processing module, and a statistical analysis module , the real-time camera monitoring module is used for the acquisition of key data; the image processing module is used for data analysis and processing; the statistical analysis module is used for statistical analysis of the constructed model data;

The working steps of an online learning efficiency evaluation system for learners based on emotion recognition are as follows:

1) Face detection based on video images. Given a video image, it is required to locate and detect a person's head posture, face position, neck, arm, and five-finger flexion and extension and posture, that is, three parts, image segmentation , detection object locking, feature extraction;

2) The analysis and judgment of facial expressions, based on the histogram of oriented gradients (HOG) of texture features, performs feature extraction for image dimensionality reduction, improves the efficiency of model extraction features, improves performance from the input level, and combines linear interpolation for weighted operations, fully Express image details and overall features, and add CBAM to the network;

3) Feature definition. The emotional states of the learners are mainly divided into three categories through the expression features, which are "happy", "neutral" and "boring". These three states are judged by combining facial expressions with twist angles and head postures .

Compared with the prior art, the present invention has the advantages that: this solution reads the head features of the learners in real time through the high-definition camera that comes with it during the online teaching process, and organically integrates the facial expression features, head posture and hand bone behavior Through the combination of scientific weight calculation, the three-point integration can comprehensively obtain the learning efficiency of learners, so as to effectively reduce the error rate of emotion recognition only from non-physiological signals and improve the error tolerance rate.

Furthermore, the real-time camera monitoring module realizes the real-time monitoring of the learning status of the students in front of the screen through the control of the notebook and the computer camera during the online class, and transmits the acquired video signal to the computer main control operation center in real time.

Further, the real-time camera monitoring module includes face detection function, facial expression recognition function, head posture analysis function, and body behavior judgment function.

Furthermore, the image collection module automatically collects the required audio and video by adjusting the reading frequency of the device and the number of digitization times within a certain period of time, as data materials, and provides a data basis for subsequent module operations.

Further, the image processing module realizes the face detection of the image target, extracts the facial feature area, divides it into 468 key points, and tracks the learner's arm, five fingers and other body behaviors in real time according to the model, providing reference data for the next module.

Further, the statistical analysis module uses the data support provided by the previous module to input the model for comparison to obtain the degree of attention and participation of students to the course content, and when the degree of participation is low at a certain moment, the time record and Real-time reminders.

Description of drawings

FIG. 1 is a schematic diagram of the architecture of the present invention.

Fig. 2 is a block diagram of facial feature point extraction of the present invention.

Fig. 3(a) is a schematic diagram of a close-up image of an individual student according to the present invention.

FIG. 3( b ) is a schematic diagram of the detection results of facial feature points in side partial images according to the present invention.

Fig. 3(c) is a schematic diagram of the detection of some limb organs in the present invention.

Fig. 4 is a frame diagram of the preliminary judgment based on expression of the present invention.

Fig. 5 is a schematic diagram of analysis of students' psychology and facial features common in online teaching of the present invention.

Fig. 6 is a schematic diagram of data for facial expression feature recognition in the embodiment.

Fig. 7(a) is a schematic diagram of judging a facial expression in the embodiment.

Fig. 7(b) is a schematic diagram of judging another facial expression in the embodiment.

Fig. 8 is a schematic diagram of real-time tracking of expression features in the embodiment.

Fig. 9(a) is a schematic diagram of the expression characteristics of the frontal face observed by the system under the state of the frontal face and the head posture rotation angle α in the embodiment.

Fig. 9(b) is a schematic diagram of the expression characteristics of the side face observed by the system under the state of the head and the head posture rotation angle α in the embodiment.

Fig. 9(c) is a schematic diagram of the expression characteristics of the frontal face observed systematically under the state of elevation angle β between the frontal face and the head posture in the embodiment.

Fig. 9(d) is a schematic diagram of the expression features of the upward face observed under the state of elevation angle β between the frontal face and the head posture in the embodiment.

Fig. 10 is a schematic diagram of system functions of the present invention.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings.

When the present invention is actually implemented, in the embodiment shown in Figure 1, a kind of student's classroom participation evaluation system based on expression recognition and gesture behavior is proposed, and the internal board mainly includes video recording board, data acquisition board, head posture, The expression and body behavior recognition section and the experimental analysis and evaluation section, specifically: 1) Implement the camera monitoring section, through the control of the notebook and computer camera during the online class, to ensure real-time monitoring of the students' learning status in front of the screen during the online class monitoring. And the acquired video signal is sent to the computer main control operation center in real time, and the multi-directional gesture processing method is initially adopted, not only to obtain facial expression features, but also to obtain body posture and five-finger features. However, intelligent analysis and judgment can not only recognize facial expression feature gestures, but also combine body postures for recognition.

2) Image collection plate. By adjusting the reading frequency of the device and the number of data conversions within a certain period of time, the required audio and video are automatically collected as data materials to provide a data basis for subsequent module operations.

3) Image processing module. Realize the face detection of the image target, extract the facial feature area, divide it into 468 key points, and track the learner's arm, five fingers and other body behaviors in real time according to the model, and provide reference data for the next module.

4) Statistical analysis module. Through the data support provided by the previous module, the input model is compared to obtain the degree of attention and participation of students to the course content, and when the degree of participation is low at a certain moment, time records and real-time reminders are carried out.

In one embodiment of the present invention, as shown in Fig. 2, Fig. 3 (a), Fig. 3 (b), Fig. 3 (c), Fig. 4 and Fig. 5, the specific working steps of the system are as follows:

(1) Face detection method based on video image

In the current online teaching process, learners will show different postures, expressions and behaviors, and affected by online teaching, each learner has a different class environment, so the scenes in the surveillance video images are more complex and changeable. But it is precisely because of online teaching that the distance between the learner and the learning screen will not exceed a certain range, and the built-in camera can usually complete relatively clear continuous shooting of the target, so it can effectively capture the learner's personality. face information.

Given a video image, it is required to locate and detect a person's head posture, face position, neck, arm, and five-finger flexion and extension and posture, that is, three parts, image segmentation, detection object locking, and feature extraction. Due to the diversity of human faces and the complexity of the environment, it is very challenging to locate and track facial and body feature points. At present, model-based positioning is the mainstream positioning method. XGBoost (eXtreme Gradient Boosting) is an algorithm or engineering implementation based on GBDT. and other fields have been widely used. Based on the introduction of the XGBoost model, this paper combines the mechanism of RESnet to realize the extraction of facial features. Based on the existing XGBoost method, combined with the existing model and data, the multi-pose facial features are extracted in the online class.

As shown in Figure 3(a), for the close-up image of an individual student, this method can effectively identify the corner of the head and the feature area of the face, and the occlusion of the glasses, hands, and even the mask is carried out through an effective noise reduction algorithm Optimized to reflect its stability and robustness. Figure 3(b) shows the detection results of face feature points for partial side images, except for individual face deflection or excessive occlusion, the feature points cannot be accurately identified. Figure 3(c) is the detection of some limbs and organs, the characteristics of the hand during the locking period, and the detailed behavior is observed.

(2) Analyzing and judging methods of facial expressions

Facial expression is the external manifestation and expression of emotions, and it is also an objective indicator for the study of human beings. The facial emotion recording technology of the famous American psychologist Paul Ekman can measure happiness, surprise, disgust, anger, fear, sadness6 emotions (Meng Zhaolan, 1987). There is no doubt that these six expressions are not suitable for education, but this article extracts their characteristics from them, so that they can be better applied to the field of education. In the normal class experience, the student's expression conveys a lot of emotional information to the teacher, which reflects the student's understanding of the course, and expresses emotional characteristics such as understanding and confusion. The facial expression shown by the student on the basis of understanding is relaxed and happy. The state of the head is fixed on the teacher and moves with the rhythm of the teacher. This means that he is very interested in the current learning content and has a high degree of participation. And when students show boredom or even resistance to the content, and the behavioral characteristics also show that they lower their heads for a long time, look left and right, or frown, or even lie down to sleep, most of them have low participation or learners cannot understand What the teacher said.

Expression is a natural expression based on physiology. It starts from human instinct and is not controlled by the brain, so it cannot be concealed, let alone disguised. Emoticons have many applications, such as the application in criminal investigation in the judicial field, to judge whether a criminal suspect is lying; it can be used in the field of prevention and treatment of mental illness, to help treat patients; it can also be used in the field of education, Help teachers better understand the emotional state of learners, so as to make up emotionally, so that students can learn better and improve learning efficiency.

The Histogram of Oriented Gradients (HOG) based on texture features extracts features for image dimensionality reduction, improves the efficiency of model extraction features, improves performance from the input level, and combines linear interpolation for weighted operations to fully express image details and overall features. Add CBAM (attention module) to the network to improve accuracy. Convolutional Block Attention Module (CBAM), proposed by SanghyunWoo et al. in 2018, is a simple and effective attention module for feedforward convolutional neural networks. Given an intermediate feature map, the CBAM module sequentially infers the attention map from two different dimensions of channel and space, and then multiplies the attention map with the input feature map to perform adaptive feature optimization. In view of the different characteristics of different expressions, we innovatively proposed different attention learning algorithm mechanisms for different expressions, using space transformation network focusing, so that the network can focus on important parts of the face, which not only reduces the amount of parameters , to alleviate the impact of gradient explosion, and to increase the efficiency and accuracy of learning, which can be better applied to smart education.

Given an image or video of a face, not all parts of the face are important in detecting a particular emotion, and in many cases we only need to focus on specific regions to perceive the underlying emotion. Based on this observation and inspired by the idea of "teaching students in accordance with their aptitude", we independently propose to add an attention mechanism, adopt different attention learning algorithms for different expressions, and convert them into our framework through a spatial transformation network to focus on important facial regions. .

In the online teaching environment, we have introduced posture recognition and facial expression recognition technology, together with head and body posture detection, to analyze the objective state of expressions and the psychological state of learners under subjective behavior. According to computer intelligent analysis, Participation, and "Alarm" can be carried out when the participation is low for a long time. Secondly, it can also facilitate teachers to grasp the teaching situation.

(3) Feature definition

We mainly divide learners' emotional states into three categories through expression features, which are happy, neutral, and bored. According to the Izzard facial coding system, these four states are judged by combining facial expressions with twist angles and head postures, as shown in Figure 5.

Furthermore, in one embodiment of the present invention, this embodiment combines the learner's head posture and facial expression behavior, and takes advantage of the high-performance computing of the computer to effectively follow up on online education, making online education The efficiency of online learning is improved, and it also enables teachers to understand the learning status of students, so as to supervise and make teaching compensation for weak links. By analyzing students' facial expressions and head posture angles, recording students' status, helping students adjust their learning styles and learning methods in a timely manner, formulating targeted programs, and realizing the continuous development of students' abilities and individuality. At the same time, if we can accurately grasp the participation of students in the classroom during teaching and communication, we can effectively improve management efficiency, understand the true psychological conditions of students, identify whether students are lying in their usual performance, and adjust management ideas in a timely manner And working method, establish the sense of trust between teachers and students, and at the same time make adequate preparations for possible emergencies.

The operating procedures of the system specifically include:

(1) Expression feature recognition

The deeper the network structure, the stronger its fitting ability should be. However, when the depth of the network reaches a certain level, the performance of the network will not increase but decrease. Conventional neural networks simply connect convolutional layers, fully-connected layers and other structures together in a certain order, and each layer structure only accepts information from the previous layer, and passes it to the next layer after processing in this layer. When the network level is deepened, this single connection method will lead to the degradation of neural network performance. This deficiency can be improved when Residual residual learning is introduced. The so-called residual learning is to add "shortcut connections" on the basis of the above-mentioned single connection method. The most important thing in ResNet is the residual learning unit, and the residual network is to add skipconnections (skip connection layer) to the network.

At the same time, due to every backpropagation process in the network, all weights will be updated at the same time, resulting in a "lack of tacit understanding" between layers, and the more layers, the more difficult it is to cooperate with each other. It is also necessary to have a certain understanding of the distribution of the input of each layer. controlled by. We applied BatchNormalization (batch normalization). BatchNormalization has some kind of regularization effect. After using this method, there is no need to use the dropout method, and it also reduces overfitting.

As the basic classification of intelligent analysis, the judgment of facial expressions is particularly important. After comparing experiments with many models such as ResNet, GoogLeNet, AlexNet, VGG, etc., we selected the ResNet model with the best effect, and compared the accuracy of the experiment Boosted to the highest, our experimental accuracy on the training set is as high as 97.59%, as shown in Figure 6. Let the output function be H(x), F(x) be the residual function, and x be the input:

H(x)=F(x)+x

By training the model with deep learning, judgments about facial expressions can be obtained, as shown in Figure 7(a) and Figure 7(b). And it can be displayed in real time over time, and the real-time tracking of expression features is shown in Figure 8:

(2) Judgment of head posture

The angle α and β in two directions are introduced to track the state of the head pose. When the student is happy or the initial participation coefficient is α(t)=-1, β(t)=-1, t is the time of real-time change, when the angle α changes with time |α(t)| When the degree of β changing with time |β(t)| is not more than 30°, and the degree |β(t)| of β does not exceed 15°, the attention degree coefficient a(t)=1, β(t)=1, otherwise the attention degree coefficient is 0. For the judgment based on the degree of participation in the online classroom, please refer to the calculation formula (1):

When the student is in a negative state attribute, or the face detection fails, or the angle deflection is greater than 30 degrees, it can be basically defined as not participating in the classroom behavior. When it lasts for more than a standard time, the system will remind. For example, the calculation formula T of classroom participation time is shown in formula (2):

Among them, N is the total number of times learners are tested in online classes, and n is the number of times that the test reports are actively participating in the class, and T _o is the total class time. Therefore, based on this formula, the class participation time can be obtained, and thus the side performance Students are interested in the classroom and learning efficiency, helping teachers to coordinate and plan.

In order to effectively observe the expression features of the human face, the head posture rotation angle α is aimed at the facial feature points, as shown in Figure 9(a) and Figure 9(b), the angle between the frontal face and the twist is the head posture angle α. As shown in Figure 9(c) and Figure 9(d), the angle between the frontal view and the upward view is the head attitude elevation angle β, and the parameters such as the head attitude rotation angle α and the head attitude elevation angle β are carefully analyzed and compared. judge.

The above shows and describes the basic principles and main features of the present invention and the advantages of the invention. Those skilled in the art should understand that the present invention is not limited by the above-mentioned embodiments. What are described in the above-mentioned embodiments and description are only to illustrate the principles of the present invention , On the premise of not departing from the spirit and scope of the present invention, there will be various changes and improvements in the present invention, and these changes and improvements all fall within the scope of the claimed invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims (6)

1. The utility model provides a learner online learning efficiency evaluation system based on emotion recognition, includes real-time camera monitoring module, image collection module, image processing module, statistical analysis module based on system architecture, its characterized in that: the real-time camera monitoring module is used for acquiring key data; the image processing module is used for analyzing and processing data; the statistical analysis module is used for carrying out statistical analysis on the constructed model data;

the working steps of the learner online learning efficiency evaluation system based on emotion recognition are as follows:

1) Based on the face detection of a video image, a video image is given, the head posture, the face position, the neck, the arms and the flexion and extension postures of a person, namely three parts, namely image segmentation, detection object locking and feature extraction are required to be positioned and detected;

2) Analyzing and judging facial expressions, performing feature extraction on image dimensionality reduction based on a direction gradient Histogram (HOG) of textural features, improving the efficiency of model extraction features, improving the performance from an input level, performing weighting operation by combining linear interpolation, fully expressing image details and overall features, and adding CBAM into a network;

3) The feature definition divides the emotional state of the learner into three categories, namely 'happy', 'neutral' and 'boring', through the expression feature, and the three states are judged through combining the facial expression with the torsion angle and the head posture.

2. The system of claim 1, wherein the system comprises: the real-time camera monitoring module realizes real-time monitoring of the learning state of the students in front of the screen through controlling the notebook computer and the computer camera in the course of the network course, and transmits the acquired video signals to the computer main control operation center in real time.

3. The system of claim 2, wherein the system comprises: the real-time camera monitoring module comprises a face detection function, an expression recognition function, a head posture analysis function and a body behavior judgment function.

4. The system of claim 1, wherein the system comprises: the image collection module automatically collects required audio and video as data materials by adjusting the reading frequency of the equipment and the number of times of datamation in a certain time, and provides a data base for the operation of subsequent modules.

5. The system of claim 1, wherein the system comprises: the image processing module realizes the face detection of an image target, extracts a face characteristic region, divides the face characteristic region into 468 key points, tracks the body behaviors of the arm, the five fingers and the like of a learner in real time according to the model and provides reference data for the next module.

6. The system of claim 1, wherein the system comprises: the statistical analysis module is used for inputting a model for comparison through data support provided by the previous module to obtain the attention and participation of students to the course content, and recording the time and reminding the students in real time when the participation at a certain moment is low.

Images