CN114973126A - Real-time visual analysis method for student participation degree of online course - Google Patents
Real-time visual analysis method for student participation degree of online course Download PDFInfo
- Publication number
- CN114973126A CN114973126A CN202210540837.XA CN202210540837A CN114973126A CN 114973126 A CN114973126 A CN 114973126A CN 202210540837 A CN202210540837 A CN 202210540837A CN 114973126 A CN114973126 A CN 114973126A
- Authority
- CN
- China
- Prior art keywords
- student
- students
- screen
- value
- head
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 57
- 230000000007 visual effect Effects 0.000 title claims abstract description 31
- 238000007781 pre-processing Methods 0.000 claims abstract description 69
- 230000008451 emotion Effects 0.000 claims abstract description 60
- 238000000034 method Methods 0.000 claims abstract description 38
- 230000006998 cognitive state Effects 0.000 claims abstract description 9
- 230000001815 facial effect Effects 0.000 claims abstract description 8
- 210000003128 head Anatomy 0.000 claims description 107
- 210000001508 eye Anatomy 0.000 claims description 104
- 210000000214 mouth Anatomy 0.000 claims description 46
- 230000036544 posture Effects 0.000 claims description 39
- 239000011159 matrix material Substances 0.000 claims description 30
- 230000006399 behavior Effects 0.000 claims description 25
- 238000012544 monitoring process Methods 0.000 claims description 22
- 238000012795 verification Methods 0.000 claims description 22
- 230000004397 blinking Effects 0.000 claims description 15
- 206010048232 Yawning Diseases 0.000 claims description 13
- 241001282135 Poromitra oscitans Species 0.000 claims description 12
- 238000007726 management method Methods 0.000 claims description 12
- 230000009471 action Effects 0.000 claims description 11
- 230000002996 emotional effect Effects 0.000 claims description 11
- 230000002159 abnormal effect Effects 0.000 claims description 9
- 238000013136 deep learning model Methods 0.000 claims description 9
- 230000008909 emotion recognition Effects 0.000 claims description 9
- 238000003062 neural network model Methods 0.000 claims description 9
- 238000013528 artificial neural network Methods 0.000 claims description 6
- 210000005252 bulbus oculi Anatomy 0.000 claims description 6
- 210000004209 hair Anatomy 0.000 claims description 6
- 230000001149 cognitive effect Effects 0.000 claims description 5
- 238000001514 detection method Methods 0.000 claims description 5
- 230000004886 head movement Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 238000013461 design Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 4
- 210000000744 eyelid Anatomy 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 241000167926 Entoprocta Species 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 3
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000013145 classification model Methods 0.000 claims description 3
- 239000003086 colorant Substances 0.000 claims description 3
- 238000013527 convolutional neural network Methods 0.000 claims description 3
- 230000004399 eye closure Effects 0.000 claims description 3
- 230000008921 facial expression Effects 0.000 claims description 3
- 210000001331 nose Anatomy 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 239000000725 suspension Substances 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 230000035559 beat frequency Effects 0.000 claims 1
- 238000012800 visualization Methods 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 2
- 206010063659 Aversion Diseases 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000036651 mood Effects 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000010223 real-time analysis Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/166—Detection; Localisation; Normalisation using acquisition arrangements
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
- G06V40/19—Sensors therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Business, Economics & Management (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Tourism & Hospitality (AREA)
- Medical Informatics (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Ophthalmology & Optometry (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Computing Systems (AREA)
- Educational Administration (AREA)
- Educational Technology (AREA)
- Databases & Information Systems (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a real-time visual analysis method for student participation of an online course, which comprises the steps of acquiring videos of students in the online course in real time; preprocessing and extracting video data to obtain facial features of students; establishing a student participation analysis model from four aspects of attention, emotion, fatigue degree and cognitive state; and carrying out visualization on the analysis result of the student participation analysis model to complete the real-time visualization analysis of the student participation of the online course. The method comprises the steps of extracting facial features of students by using a computer vision method, then establishing a student participation degree analysis model, and then helping teachers analyze student participation degrees from multiple angles by using a visual analysis method; compared with the traditional method for linearly viewing the student videos, the method provided by the invention can automatically extract the student participation information and perform visual display, and is high in reliability, good in accuracy, high in real-time performance, convenient and fast.
Description
Technical Field
The invention belongs to the field of image data processing, and particularly relates to a real-time visual analysis method for student participation of an online course.
Background
With the development of economic technology and the improvement of living standard of people, people can learn new knowledge and new skills more actively. As a result, more and more online courses have appeared on the network and people are more and more willing to participate in these online courses.
The participation degree is an important evaluation index of students in online courses, and the high degree of the participation degree directly influences the quality of the courses and the learning quality of the students. Therefore, real-time assessment and visualization of student participation in a course has been one of the research focus of online course platforms.
However, currently, various problems often exist in real-time assessment aiming at the participation degree of students in online courses. Some researchers obtain the participation-related characteristics of students through various detection means, such as a heart rate sensor, an eye tracker, a computer vision method and the like, and then calculate the participation of the students through an automatic method. However, the automated method still has difficulty in solving the above-described problems. Firstly, an automatic classification method usually directly gives out student participation classes, but because teachers do not understand logic behind models, reasons behind student participation are difficult to analyze; secondly, as the number of students is large, data is real-time, context information is contained, the data scale is large and complex, and teachers are difficult to analyze in real time; finally, most of current student participation degree analysis work is to analyze offline historical data, and the real-time analysis requirements of online classes are difficult to meet in the aspects of analysis speed and perception efficiency.
Disclosure of Invention
The invention aims to provide a real-time visual analysis method for student participation of an online course, which is high in reliability, accuracy and real-time performance, convenient and quick.
The invention provides a real-time visual analysis method for student participation of an online course, which comprises the following steps:
s1, acquiring videos of students in online courses in real time;
s2, preprocessing the video data acquired in the step S1 so as to extract and obtain facial features of students;
s3, establishing a student participation analysis model from four aspects of attention, emotion, fatigue degree and cognitive state;
and S4, visualizing the analysis result of the student participation degree analysis model established in the step S3, thereby completing the real-time visual analysis of the student participation degree of the online course.
The step S1 of obtaining videos of students in online courses in real time specifically includes the following steps:
collecting real-time videos of students in the online course by using a camera;
setting the frame number of the collected images per second according to the blinking frequency of human eyes;
before the video acquisition begins, images { I) of students looking at the center of the screen and observing four vertexes around the screen at the center of the screen are acquired c ,I lt ,I lb ,I rt ,I rb The head direction and the sight line direction are checked; wherein I c For the student looking forward at the image in the center of the screen, I lt For the student viewing the top left corner of the screen at the center of the screen, I lb For the student viewing the image at the apex of the lower left corner of the screen at the centre of the screen, I rt For the student viewing the top right corner of the screen at the center of the screen, I rb For students viewing the screen from the right lower part of the screen at the center of the screenThe image at the vertex of the corner.
The preprocessing of the video data obtained in step S1 in step S2 to extract and obtain facial features of the student specifically includes the following steps:
preprocessing the client side where the student is located; the preprocessing comprises face recognition preprocessing, face alignment preprocessing, emotion recognition preprocessing, head posture estimation preprocessing and sight line estimation preprocessing;
the face recognition preprocessing is to detect whether face information exists in the acquired image by adopting a deep learning model; the face alignment preprocessing is to adopt a neural network model to extract face coordinate position information in the acquired image; the emotion recognition preprocessing is to recognize real-time emotions of students by adopting a neural network model; the emotions include anger, nature, sadness, surprise, happiness, disgust, and fear; the head posture estimation preprocessing comprises the steps of extracting a head posture angle of a person from an acquired image by adopting a head posture technology, and judging the relation between the head of the student and a screen according to the head posture angle; and the sight estimation preprocessing comprises the steps of extracting the sight direction of the student by adopting a full-face-based sight estimation algorithm and judging the relation between the sight of the student and a screen according to the sight direction.
The face recognition preprocessing specifically comprises the following steps:
detecting whether the image contains face information or not by adopting a multitask cascade convolution neural network deep learning model; detecting the position (x, y, w, h) of a face matrix in an image by using a multitask cascade convolution neural network deep learning model, wherein the position (x, y) represents the coordinate of the upper left corner of the face matrix when the upper left corner of the image is taken as a coordinate origin, w represents the width of a rectangular area corresponding to the detected face matrix, and h is the height of the rectangular area corresponding to the detected face matrix; in specific implementation, if the face matrix is obtained through detection, the face information is identified.
The face alignment preprocessing specifically comprises the following steps:
extracting the coordinates of key points of the human face by adopting a cascade convolution neural network model; in extracting coordinates, in the form of an imageThe upper left corner is the origin, and the extracted coordinates of the key points of the human face are [ x ] i ,y i ]。
The emotion recognition preprocessing specifically comprises the following steps:
adopting ResNet-50 as a convolutional neural network model, and adopting FER2013 facial expression recognition public data set for training to obtain a final preprocessing model; the emotions output by the preprocessing model include anger, nature, sadness, surprise, joy, disgust, and fear.
The head pose estimation preprocessing specifically comprises the following steps:
acquiring the head posture of a person from the image by adopting a head posture estimation algorithm; the head posture is represented by three Euler angles, namely a pitch angle, a yaw angle and a roll angle, and is sequentially used for representing nodding, shaking and turning;
extracting the obtained head posture by adopting a method of projecting 2D to 3D based on key points as a head posture estimation algorithm; the head posture is represented by a three-dimensional vector [ Pitch, Yaw and Roll ], wherein Pitch is a Pitch angle, Yaw is a Yaw angle, and Roll is a Roll angle;
before the preprocessing of head posture estimation, carrying out pre-verification; during verification, extracting the head attitude angles of the student looking at the center of the screen and the head attitude angles of four directions of the student observing the screen around the head of the student at the center of the screen; during verification, proportional conversion processing is carried out on the head postures of the students, so that the Pitch value and the Yaw value when the students watch the center of the screen are both 0, and the absolute values of the Pitch angle of the heads and the Yaw angle when the students watch the upper edge and the lower edge of the screen and the Yaw angle when the students watch the left edge and the right edge of the screen are 0.5;
during verification, the Pitch angle Pitch after verification is calculated by adopting the following formula 2 Is composed of
In the formula Pitch 1 To detect the resulting Pitch angle, Pitch c Pitch angle when the student is looking straight at the center of the screen, Pitch t The pitch angle when the student watches the upper border of the screen;
after checking, if the deflection angle value is greater than 1, directly correcting the deflection angle value to be the maximum value 1;
and finally, when the deflection angle is between-0.5 and 0.5, the head direction of the student is determined to face the screen.
The sight line estimation preprocessing specifically comprises the following steps:
extracting the sight direction of the student by adopting a full-face-based sight estimation algorithm to obtain the sight direction of the student as [ X, Y ], wherein X is the rotation angle of the sight in the horizontal direction, and Y is the rotation angle of the sight in the vertical direction;
before the sight line estimation preprocessing, the preliminary verification is carried out; during verification, line-of-sight angles of the center of the screen and the periphery of the screen are extracted when the head of a student is in the center of the screen; then, according to the check result, converting the line-of-sight data to enable the line-of-sight of the student when watching the center of the screen to be [0,0], and enabling the absolute value of the angle corresponding to the line-of-sight of the student when watching the periphery of the screen to be 0.5;
during verification, the angle value Y of the verified sight line direction is calculated by adopting the following formula 2 Is composed of
In the formula Y 1 For detecting the resulting angle value of the direction of sight, Y c The angle value, Y, of the direction of sight when the student is looking straight at the center of the screen t The angle value of the sight line direction when the student watches the boundary on the screen;
after checking, if the angle value of the sight line direction is greater than 1, directly correcting the angle value to be the maximum value 1;
and finally, when the angle value of the sight line direction is-0.5, determining the watching screen of the student.
Step S3, establishing a student participation analysis model from four aspects of attention, emotion, fatigue degree and cognitive state, specifically comprising the following steps:
attention is paid to the following aspects:
the state variable S of the watching screen of the student is calculated by the following formula and is S | < X | < 0.5 & | < Y | < 0.5'; in the formula, X and Y are the pretreated sight line directions of the students; the value rule of "a" is: if a is true, "a" is 1, if a is false, "a" is 0; and operation; when the state variable S of the student watching screen is 1, the student watching screen is indicated; when the state variable S of the student watching screen is 0, the student does not watch the screen;
receiving the sight direction [ X, Y ] of the student by adopting a sliding window with the time span of 30s]And the student watches the screen state variable S to obtain N ═ X 1 ,X 2 ,...,X n ]、M＝[Y 1 ,Y 2 ,...,Y n ]And W ═ S 1 ,S 2 ,...,S n ]Wherein X is i Is the horizontal gaze direction, Y, of the student in the ith frame i Is the vertical gaze direction of the student in the ith frame, S i State variables of the screen watched by students in the ith frame; calculating to obtain the student stupefied state variable D ═ (max (N) -min (N) ≦ T'&"(max (M) -min (M) ≦ T", where max (N) is the maximum of the elements in N, min (N) is the minimum of the elements in N, D is 1 indicating no student's loss, D is 0 indicating student's loss; finally, the attention variable A of the student is calculated by the following formula
The fatigue degree is as follows:
(1) aiming at students, selecting an average eye length-width ratio B2, a blinking frequency P2, an average blinking time length D2 and a yawning frequency M2 as indexes to model fatigue degrees of the students;
extracting eye key point information and mouth key point information of the human face: wherein the eye key points include 6 key points, eye key point E1 at the outermost eye corner, eye key point E4 at the innermost eye corner, eye key point E2 at the outer side 1/4 of the superior eye contour, eye key point E4 at the inner side 1/4 of the superior eye contour, eye key point E6 at the outer side 1/4 of the inferior eye contour, and eye key point E5 at the inner side 1/4 of the inferior eye contour; the mouth keypoint information comprises 4 keypoints, a leftmost keypoint M1 of the mouth contour, a rightmost keypoint M3 of the mouth contour, a topmost keypoint M2 of the upper lip, and a bottommost keypoint M4 of the lower lip; each key point is a two-dimensional coordinate;
calculated by the following formulaTo the student's eye aspect ratio α is
In the formula, | E2-E6| represents the straight-line distance between the key point E2 and the key point E6; | E3-E5| represents the straight-line distance between the key point E3 and the key point E5; | E1-E4| represents the straight-line distance between the key point E1 and the key point E4;
the length-width ratio beta of the mouth of the student is calculated by the following formula
Wherein | M3-M1| represents the linear distance between the key point M3 and the key point M1, | M2-M4| represents the linear distance between the key point M2 and the key point M4;
the eye length-width ratio and the mouth length-width ratio of each student are obtained in advance and normalized, so that the eye length-width ratio alpha is 0 when the eyes are completely closed, and the eye length-width ratio alpha is 1 when the eyes are completely opened; the mouth aspect ratio β is 0 when the mouth is fully closed and 1 when the mouth is fully open;
when the aspect ratio alpha of the eyes of the student is detected to be smaller than a set threshold value, the student is determined to be closed;
(2) intercepting the video image by adopting a sliding window with the duration of 30s to obtain F ═ B 1 ,B 2 ,...,B n ]In which B is i The eye length-width ratio of the ith frame image in the sliding window is obtained; the average eye length-width ratio B of the students is calculated by the following formula
Defining the blinking frequency P as the number of blinking actions of the students in 30 s;
intercepting the video image by adopting a sliding window with the duration of 30s to obtain G ═ D 1 ,D 2 ,...,D m ]Wherein m blinks are detected in a sliding window, D i The time length of the ith blink; the average blink time length D of the student is calculated by the following formula
When the length-width ratio of the mouth part is larger than a set threshold value and the duration time is larger than the set threshold value, determining that the student has yawning behavior; the frequency M of the yawning of the students is the frequency of the yawning within 30 s;
analyzing the fatigue degree of the students by adopting a dynamic weight fuzzy comprehensive evaluation method: finally, the initial average eye length-width ratio B weight, the blink frequency P weight, the average blink time length D weight and the frequency of the beat-down frequency M weight are sequentially 0.42, 0.13, 0.32 and 0.13; when B is less than 0.2, adjusting the weight of the average eye length-width ratio B, the weight of the blink frequency P, the weight of the average blink time length D and the weight of the frequency of the beat-down frequency M to be 0.8, 0.1 and 0 in sequence; each index corresponds to a corresponding value V at different values; a value of 0 when B is 1, a value of 0.2 when B is 0.8, a value of 0.4 when B is 0.6, a value of 0.6 when B is 0.4, a value of 0.8 when B is 0.2, and a value of 1 when B is 0; the value of P is 0 when P is between 6 and 8, the value of P is 0.2 when P is 5 or 9, the value of P is 4 or 0.4 when P is between 10 and 12, the value of P is 3 or 0.6 when P is between 12 and 14, the value of P is 2 or 0.8 when P is between 15 and 16, and the value of P is 1 when P is less than or equal to 1 or P is more than 16; d has a value of 0 in the range of 0.2 to 0.4, a value of 0.2 in the range of 0.5 to 0.6, a value of 0.4 in the range of 0.7 to 0.8, a value of 0.6 in the range of 0.8 to 1.5, a value of 0.8 in the range of 1.5 to 3, and a value of 1 when D is 3 or more; the value is 0 when M is equal to 0, 0.4 when M is equal to 1, 0.6 when M is equal to 2, 0.8 when M is equal to 3, and 1 when M is greater than or equal to 4; finally, the fatigue degree F is calculated as: f ═ V B *W B +V P *W P +V D *W D +V M *W M Wherein W is B Is the average eye aspect ratio B weight, W P As a weight of blink frequency P, W D Is the average blink time length D weight, W M For weighting the frequency M of the frequency of the down-frequency, V B Is the value of the aspect ratio B of the eye, V P Value of blink frequency P, V D Value of average blink duration D, V M The value of the frequency M is found.
Emotional aspect:
classifying the emotion into anger, nature, sadness, surprise, joy, disgust and fear by adopting an emotion classification model;
cognitive status aspects:
stipulating that the student shakes head when understanding the course content and shakes head when not understanding the course content;
detecting the head movement of the student: extracting the yaw angle and the pitch angle of the head of the student, and then establishing two sliding windows with the time span of 3s to respectively receive the yaw angle and the pitch angle;
the following steps are adopted to detect the nodding behavior of the student,
stipulating that if the head direction continuously descends to exceed a set threshold value, judging that head lowering action occurs; if the head direction continuously rises and exceeds a set threshold value, judging that head-up behavior occurs; and if the head-lowering behavior and the head-raising behavior are detected simultaneously in the observation interval, judging that the head-nodding behavior occurs.
The step S4 of visualizing the analysis result of the student participation analysis model established in the step S3 specifically includes the following steps:
(1) constructing an abstract view, a monitoring view, a distribution view and a personal view for visualizing the analysis result; the abstract view is used for presenting the participation degree of all students in real time; the monitoring view is used for presenting the participation degree of a specified student group in real time; the distribution view is used for presenting the participation degree distribution condition of the students in a specified time period; the personal view is used for displaying the participation degree related characteristics of the individual students;
(2) and (3) abstract view: constructing an abstract attempt based on the stacked histogram for presenting four indexes of emotion, fatigue degree, attention and student state of the student at the same time; presenting the participation degrees of students from three angles, namely attention, fatigue degree and emotion;
overview of student participation from attention and fatigue perspective: the Y axis represents attention intervals of different degrees and is divided into four groups from low to high; the X-axis represents the number of students in each group; each column consists of a plurality of small rectangles, each rectangle represents a student, and the color of each rectangle corresponds to the emotion of the student; the fatigue degree is represented by an embedded dark matrix in each small rectangle; wherein the height of the embedded matrix represents a value of fatigue level; sequencing according to a mode of reducing fatigue degree so as to avoid the problem of visual confusion; displaying the cognitive state and the stubborn state of the student by using circular icons with different colors below each rectangle; by summarizing the views of student participation from an attention perspective and a fatigue perspective, a teacher can quickly summarize the participation of students;
overview of student participation from an emotional perspective: the Y axis represents the emotion types of students, and the length of the rectangular column corresponds to the number of students in a certain emotion; each column is composed of a plurality of rectangles, each rectangle represents a student, two dark rectangles are embedded in the rectangles, the height of the upper rectangle corresponds to the attention of the student, and the height of the lower rectangle corresponds to the fatigue degree of the student; sorting according to attention; the teacher can change different angles according to the teaching scene to overview the participation of students; when the teacher suspends the mouse above the view, the view stops updating in real time, so that the teacher can conveniently check or select students; when the mouse is suspended above a certain student, the detailed information of the student can be presented; the information of students can be checked when clicking; the teacher can select interested student groups from the view and further analyze the student groups;
(3) and (3) monitoring view: participation to help teachers keep their attention on interested students; constructing a human face model; each face represents a student, the color of the hair is used for corresponding to the emotion of the student, and the emotion color is consistent with the abstract view; using the relative position of the eyeball in the eye to correspond to the sight direction of the student; if the student looks at the screen, the eyeball is in the center of the eyes; encoding a head rotation direction using relative positions of hair, eyes, nose and mouth in a human face; the degree of fatigue of the student is encoded using the height of the eyelid; through the monitoring view, the teacher can sense abnormal participation behaviors of students;
(4) distribution view: the system is used for helping teachers understand the participation degree distribution condition of students in a certain time period; the distribution view is a matrix, and each grid in the matrix represents a student; representing each student by a glyph design, wherein the middle pie chart represents the emotional distribution condition of the students in a specified time period; the circle next to the pie chart represents attention, the angle of the dark circle corresponds to the average attention over the time period, and the angle of the top of the light circle corresponds to the upper quartile of attention; the rings on the outermost layer represent fatigue degree, the angle of the dark rings corresponds to the average fatigue degree, and a full circle represents complete fatigue; the angle of the top of the light-colored ring represents the upper quartile of the fatigue degree; the distribution views can be sorted in various ways, and the student positions can be kept fixed and updated in real time after sorting is completed; after the teacher selects part of students from the abstract view, the distribution view can sort the student groups from high to low according to attention by default; a time bar sliding block exists in the distribution view, and a teacher can select a certain time period interval through the sliding block and check corresponding information; the teacher can check the detailed information of the students in a mouse suspension mode and add the students into the monitoring view and the personal view in a clicking mode;
(5) personal view: the system comprises a teacher management system, a student management system and a teacher management system, wherein the teacher management system is used for providing teacher with relevant characteristic information of student participation and original images of students; the time span presented in the personal view is consistent with the time slider in the distribution view; the personal view comprises a sight line direction, a head direction, an eye length-width ratio, a mouth length-width ratio, a student original image and student action information; the X-axis in the personal view represents time; a group of dotted lines are respectively arranged above and below the personal view and used for representing the sight line direction of the students when watching the upper edge and the lower edge of the screen; a flow graph exists in the dotted lines, the width of the flow graph is equal to the width between the two dotted lines, and the position represents the direction of the head; the upper flowsheet shows the head rotating direction around the Y axis, and the lower flowsheet shows the head rotating direction around the X axis; solid lines color coded are used to represent the direction of sight; if the line is between two color lines, the screen is viewed, and if the line is outside the color lines, the line indicates that the student is out of sight of the screen. The color of the line encodes the student's emotion classification; the color of the emotion is consistent with the abstract view; two gantt charts exist in the middle of the two river charts, the first gantt chart represents blink information, the blank area in the gantt chart represents eye closure, and the transparency of the matrix in the gantt chart encodes the average eye aspect ratio between blinks; the second Gantt chart presents mouth opening information, if a rectangle with a specific color appears, mouth opening is represented, and if the rectangle with the specific color does not appear, mouth closing is represented; the actions of the students are represented by color-coded circular icons below the first Gantt chart, the first color represents nodding heads, and the second color represents shaking heads; when a teacher clicks on a flowsheet at a certain moment in the flowsheet, the original image of the student at the moment is displayed to help the teacher verify the own analysis result.
The invention provides a real-time visual analysis method for student participation of an online course, which is characterized in that a computer vision method is used for extracting facial features of students, then a student participation analysis model is established, and then a visual analysis method is used for helping teachers analyze student participation from multiple angles; compared with the traditional method for linearly viewing the student videos, the method provided by the invention can automatically extract the student participation information and perform visual display, and is high in reliability, good in accuracy, high in real-time performance, convenient and fast.
Drawings
FIG. 1 is a schematic process flow diagram of the process of the present invention.
Fig. 2 is a schematic diagram of 68 key coordinate points of a human face in the invention.
Fig. 3 is a schematic diagram of the coordinates of eye and mouth key points of the present invention.
Fig. 4 is a schematic view of the abstract view of the present invention from the perspective of attention.
FIG. 5 is a schematic view of the abstract view of the present invention from an emotional perspective.
Fig. 6 is a schematic view of the monitoring of the present invention.
Figure 7 is a schematic view of the distribution of the present invention,
FIG. 8 is a schematic diagram of the glyph design in the distribution view of the present invention.
Figure 9 is a schematic personal view of the present invention.
Fig. 10 is a schematic view of attention distribution according to a first embodiment of the invention.
Fig. 11 is a schematic diagram of cognitive status distribution of a student according to a first embodiment of the present invention.
Fig. 12 is a schematic distribution view according to a first embodiment of the present invention.
Fig. 13 is a schematic monitoring view according to a second embodiment of the present invention.
Fig. 14 is a personal view diagram of a second embodiment of the present invention.
Fig. 15 is a schematic view of a student 200048 according to the second embodiment of the present invention during class sleeping.
Fig. 16 is a view illustrating the monitoring of the participation of different students according to the third embodiment of the present invention.
Fig. 17 is a schematic view of the distribution and personal views of the third embodiment.
Detailed Description
FIG. 1 is a schematic flow chart of the method of the present invention: the invention provides a real-time visual analysis method for student participation of an online course, which comprises the following steps:
s1, acquiring videos of students in online courses in real time; the method specifically comprises the following steps:
collecting real-time videos of students in the online course by using a camera;
setting the frame number of the collected images per second according to the blinking frequency of human eyes;
before the video acquisition begins, images { I) of students looking at the center of the screen and observing four vertexes around the screen at the center of the screen are acquired c ,I lt ,I lb ,I rt ,I rb The head direction and the sight line direction are checked; in which I c For the student looking forward at the image in the center of the screen, I lt For students viewing the top left corner vertex of the screen at the center of the screen, I lb For the student viewing the image at the apex of the lower left corner of the screen at the center of the screen, I rt For the student viewing the top right corner of the screen at the center of the screen, I rb The image of the top point of the lower right corner of the screen is observed at the center of the screen for students;
s2, preprocessing the video data acquired in the step S1 so as to extract and obtain facial features of students; the method specifically comprises the following steps:
preprocessing the client side where the student is located; the preprocessing comprises face recognition preprocessing, face alignment preprocessing, emotion recognition preprocessing, head posture estimation preprocessing and sight line estimation preprocessing;
the face recognition preprocessing comprises the steps of detecting whether face information exists in an acquired image or not by adopting a deep learning model; the face alignment preprocessing is to adopt a neural network model to extract face coordinate position information in the acquired image; the emotion recognition preprocessing is to recognize real-time emotions of students by adopting a neural network model; the emotions include anger, nature, sadness, surprise, happiness, disgust, and fear; the head posture estimation preprocessing comprises the steps of extracting a head posture angle of a person from an acquired image by adopting a head posture technology, and judging the relation between the head of the student and a screen according to the head posture angle; the sight line estimation preprocessing comprises the steps of extracting sight line directions of students by adopting a full-face-based sight line estimation algorithm, and judging the relation between the sight lines of the students and a screen according to the sight line directions;
in specific implementation, the face recognition preprocessing specifically comprises the following steps:
detecting whether the image contains face information or not by adopting a multitask cascade convolution neural network deep learning model; detecting the position (x, y, w, h) of a face matrix in an image by using a multitask cascade convolution neural network deep learning model, wherein the position (x, y) represents the coordinate of the upper left corner of the face matrix when the upper left corner of the image is taken as a coordinate origin, w represents the width of a rectangular area corresponding to the detected face matrix, and h is the height of the rectangular area corresponding to the detected face matrix; in specific implementation, if a face matrix is obtained through detection, the face information is identified;
the face alignment preprocessing specifically comprises the following steps:
extracting the coordinates of key points of the human face by adopting a cascade convolution neural network model; when coordinates are extracted, the upper left corner of the image is taken as an origin, and the extracted coordinates of the key points of the human face are taken as [ x ] i ,y i ]；
The emotion recognition preprocessing specifically comprises the following steps:
adopting ResNet-50 as a convolutional neural network model, and adopting FER2013 facial expression recognition public data set for training to obtain a final preprocessing model; the emotions output by the preprocessing model include anger, nature, sadness, surprise, joy, disgust, and fear;
the head pose estimation preprocessing specifically comprises the following steps:
acquiring the head posture of a person from the image by adopting a head posture estimation algorithm; the head posture is represented by three Euler angles, namely a pitch angle, a yaw angle and a roll angle, and is sequentially used for representing nodding, shaking and turning;
extracting the obtained head posture by adopting a method of projecting 2D to 3D based on key points as a head posture estimation algorithm; the head posture is represented by a three-dimensional vector [ Pitch, Yaw and Roll ], wherein Pitch is a Pitch angle, Yaw is a Yaw angle, and Roll is a Roll angle;
before the preprocessing of head posture estimation, carrying out pre-verification; during calibration, extracting the head attitude angles of the student looking at the center of the screen and the head attitude angles of the student in four directions when the head of the student observes the screen at the center of the screen; during verification, proportional conversion processing is carried out on the head postures of the students, so that the Pitch value and the Yaw value of the students watching the center of the screen are both 0, and the Pitch angle of the heads of the students watching the upper edge and the lower edge of the screen and the absolute value of the Yaw angle of the students watching the left edge and the right edge of the screen are 0.5;
during verification, the Pitch angle Pitch after verification is calculated by adopting the following formula 2 Is composed of
In the formula Pitch 1 For detecting the resulting Pitch angle, Pitch c Pitch angle when the student is looking straight at the center of the screen, Pitch t The pitch angle when the student watches the upper border of the screen;
after checking, if the deflection angle value is greater than 1, directly correcting the deflection angle value to be the maximum value 1;
finally, when the deflection angle is between-0.5 and 0.5, the head direction of the student is determined to face the screen;
the sight line estimation preprocessing specifically comprises the following steps:
extracting the sight direction of the student by adopting a full-face-based sight estimation algorithm, and obtaining the sight direction of the student as [ X, Y ], wherein X is the rotation angle of the sight in the horizontal direction, and Y is the rotation angle of the sight in the vertical direction;
before the sight line estimation preprocessing, the preliminary verification is carried out; during verification, line-of-sight angles of the center of the screen and the periphery of the screen are extracted when the head of a student is in the center of the screen; then, according to the check result, converting the line-of-sight data to enable the line-of-sight of the student when watching the center of the screen to be [0,0], and enabling the absolute value of the angle corresponding to the line-of-sight of the student when watching the periphery of the screen to be 0.5;
during verification, the angle value Y of the verified sight line direction is calculated by adopting the following formula 2 Is composed of
In the formula Y 1 For detecting the resulting angle value of the direction of sight, Y c The angle value, Y, of the direction of sight when the student is looking straight at the center of the screen t The angle value of the sight line direction when the student watches the boundary on the screen;
after checking, if the angle value of the sight line direction is greater than 1, directly correcting the angle value to be the maximum value 1;
finally, when the angle value of the sight line direction is-0.5, the student is determined to watch the screen;
s3, establishing a student participation analysis model from four aspects of attention, emotion, fatigue degree and cognitive state; the method specifically comprises the following steps:
attention is paid to the following aspects:
the state variable S of the watching screen of the student is calculated by the following formula and is S | < X | < 0.5 & | < Y | < 0.5'; in the formula, X and Y are the pretreated sight line directions of the students; the value rule of "a" is: if a is true, "a" is 1, if a is false, "a" is 0; and operation; when the state variable S of the student watching screen is 1, the student watching screen is indicated; when the state variable S of the screen watched by the student is 0, indicating that the student does not watch the screen;
receiving the sight direction [ X, Y ] of the student by adopting a sliding window with the time span of 30s]And the student watches the screen state variable S to obtain N ═ X 1 ,X 2 ,...,X n ]、M＝[Y 1 ,Y 2 ,...,Y n ]And W ═ S 1 ,S 2 ,...,S n ]Wherein X is i Is the horizontal gaze direction, Y, of the student in the ith frame i Is the vertical gaze direction of the student in the ith frame, S i Viewing a screen state variable for an ith middle school student; calculating to obtain the student stupefied state variable D ═ (max (N) -min (N) ≦ T'&"(max (M) -min (M) ≦ T", where max (N) is the maximum of the elements in N, min (N) is the minimum of the elements in N, D is 1 indicating no student's loss, D is 0 indicating student's loss; finally, the attention variable A of the student is calculated by the following formula
The fatigue degree is as follows:
(1) aiming at students, selecting an average eye length-width ratio B2, a blinking frequency P2, an average blinking time length D2 and a yawning frequency M2 as indexes to model fatigue degrees of the students;
extracting eye key point and mouth key point information of the human face: wherein the eye key points include 6 key points, eye key point E1 at the outermost eye corner, eye key point E4 at the innermost eye corner, eye key point E2 at the outer side 1/4 of the superior eye contour, eye key point E4 at the inner side 1/4 of the superior eye contour, eye key point E6 at the outer side 1/4 of the inferior eye contour, and eye key point E5 at the inner side 1/4 of the inferior eye contour; the mouth keypoint information comprises 4 keypoints, a leftmost keypoint M1 of the mouth contour, a rightmost keypoint M3 of the mouth contour, a topmost keypoint M2 of the upper lip, and a bottommost keypoint M4 of the lower lip; each key point is a two-dimensional coordinate;
the length-width ratio alpha of eyes of the students is calculated by the following formula
In the formula, | E2-E6| represents the straight-line distance between the key point E2 and the key point E6; | E3-E5| represents the straight-line distance between the key point E3 and the key point E5; | E1-E4| represents the straight-line distance between the key point E1 and the key point E4;
the length-width ratio beta of the mouth of the student is calculated by the following formula
Wherein | M3-M1| represents the linear distance between the key point M3 and the key point M1, | M2-M4| represents the linear distance between the key point M2 and the key point M4;
the eye length-width ratio and the mouth length-width ratio of each student are obtained in advance and normalized, so that the eye length-width ratio alpha is 0 when the eyes are completely closed, and the eye length-width ratio alpha is 1 when the eyes are completely opened; the mouth aspect ratio β is 0 when the mouth is fully closed and 1 when the mouth is fully open;
when the aspect ratio alpha of the eyes of the student is detected to be smaller than a set threshold value, the student is determined to be closed;
(2) intercepting the video image by adopting a sliding window with the duration of 30s to obtain F ═ B 1 ,B 2 ,...,B n ]In which B is i The eye length-width ratio of the ith frame image in the sliding window; the average eye length-width ratio B of the students is calculated by the following formula
Defining the blinking frequency P as the number of blinking actions of the students in 30 s;
intercepting the video image by adopting a sliding window with the duration of 30s to obtain G ═ D 1 ,D 2 ,...,D m ]Wherein m blinks are detected in a sliding window, D i The time length of the ith blink; the average blink time length D of the student is calculated by the following formula
When the length-width ratio of the mouth part is larger than a set threshold value and the duration time is larger than the set threshold value, determining that the student has yawning behavior; the frequency M of the yawning of the students is the frequency of the yawning within 30 s;
analyzing the fatigue degree of the student by adopting a dynamic weight fuzzy comprehensive evaluation method: finally, the product is processedObtaining the initial average eye length-width ratio B weight, the blink frequency P weight, the average blink time length D weight and the frequency of the beat-to-fall M weight which are 0.42, 0.13, 0.32 and 0.13 in sequence; when B is less than 0.2, adjusting the weight of the average eye length-width ratio B, the weight of the blink frequency P, the weight of the average blink time length D and the weight of the frequency of the beat-down frequency M to be 0.8, 0.1 and 0 in sequence; each index corresponds to a corresponding value V at different values; a value of 0 when B is 1, a value of 0.2 when B is 0.8, a value of 0.4 when B is 0.6, a value of 0.6 when B is 0.4, a value of 0.8 when B is 0.2, and a value of 1 when B is 0; the value of P is 0 when P is in the range of 6-8, the value of P is 0.2 when P is 5 or 9, the value of P is 0.4 when P is equal to 4 or in the range of 10-12, the value of P is 0.6 when P is equal to 3 or in the range of 12-14, the value of P is 0.8 when P is equal to 2 or in the range of 15-16, and the value of P is 1 when P is less than or equal to 1 or P is more than 16; d has a value of 0 in the range of 0.2 to 0.4, a value of 0.2 in the range of 0.5 to 0.6, a value of 0.4 in the range of 0.7 to 0.8, a value of 0.6 in the range of 0.8 to 1.5, a value of 0.8 in the range of 1.5 to 3, and a value of 1 when D is 3 or more; the value is 0 when M is equal to 0, 0.4 when M is equal to 1, 0.6 when M is equal to 2, 0.8 when M is equal to 3, and 1 when M is greater than or equal to 4; and finally calculating the fatigue degree F as follows: f ═ V B *W B +V P *W P +V D *W D +V M *W M Wherein W is B Is the average eye aspect ratio B weight, W P As a weight of blink frequency P, W D Is the average blink time length D weight, W M For weighting the frequency M of the frequency of the down-frequency, V B Value of ocular aspect ratio B, V P Value of blink frequency P, V D Value of average blink duration D, V M The value of the frequency M of the dozen frequency of the yawns;
emotional aspect:
classifying the emotion into anger, nature, sadness, surprise, joy, disgust and fear by adopting an emotion classification model;
cognitive status aspects:
stipulating that the student shakes head when understanding the course content and shakes head when not understanding the course content;
detecting the head movement of the student: extracting the yaw angle and the pitch angle of the head of the student, and then establishing two sliding windows with the time span of 3s to respectively receive the yaw angle and the pitch angle;
the method comprises the following steps of detecting the nodding behavior and the shaking behavior of students:
defining that if the head direction continuously drops and exceeds a set threshold Tnod, the head-lowering behavior is considered to occur; conversely, if the head direction continuously rises and exceeds the set threshold Tnod, the head-up behavior is considered to occur; in the observation interval, if the head-lowering behavior and the head-raising behavior are detected simultaneously, judging that the head-nodding behavior occurs; through a large number of experimental result statistics, the value of the set threshold Tnod is preferably 0.25;
s4, visualizing the analysis result of the student participation degree analysis model established in the step S3, thereby completing the real-time visual analysis of the student participation degree of the online course; the method specifically comprises the following steps:
(1) constructing an abstract view, a monitoring view, a distribution view and a personal view for visualizing the analysis result; the abstract view is used for presenting the participation degree of all students in real time; the monitoring view is used for presenting the participation degree of a designated student group in real time; the distribution view is used for presenting the participation degree distribution condition of the students in a specified time period; the personal view is used for displaying the participation degree related characteristics of the individual students;
(2) and (3) abstract view: constructing an abstract attempt based on the stacked histogram for presenting four indexes of emotion, fatigue degree, attention and student state of the student at the same time; presenting the participation degrees of students from three angles, namely attention, fatigue degree and emotion;
a view outlining student participation from an attention and fatigue perspective (as shown in fig. 3): the Y axis represents attention intervals of different degrees and is divided into four groups from low to high; the X-axis represents the number of students in each group; each column consists of a plurality of small rectangles, each rectangle represents a student, and the color of each rectangle corresponds to the emotion of the student; the fatigue degree is represented by an embedded dark matrix in each small rectangle; wherein the height of the embedded matrix represents a value of fatigue level; sequencing according to a mode of reducing the fatigue degree so as to avoid the problem of visual confusion; displaying the cognitive state and the stubborn state of the student by using circular icons with different colors below each rectangle; by summarizing the views of student participation from an attention perspective and a fatigue perspective, a teacher can quickly summarize the participation of students;
overview of student participation from an emotional perspective (as shown in FIG. 4): the Y axis represents the emotion types of students, and the length of the rectangular column corresponds to the number of students in a certain emotion; each column consists of a plurality of rectangles, each rectangle represents a student, two dark rectangles are embedded in the rectangles, the height of the upper rectangle corresponds to the attention of the student, and the height of the lower rectangle corresponds to the fatigue degree of the student; sorting according to attention; the teacher can change different angles according to the teaching scene to overview the participation of students; when the teacher suspends the mouse above the view, the view stops updating in real time, so that the teacher can conveniently check or select students; when the mouse is suspended above a certain student, the detailed information of the student can be presented; the information of students can be checked when clicking; the teacher can select interested student groups from the view and further analyze the student groups;
(3) and (3) monitoring view: participation to help teachers keep their attention on interested students; constructing a face model (as shown in fig. 5); each face represents a student, the color of the hair is used for corresponding to the emotion of the student, and the emotion color is consistent with the abstract view; using the relative position of the eyeball in the eye to correspond to the sight direction of the student; if the student looks at the screen, the eyeball is in the center of the eyes; encoding a head rotation direction using relative positions of hair, eyes, nose and mouth in a human face; the degree of fatigue of the student is encoded using the height of the eyelid; through the monitoring view, the teacher can sense abnormal participation behaviors of students;
(4) distribution view: for helping the teacher understand the student's engagement distribution over a certain period of time (as shown in fig. 6); the distribution view is a matrix, and each grid in the matrix represents a student; representing each student with a glyph design, wherein the middle pie chart (shown in FIG. 7) represents the emotional distribution of the student over a specified time period; the circle next to the pie chart represents attention, the angle of the dark circle corresponds to the average attention over the time period, and the angle of the top of the light circle corresponds to the upper quartile of attention; the rings on the outermost layer represent fatigue degree, the angle of the dark rings corresponds to the average fatigue degree, and a full circle represents complete fatigue; the angle of the top of the light-colored ring represents the upper quartile of the fatigue degree; the distribution views can be sorted in various ways, and the student positions can be kept fixed and updated in real time after sorting is completed; after the teacher selects a part of students from the abstract view, the distribution view ranks the student groups from high attention to low attention by default; a time bar sliding block exists in the distribution view, and a teacher can select a certain time period interval through the sliding block and check corresponding information; the teacher can check the detailed information of the students in a mouse suspension mode and add the students into the monitoring view and the personal view in a clicking mode;
(5) personal view: the system comprises a teacher management system, a student management system and a teacher management system, wherein the teacher management system is used for providing teacher with relevant characteristic information of student participation and original images of students; the time span presented in the personal view is consistent with the time slider in the distribution view; the personal view comprises a sight line direction, a head direction, an eye length-width ratio, a mouth length-width ratio, a student original image and student action information; the X-axis in the personal view (shown in fig. 8) represents time; a group of dotted lines are respectively arranged above and below the personal view and used for representing the sight line direction of the students when watching the upper edge and the lower edge of the screen; a flow graph exists in the dotted lines, the width of the flow graph is equal to the width between the two dotted lines, and the position represents the direction of the head; the upper flowsheet shows the head rotating direction around the Y axis, and the lower flowsheet shows the head rotating direction around the X axis; solid lines color coded are used to represent the direction of sight; if the line is between the two color lines, the screen is viewed, and if the line is outside the color lines, the line indicates that the sight line of the student is separated from the screen. The color of the line encodes the student's emotion classification; the color of the emotion is consistent with the abstract view; two gantt charts exist in the middle of the two river charts, the first gantt chart represents blink information, the blank area in the gantt chart represents eye closure, and the transparency of the matrix in the gantt chart encodes the average eye aspect ratio between blinks; the second Gantt chart presents mouth opening information, if a rectangle with a specific color appears, mouth opening is represented, and if the rectangle with the specific color does not appear, mouth closing is represented; the actions of the students are represented by color-coded circular icons below the first Gantt chart, the first color represents nodding heads, and the second color represents shaking heads; when a teacher clicks on a flowsheet at a certain moment in the flowsheet, the original image of the student at the moment is displayed to help the teacher verify the own analysis result.
The following experiment is combined to prove the practicability and effectiveness of the invention.
The experimental process comprises the following steps: in order to verify the effectiveness and the usability of the system, one of the cooperative teachers is invited to develop an online classroom teaching experiment by using the student participation degree visual analysis method provided by the invention. The teacher in this experiment was a lesson of 53 students teaching machine learning. The method comprises the steps of firstly requiring students to log in a data acquisition and preprocessing system on a browser, opening a camera through the browser, completing checking work of sight and head directions before class, and placing the browser in a corner of a desktop. Then, each student is clearly informed that the teacher can sense whether the student is in front of the screen or not through an image detection method or not. In addition, students are informed that the knowledge cognition conditions of the students can be transferred to the teacher at any time through the head nodding or head shaking behaviors. And finally introducing a using method of the student participation degree visual analysis method for the teacher. An additional display screen is provided for the teacher to display the visual analysis system. During the experiment, the effectiveness and usability of the system was demonstrated by three cases.
Case analysis one: quick overview student participation
In the online teaching process, a teacher can check the participation condition of all students irregularly, and then various teaching means are adopted according to the states of the students, so that the learning atmosphere in a classroom is enhanced, and the teaching quality is improved. However, in an online classroom, since it is difficult for a teacher to quickly sense participation of students, the teacher often takes his or her own lessons and rarely interacts with the students. In the experiment, a teacher quickly senses the participation conditions of all students by using the proposed participation degree visual analysis method and makes a series of active teaching decisions. First he chooses to order by attention in the abstract view. At the beginning of the course, the students are more focused, but after the course is taken for a while, the teacher finds that most of the students' attention is shifted to the range of 0.25-0.75, and the feelings of some of the students are difficult, and the fatigue degree is improved compared with the former, as shown in fig. 9. The teacher being aware of his course may be bored by the students. Thus, the teacher stops, and says: "please note to listen and talk, and then i talk is the key point. "and increase the volume appropriately in the following lessons, and also accentuate the mood when speaking the important knowledge points. After the teacher takes a series of teaching measures, the attention of the students is restored to the range of 0.75-1.
Because the cognitive state feedback function is provided, teachers can frequently interact with students in the course of lessons. For example, after a teacher finishes speaking a knowledge point, the teacher presents the student with the question: "do everyone understand it? "after asking the question, the teacher looks at the student's nodding and shaking head in the abstract view. He found that most students presented red circles underneath, but there were several students who showed green circles as shown in fig. 10, which indicated that most students were understandably, but that there were several students who were still in doubt. He then stops giving lessons and asks the students to type the question into the chat interface. And the course is continued after the student questions are solved.
In addition, the teacher is also very concerned with the emotional state of the students. The teacher says that he can try to keep the students in a natural and happy emotional state during the course of the class. However, in the course of lessons, the teacher finds that more negative emotions such as difficulty, anger, depression and the like are added in the abstract view. Then the students who are ranked according to emotion are selected in the abstract view, then the negative emotions are swiped, and the emotion distribution of the students in the past half minute is checked in the distribution view, as shown in fig. 11, and the students find that the negative emotions have lasted for a period of time. The teacher therefore temporarily breaks the lesson, giving the students a small joke to animate the classroom atmosphere. Then the number of happy students in the abstract view increases significantly. The teacher shows that he gets timely feedback, and the teaching confidence is obviously improved. During the following lecture, he slows down the classroom rhythm and makes every knowledge point as clear as possible.
The cases show that the invention can effectively transmit the participation information of students to teachers, and is helpful for teachers to make appropriate teaching decisions. In addition, the invention can effectively promote the interaction between teachers and students and improve the teaching quality and the learning effect of an online classroom.
Case analysis two: identifying abnormal participation students
In an online classroom, teachers and students have a serious space-time isolation problem, and teachers are difficult to find students who are out of class or participate in abnormal situations, so that the students cannot be corrected in time, and the learning effect is poor. The invention can help teachers quickly find students with abnormal participation. For example, during the course of the experiment, the teacher found that there was always one student in the 0-0.25 interval of attention, and the color was black from time to time, indicating that the student was not detected occasionally. For analysis reasons, the teacher views the student's personal information in the summary view and adds the student to the monitoring view and the personal view. The teacher finds that the student always heads down in the monitoring view, with the direction of sight also facing downwards, as shown in fig. 12. The teacher then sets the time slider to the last 90s and then looks at the student's personal view, as shown in FIG. 13. So he knows that the student starts to lower his head around 380s and stays for a long period of time. Therefore, the teacher suspects that the student stooped low to play the cell phone. Then the teacher clicks the latest time point to see the original image of the student, and the teacher insists on his own judgment by watching the original image of the student. The student is then given verbal warning.
The teacher also adjusts the time span of the distribution view to the last 30s during the course to see the student's engagement during the last period. Through the distribution view, the teacher can easily detect the abnormal participation of students. For example, the teacher finds that the fatigue level of the student 200048 is increased to 1, as shown in fig. 14(a), and the teacher adds it to the monitoring view and keeps observing it for a while, and finds that the eyelids thereof are always fully closed, as shown in fig. 14 (b). Finally, the teacher looks at the details of the student in the personal view, and as shown in fig. 14(c), he finds that the aspect ratio of the eyes of the student is less than the eye-closing threshold in the last period of time, and thus the teacher judges that the student is asleep. The teacher calls the criticism but the students do not respond, so the teacher calls other students to wake up the students.
From the distribution view, the teacher also found a pie in the middle of a student that was almost entirely purple, indicating that the student had a feeling of aversion to the lesson. When the teacher finds this, he asks the student to pull the student back into the classroom.
The teacher shows that the system can effectively help him to find students with abnormal participation, understand the reasons behind the abnormal participation of the students and then adopt effective teaching intervention measures. This is something that is difficult to do in normal on-line teaching.
Case analysis three: comparing the participation of different students
It is important for teachers to compare the participation of different students. Through comparison among students, the teacher can find students who are active or inactive in class, which provides favorable support for the teacher to score classroom performance and can help the teacher to find interested students. In this trial, the teacher found that more of the student's emotions were in a negative state, so he ordered them in the abstract view by emotion. He then selects students whose emotions are too and dislike to view in the monitor view. In order to invigorate the atmosphere, the teacher speaks a joke and most of the students' emotions in the monitoring view turn into happy or natural. But the teacher still finds that the student 100015 is still in a difficult state by comparison, as shown in fig. 15, whereby the teacher judges that the student needs some help. He then calls the student and connects to him to ask if he has any difficulties.
After the lesson is completed, the teacher sets the time slider in the distribution view to the duration of the entire lesson and then compares the participation of different students. As shown in fig. 16(a), the teacher finds students with high participation and students with low participation through the distribution view. Finally he compares the participation details of different students in the personal view. As shown in fig. 16(b), the student with high participation finds that the head movements of the student are relatively small, the visual line fluctuation is small, and the emotion is relatively positive, while the head movements of the student with low participation are large, the visual line fluctuation is large, and the student has many negative emotions.
Claims (10)
1. A real-time visual analysis method for student participation of online courses comprises the following steps:
s1, acquiring videos of students in online courses in real time;
s2, preprocessing the video data acquired in the step S1 so as to extract and obtain facial features of the students;
s3, establishing a student participation analysis model from four aspects of attention, emotion, fatigue degree and cognitive state;
and S4, visualizing the analysis result of the student participation degree analysis model established in the step S3, thereby completing the real-time visual analysis of the student participation degree of the online course.
2. The method for the real-time visual analysis of student participation in online lessons according to claim 1, wherein the step S1 of obtaining the video of the student in the online lesson in real time specifically comprises the following steps:
collecting real-time videos of students in the online course by using a camera;
setting the frame number of the collected images per second according to the blinking frequency of human eyes;
before the video acquisition begins, images { I) of students looking at the center of the screen and observing four vertexes around the screen at the center of the screen are acquired c ,I lt ,I lb ,I rt ,I rb The head direction and the sight line direction are checked; wherein I c For the student looking forward at the image in the center of the screen, I lt For the student viewing the top left corner of the screen at the center of the screen, I lb For the student viewing the image at the apex of the lower left corner of the screen at the centre of the screen, I rt For the student viewing the top right corner of the screen at the center of the screen, I rb The student views the image at the center of the screen at the vertex of the lower right corner of the screen.
3. The method for the real-time visual analysis of student participation in online lessons as claimed in claim 2, wherein the step S2 is performed to pre-process the video data obtained in step S1, so as to extract the facial features of the student, and the method comprises the following steps:
preprocessing the client side where the student is located; the preprocessing comprises face recognition preprocessing, face alignment preprocessing, emotion recognition preprocessing, head posture estimation preprocessing and sight line estimation preprocessing;
the face recognition preprocessing comprises the steps of detecting whether face information exists in an acquired image or not by adopting a deep learning model; the face alignment preprocessing is to adopt a neural network model to extract face coordinate position information in the acquired image; the emotion recognition preprocessing is to recognize real-time emotions of students by adopting a neural network model; the emotions include anger, nature, sadness, surprise, happiness, disgust, and fear; the head posture estimation preprocessing comprises the steps of extracting a head posture angle of a person from an acquired image by adopting a head posture technology, and judging the relation between the head of the student and a screen according to the head posture angle; and the sight estimation preprocessing comprises the steps of extracting the sight direction of the student by adopting a full-face-based sight estimation algorithm and judging the relation between the sight of the student and a screen according to the sight direction.
4. The method for the real-time visual analysis of student participation in an online course as claimed in claim 3, wherein said face recognition preprocessing specifically comprises the steps of:
detecting whether the image contains face information or not by adopting a multitask cascade convolution neural network deep learning model; detecting the position (x, y, w, h) of a face matrix in an image by using a multitask cascade convolution neural network deep learning model, wherein the position (x, y) represents the coordinate of the upper left corner of the face matrix when the upper left corner of the image is taken as a coordinate origin, w represents the width of a rectangular area corresponding to the detected face matrix, and h is the height of the rectangular area corresponding to the detected face matrix; in specific implementation, if the face matrix is obtained through detection, it indicates that face information is recognized.
5. The method for the real-time visual analysis of student participation in an online course as claimed in claim 4, wherein said face alignment preprocessing specifically comprises the steps of:
extracting the coordinates of key points of the human face by adopting a cascade convolution neural network model; when coordinates are extracted, the upper left corner of the image is taken as an origin, and the extracted coordinates of the key points of the human face are taken as [ x ] i ,y i ]。
6. The method as claimed in claim 5, wherein the emotion recognition preprocessing comprises the following steps:
adopting ResNet-50 as a convolutional neural network model, and adopting FER2013 facial expression recognition public data set for training to obtain a final preprocessing model; the emotions output by the preprocessing model include anger, nature, sadness, surprise, joy, disgust, and fear.
7. The method of claim 6, wherein the pre-processing of head pose estimation comprises the following steps:
acquiring the head posture of a person from the image by adopting a head posture estimation algorithm; the head posture is represented by three Euler angles, namely a pitch angle, a yaw angle and a roll angle, and is sequentially used for representing nodding, shaking and turning;
extracting the obtained head posture by adopting a method of projecting 2D to 3D based on key points as a head posture estimation algorithm; the head posture is represented by a three-dimensional vector [ Pitch, Yaw and Roll ], wherein Pitch is a Pitch angle, Yaw is a Yaw angle, and Roll is a Roll angle;
before the preprocessing of head posture estimation, carrying out pre-verification; during calibration, extracting the head attitude angles of the student looking at the center of the screen and the head attitude angles of the student in four directions when the head of the student observes the screen at the center of the screen; during verification, proportional conversion processing is carried out on the head postures of the students, so that the Pitch value and the Yaw value of the students watching the center of the screen are both 0, and the Pitch angle of the heads of the students watching the upper edge and the lower edge of the screen and the absolute value of the Yaw angle of the students watching the left edge and the right edge of the screen are 0.5;
during verification, the Pitch angle Pitch after verification is calculated by adopting the following formula 2 Is composed of
In the formula Pitch 1 For detecting the resulting Pitch angle, Pitch c Pitch angle when the student is looking straight at the center of the screen, Pitch t The pitch angle when the student watches the upper border of the screen;
after checking, if the deflection angle value is greater than 1, directly correcting the deflection angle value to be the maximum value 1;
and finally, when the deflection angle is between-0.5 and 0.5, the head direction of the student is determined to face the screen.
8. The method as claimed in claim 7, wherein the step of pre-processing the sight line estimation comprises the steps of:
extracting the sight direction of the student by adopting a full-face-based sight estimation algorithm to obtain the sight direction of the student as [ X, Y ], wherein X is the rotation angle of the sight in the horizontal direction, and Y is the rotation angle of the sight in the vertical direction;
before the sight line estimation preprocessing, the preliminary verification is carried out; during verification, line-of-sight angles of the center of the screen and the periphery of the screen are extracted when the head of a student is in the center of the screen; then according to the check result, converting the line-of-sight data to make the line-of-sight of the student when watching the center of the screen be [0,0], and making the absolute value of the angle corresponding to the line-of-sight of the student when watching the periphery of the screen be 0.5;
during verification, the angle value Y of the verified sight line direction is calculated by adopting the following formula 2 Is composed of
In the formula Y 1 For detecting the resulting angle value of the direction of sight, Y c The angle value, Y, of the direction of sight when the student is looking straight at the center of the screen t For the direction of sight of students looking at the border on the screenThe angle value of (d);
after checking, if the angle value of the sight line direction is greater than 1, directly correcting the angle value to be the maximum value 1;
and finally, when the angle value of the sight line direction is-0.5, determining the watching screen of the student.
9. The method for the real-time visual analysis of student participation in online lessons as claimed in claim 8, wherein said step S3 of establishing a student participation analysis model from four aspects of attention, emotion, fatigue and cognitive status comprises the following steps:
attention is paid to the following aspects:
the state variable S of the watching screen of the student is calculated by the following formula and is S | < X | < 0.5 & | < Y | < 0.5'; in the formula, X and Y are the pretreated sight line directions of the students; the value rule of "a" is: if a is true, "a" is 1, if a is false, "a" is 0; and operation; when the state variable S of the student watching screen is 1, the student watching screen is indicated; when the state variable S of the student watching screen is 0, the student does not watch the screen;
receiving the sight direction [ X, Y ] of the student by adopting a sliding window with the time span of 30s]And the student watches the screen state variable S to obtain N ═ X 1 ,X 2 ,...,X n ]、M＝[Y 1 ,Y 2 ,...,Y n ]And W ═ S 1 ,S 2 ,...,S n ]Wherein X is i Is the horizontal gaze direction, Y, of the student in the ith frame i Is the vertical gaze direction of the student in the ith frame, S i A state variable of a screen for the students in the ith frame; calculating to obtain the student stupefied state variable D ═ (max (N) -min (N) ≦ T'&"(max (M) -min (M) ≦ T", where max (N) is the maximum of the elements in N, min (N) is the minimum of the elements in N, D is 1 indicating no student's loss, D is 0 indicating student's loss; finally, the attention variable A of the student is calculated by the following formula
The fatigue degree is as follows:
(1) aiming at students, selecting an average eye length-width ratio B2, a blinking frequency P2, an average blinking time length D2 and a yawning frequency M2 as indexes to model fatigue degrees of the students;
extracting eye key point and mouth key point information of the human face: wherein the eye key points include 6 key points, eye key point E1 at the outermost eye corner, eye key point E4 at the innermost eye corner, eye key point E2 at the outer side 1/4 of the superior eye contour, eye key point E4 at the inner side 1/4 of the superior eye contour, eye key point E6 at the outer side 1/4 of the inferior eye contour, and eye key point E5 at the inner side 1/4 of the inferior eye contour; the mouth keypoint information includes 4 keypoints, a leftmost keypoint M1 of the mouth contour, a rightmost keypoint M3 of the mouth contour, a topmost keypoint M2 of the upper lip, and a bottommost keypoint M4 of the lower lip; each key point is a two-dimensional coordinate;
the length-width ratio alpha of eyes of the students is calculated by the following formula
In the formula, | E2-E6| represents the straight-line distance between the key point E2 and the key point E6; | E3-E5| represents the straight-line distance between the key point E3 and the key point E5; | E1-E4| represents the straight-line distance between the key point E1 and the key point E4;
the length-width ratio beta of the mouth of the student is calculated by the following formula
Wherein | M3-M1| represents the linear distance between the key point M3 and the key point M1, | M2-M4| represents the linear distance between the key point M2 and the key point M4;
the eye length-width ratio and the mouth length-width ratio of each student are obtained in advance and normalized, so that the eye length-width ratio alpha is 0 when the eyes are completely closed, and the eye length-width ratio alpha is 1 when the eyes are completely opened; the mouth aspect ratio β is 0 when the mouth is fully closed and 1 when the mouth is fully open;
when the aspect ratio alpha of the eyes of the student is detected to be smaller than a set threshold value, the student is determined to be closed;
(2) intercepting the video image by adopting a sliding window with the duration of 30s to obtain F ═ B 1 ,B 2 ,...,B n ]In which B is i The eye length-width ratio of the ith frame image in the sliding window; the average eye length-width ratio B of the students is calculated by the following formula
Defining the blinking frequency P as the number of blinking actions of the students in 30 s;
intercepting the video image by adopting a sliding window with the duration of 30s to obtain G ═ D 1 ,D 2 ,...,D m ]Wherein m blinks are detected in a sliding window, D i The time length of the ith blink; the average blink time length D of the student is calculated by the following formula
When the length-width ratio of the mouth part is larger than a set threshold value and the duration time is larger than the set threshold value, determining that the student has yawning behavior; the frequency M of the yawning of the students is the frequency of the yawning within 30 s;
analyzing the fatigue degree of the students by adopting a dynamic weight fuzzy comprehensive evaluation method: finally, the initial average eye length-width ratio B weight, the blink frequency P weight, the average blink time length D weight and the frequency of the beat-down frequency M weight are sequentially 0.42, 0.13, 0.32 and 0.13; when B is less than 0.2, adjusting the weight of the average eye length-width ratio B, the weight of the blink frequency P, the weight of the average blink time length D and the weight of the frequency of the beat-down frequency M to be 0.8, 0.1 and 0 in sequence; each index corresponds to a corresponding value V when the index is in different values; a value of 0 when B is 1, a value of 0.2 when B is 0.8, a value of 0.4 when B is 0.6, a value of 0.6 when B is 0.4, a value of 0.8 when B is 0.2, and a value of 1 when B is 0; p has a value of 0 in the range of 6 to 8, a value of 0.2 in the range of 5 or 9, P is 4 or has a value of 0.4 in the range of 10 to 12, P is 3 or has a value of 0.6 in the range of 12 to 14, P is 2 or has a value of 0.8 in the range of 15 to 16, P is less than or equal to 1 or P is more than 16A value of 1; d has a value of 0 in the range of 0.2 to 0.4, a value of 0.2 in the range of 0.5 to 0.6, a value of 0.4 in the range of 0.7 to 0.8, a value of 0.6 in the range of 0.8 to 1.5, a value of 0.8 in the range of 1.5 to 3, and a value of 1 when D is 3 or more; the value is 0 when M is equal to 0, 0.4 when M is equal to 1, 0.6 when M is equal to 2, 0.8 when M is equal to 3, and 1 when M is greater than or equal to 4; finally, the fatigue degree F is calculated as: f ═ V B *W B +V P *W P +V D *W D +V M *W M Wherein W is B Is the average eye aspect ratio B weight, W P As a weight of blink frequency P, W D Is the average blink time length D weight, W M For weighting the frequency M of the frequency of the down-frequency, V B Is the value of the aspect ratio B of the eye, V P Value of blink frequency P, V D Value of average blink duration D, V M The value of the frequency M of the beat frequency and the frequency of the defaulting;
emotional aspect:
classifying emotions into anger, nature, sadness, surprise, joy, disgust and fear by adopting an emotion classification model;
cognitive status aspects:
stipulating that the student shakes head when understanding the course content and shakes head when not understanding the course content;
detecting the head movement of the student: extracting a yaw angle and a pitch angle of the head of a student, and then establishing two sliding windows with the time span of 3s to respectively receive the yaw angle and the pitch angle;
the method comprises the following steps of detecting the nodding behavior and the shaking behavior of students:
stipulating that if the head direction continuously descends to exceed a set threshold value, judging that head lowering action occurs; if the head direction continuously rises and exceeds a set threshold value, judging that head-up behavior occurs; and if the head-lowering behavior and the head-raising behavior are detected simultaneously in the observation interval, judging that the head-nodding behavior occurs.
10. The method for the real-time visual analysis of student participation in online courses as claimed in claim 9, wherein said step S4 of visualizing the analysis results of the student participation analysis model established in step S3 specifically comprises the steps of:
(1) constructing an abstract view, a monitoring view, a distribution view and a personal view for visualizing the analysis result; the abstract view is used for presenting the participation degree of all students in real time; the monitoring view is used for presenting the participation degree of a designated student group in real time; the distribution view is used for presenting the participation degree distribution condition of the students in a specified time period; the personal view is used for displaying the participation degree related characteristics of the individual students;
(2) and (3) abstract view: constructing an abstract attempt based on the stacked histogram for presenting four indexes of emotion, fatigue degree, attention and student state of the student at the same time; presenting the participation degrees of students from three angles, namely attention, fatigue degree and emotion;
overview of student participation from attention and fatigue perspective: the Y axis represents attention intervals of different degrees and is divided into four groups from low to high; the X-axis represents the number of students in each group; each column consists of a plurality of small rectangles, each rectangle represents a student, and the color of each rectangle corresponds to the emotion of the student; the fatigue degree is represented by an embedded dark matrix in each small rectangle; wherein the height of the embedded matrix represents a value of fatigue level; sequencing according to a mode of reducing fatigue degree so as to avoid the problem of visual confusion; displaying the cognitive state and the stubborn state of the student by using circular icons with different colors below each rectangle; by summarizing the views of student participation from an attention perspective and a fatigue perspective, a teacher can quickly summarize the participation of students;
overview of student participation from an emotional perspective: the Y axis represents the emotion types of students, and the length of the rectangular column corresponds to the number of students in a certain emotion; each column consists of a plurality of rectangles, each rectangle represents a student, two dark rectangles are embedded in the rectangles, the height of the upper rectangle corresponds to the attention of the student, and the height of the lower rectangle corresponds to the fatigue degree of the student; sorting according to attention; the teacher can change different angles according to the teaching scene to overview the participation of students; when the teacher floats the mouse above the view, the view stops updating in real time, so that the teacher can conveniently check or select students; when the mouse is suspended above a certain student, the detailed information of the student can be presented; the information of students can be checked when clicking; the teacher can select interested student groups from the view and further analyze the student groups;
(3) and (3) monitoring view: participation to help teachers keep their attention on interested students; constructing a human face model; each face represents a student, the color of the hair is used for corresponding to the emotion of the student, and the emotion color is consistent with the abstract view; using the relative position of the eyeball in the eye to correspond to the sight direction of the student; if the student looks at the screen, the eyeball is in the center of the eyes; encoding a head rotation direction using relative positions of hair, eyes, nose and mouth in a human face; the degree of fatigue of the student is encoded using the height of the eyelid; through the monitoring view, the teacher can sense abnormal participation behaviors of students;
(4) distribution view: the system is used for helping teachers understand the participation degree distribution condition of students in a certain time period; the distribution view is a matrix, and each grid in the matrix represents a student; representing each student by a glyph design, wherein a pie chart in the middle represents the emotion distribution of the student in a specified time period; the circle next to the pie chart represents attention, the angle of the dark circle corresponds to the average attention over the time period, and the angle of the top of the light circle corresponds to the upper quartile of attention; the rings on the outermost layer represent fatigue degree, the angle of the dark rings corresponds to the average fatigue degree, and a full circle represents complete fatigue; the angle of the top of the light-colored ring represents the upper quartile of the fatigue degree; the distribution views can be sorted in various ways, and the student positions can be kept fixed and updated in real time after sorting is completed; after the teacher selects part of students from the abstract view, the distribution view can sort the student groups from high to low according to attention by default; a time bar sliding block exists in the distribution view, and a teacher can select a certain time period interval through the sliding block and check corresponding information; the teacher can check the detailed information of the students in a mouse suspension mode and add the students into the monitoring view and the personal view in a clicking mode;
(5) personal view: the system comprises a teacher management system, a student management system and a teacher management system, wherein the teacher management system is used for providing teacher with relevant characteristic information of student participation and original images of students; the time span presented in the personal view is consistent with the time slider in the distribution view; the personal view comprises a sight line direction, a head direction, an eye length-width ratio, a mouth length-width ratio, a student original image and student action information; the X-axis in the personal view represents time; a group of dotted lines are respectively arranged above and below the personal view and used for representing the sight line direction of the students when watching the upper edge and the lower edge of the screen; a flow graph exists in the dotted line, the width of the flow graph is equal to the width between the two dotted lines, and the position represents the direction of the head; the upper flowsheet shows the head rotating direction around the Y axis, and the lower flowsheet shows the head rotating direction around the X axis; solid lines color coded are used to represent the direction of sight; if the line is between the two color lines, the screen is viewed, and if the line is outside the color lines, the sight of the student is separated from the screen; the color of the line encodes the student's emotion classification; the color of the emotion is consistent with the abstract view; two gantt charts exist in the middle of the two river charts, the first gantt chart represents blink information, the blank area in the gantt chart represents eye closure, and the transparency of the matrix in the gantt chart encodes the average eye aspect ratio between blinks; the second Gantt chart presents mouth opening information, if a rectangle with a specific color appears, mouth opening is represented, and if the rectangle with the specific color does not appear, mouth closing is represented; the actions of the students are represented by color-coded circular icons below the first Gantt chart, the first color represents nodding heads, and the second color represents shaking heads; when a teacher clicks on a flowsheet at a certain moment in the flowsheet, the original image of the student at the moment is displayed to help the teacher verify the own analysis result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210540837.XA CN114973126A (en) | 2022-05-17 | 2022-05-17 | Real-time visual analysis method for student participation degree of online course |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210540837.XA CN114973126A (en) | 2022-05-17 | 2022-05-17 | Real-time visual analysis method for student participation degree of online course |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114973126A true CN114973126A (en) | 2022-08-30 |
Family
ID=82984234
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210540837.XA Pending CN114973126A (en) | 2022-05-17 | 2022-05-17 | Real-time visual analysis method for student participation degree of online course |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114973126A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115439915A (en) * | 2022-10-12 | 2022-12-06 | 首都师范大学 | Classroom participation identification method and device based on region coding and sample balance optimization |
| CN116453198A (en) * | 2023-05-06 | 2023-07-18 | 广州视景医疗软件有限公司 | Sight line calibration method and device based on head posture difference |
| CN117591058A (en) * | 2024-01-18 | 2024-02-23 | 浙江华创视讯科技有限公司 | Display method, device and storage medium for multi-person speech |
-
2022
- 2022-05-17 CN CN202210540837.XA patent/CN114973126A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115439915A (en) * | 2022-10-12 | 2022-12-06 | 首都师范大学 | Classroom participation identification method and device based on region coding and sample balance optimization |
| CN116453198A (en) * | 2023-05-06 | 2023-07-18 | 广州视景医疗软件有限公司 | Sight line calibration method and device based on head posture difference |
| CN116453198B (en) * | 2023-05-06 | 2023-08-25 | 广州视景医疗软件有限公司 | Sight line calibration method and device based on head posture difference |
| CN117591058A (en) * | 2024-01-18 | 2024-02-23 | 浙江华创视讯科技有限公司 | Display method, device and storage medium for multi-person speech |
| CN117591058B (en) * | 2024-01-18 | 2024-05-28 | 浙江华创视讯科技有限公司 | Display method, device and storage medium for multi-person speech |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114973126A (en) | Real-time visual analysis method for student participation degree of online course | |
| CN110287790B (en) | Learning state hybrid analysis method oriented to static multi-user scene | |
| Dewan et al. | A deep learning approach to detecting engagement of online learners | |
| CN110175501B (en) | Face recognition-based multi-person scene concentration degree recognition method | |
| CN108175425B (en) | Analysis processing device and cognitive index analysis method for potential value test | |
| Dubbaka et al. | Detecting learner engagement in MOOCs using automatic facial expression recognition | |
| CN112883867A (en) | Student online learning evaluation method and system based on image emotion analysis | |
| CN111814556A (en) | Teaching assistance method and system based on computer vision | |
| CN111507227A (en) | Multi-student individual segmentation and state autonomous identification method based on deep learning | |
| CN115205764B (en) | Online learning concentration monitoring method, system and medium based on machine vision | |
| CN113779289A (en) | Drawing step reduction system based on artificial intelligence | |
| CN112686462A (en) | Student portrait-based anomaly detection method, device, equipment and storage medium | |
| CN115937928A (en) | Learning state monitoring method and system based on multi-vision feature fusion | |
| Pablos et al. | Dynamic facial emotion recognition oriented to HCI applications | |
| CN113239794B (en) | Online learning-oriented learning state automatic identification method | |
| Daza et al. | MATT: Multimodal Attention Level Estimation for e-learning Platforms | |
| CN113887386A (en) | Fatigue detection method based on multi-feature fusion of deep learning and machine learning | |
| CN114187640A (en) | Learning situation observation method, system, equipment and medium based on online classroom | |
| Shah et al. | Assessment of student attentiveness to e-learning by monitoring behavioural elements | |
| CN113095259A (en) | Remote online course teaching management method | |
| CN112115779B (en) | Interpretable classroom student emotion analysis method, system, device and medium | |
| CN115966003A (en) | System for evaluating online learning efficiency of learner based on emotion recognition | |
| CN115829234A (en) | Automatic supervision system based on classroom detection and working method thereof | |
| CN114022918A (en) | Multi-posture-based learner excitement state label algorithm | |
| CN113505750A (en) | Identification method, device, electronic equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |