CN112784733A - Emotion recognition method and device based on online education and electronic equipment - Google Patents
Emotion recognition method and device based on online education and electronic equipment Download PDFInfo
- Publication number
- CN112784733A CN112784733A CN202110080001.1A CN202110080001A CN112784733A CN 112784733 A CN112784733 A CN 112784733A CN 202110080001 A CN202110080001 A CN 202110080001A CN 112784733 A CN112784733 A CN 112784733A
- Authority
- CN
- China
- Prior art keywords
- feature point
- distance
- user
- face
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 60
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 34
- 230000008451 emotion Effects 0.000 claims abstract description 33
- 238000000605 extraction Methods 0.000 claims abstract description 28
- 238000005070 sampling Methods 0.000 claims abstract description 20
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 15
- 210000000744 eyelid Anatomy 0.000 claims description 90
- 238000004590 computer program Methods 0.000 claims description 20
- 238000012163 sequencing technique Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 10
- 230000008569 process Effects 0.000 description 10
- 238000004422 calculation algorithm Methods 0.000 description 9
- 230000002996 emotional effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000001815 facial effect Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000014509 gene expression Effects 0.000 description 2
- 206010041349 Somnolence Diseases 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012847 principal component analysis method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000007306 turnover Effects 0.000 description 1
- 239000013598 vector Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/171—Local features and components; Facial parts ; Occluding parts, e.g. glasses; Geometrical relationships
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
- G06Q50/205—Education administration or guidance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Educational Administration (AREA)
- Tourism & Hospitality (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Strategic Management (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Educational Technology (AREA)
- Biomedical Technology (AREA)
- Primary Health Care (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Economics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Marketing (AREA)
- Evolutionary Biology (AREA)
- General Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
A method of emotion recognition based on online education, the method comprising: the method comprises the steps of inputting a face sampling image of a user into a pre-configured feature extraction model to obtain an eye feature point set, wherein the eye feature point set comprises at least two feature points, the feature extraction model is a trained convolutional neural network model, then calculating distance information between the feature points according to position information corresponding to the at least two feature points, and finally determining the current emotion of the user based on a comparison result of the distance information and a corresponding standard value. The emotion state of the current user is judged by performing feature decomposition on the face sampling image of the user, so that the technical problem that the emotion state of the user cannot be judged visually in the traditional online education is solved, a teacher giving lessons can directly acquire the emotion state of the user, corresponding lesson adjustment is performed, and higher-quality online education is realized.
Description
Technical Field
The application belongs to the technical field of data processing, and particularly relates to an emotion recognition method and device based on online education, a storage medium and electronic equipment.
Background
On-line education, which is a teaching mode taking a network as a medium, and through the network, students and teachers can develop teaching activities even if the distance between the students and the teachers is ten thousand; in addition, by means of the network courseware, the students can learn at any time and any place, the limitation of time and space is broken really, and online education is the most convenient learning mode for employees who work busy and have unfixed learning time.
Compared with the traditional field education, the on-line education has the advantages that the teacher giving lessons cannot visually judge the emotional state of the lesson-listening user (for example, the lesson-listening user is in a sleepy state for a long time or in a deep thinking state for a long time), the teaching mode cannot be flexibly adjusted directly according to the feedback of the emotional state of the lesson-listening user, and the teaching quality is further influenced to a certain extent.
Content of application
The embodiment of the application provides an emotion recognition method, device, storage medium and electronic equipment based on online education, and aims to solve the technical problem that the teaching quality is affected because a teacher in traditional online education cannot directly acquire the emotion state of a user who listens to a lesson, so that the teacher in the lesson can directly acquire the emotion state of the user who listens to the lesson and flexibly adjust the teaching mode, and the teaching quality is guaranteed.
In order to solve the above technical problem, a first aspect of embodiments of the present application provides an emotion recognition method based on online education, the method including:
inputting a face sampling image of a user into a pre-configured feature extraction model to obtain an eye feature point set, wherein the eye feature point set comprises at least two feature points, and the feature extraction model is a trained convolutional neural network model;
calculating distance information between the characteristic points according to the position information corresponding to the at least two characteristic points;
and determining the current emotion of the user based on the comparison result of the distance information and the corresponding standard value.
In order to solve the above technical problem, a second aspect of embodiments of the present application provides an emotion recognition method based on online education, the apparatus including:
the system comprises an acquisition module, a feature extraction module and a feature extraction module, wherein the acquisition module is used for inputting a face sampling image of a user into a pre-configured feature extraction model and acquiring an eye feature point set, the eye feature point set comprises at least two feature points, and the feature extraction model is a trained convolutional neural network model;
the calculation module is used for calculating distance information between the characteristic points according to the position information corresponding to the at least two characteristic points;
and the determining module is used for determining the current emotion of the user based on the comparison result of the distance information and the corresponding standard value.
In order to solve the above technical problem, a third aspect of embodiments of the present application provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the emotion recognition method based on online education as described above.
In order to solve the above technical problem, a fourth aspect of embodiments of the present application provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements an emotion recognition method based on online education as described above.
A fifth aspect of embodiments of the present application provides a computer-readable storage medium, which, when run on a terminal device, causes the terminal device to execute the method for emotion recognition based on online education provided in the first aspect of embodiments of the present application.
Compared with the prior art, the implementation mode of the invention has the following beneficial effects: the method comprises the steps of inputting a face sampling image of a user into a pre-configured feature extraction model to obtain an eye feature point set, wherein the eye feature point set comprises at least two feature points, the feature extraction model is a trained convolutional neural network model, then calculating distance information between the feature points according to position information corresponding to the at least two feature points, and finally determining the current emotion of the user based on a comparison result of the distance information and a corresponding standard value. The emotion state of the current user is judged by performing feature decomposition on the face sampling image of the user, so that the technical problem that the emotion state of the user cannot be judged visually in the traditional online education is solved, a teacher giving lessons can directly acquire the emotion state of the user, corresponding lesson adjustment is performed, and higher-quality online education is realized.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required to be used in the description of the embodiments or the prior art will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without inventive labor.
FIG. 1 is a schematic flow chart of a first implementation process of an emotion recognition method based on online education provided by an embodiment of the application;
FIG. 2 is a flowchart illustrating a second implementation process of an emotion recognition method based on online education according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an emotion recognition device based on online education according to an embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It should be understood that the order of writing each step in this embodiment does not mean the order of execution, and the order of execution of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of this embodiment.
Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.
In order to explain the technical means described in the present application, the following description will be given by way of specific embodiments.
Referring to fig. 1, which is a flowchart illustrating a first implementation procedure of an emotion recognition method based on online education provided by an embodiment of the present application, for convenience of explanation, only parts related to the embodiment of the present application are shown.
A preferred embodiment of the present disclosure may be a method for emotion recognition based on online education, the method including:
s101, inputting a face sampling image of a user into a pre-configured feature extraction model to obtain an eye feature point set, wherein the eye feature point set comprises at least two feature points, and the feature extraction model is a trained convolutional neural network model and can be a trained convolutional neural network model.
In this embodiment, the emotion recognition method based on online education may be applied to an online education system, the face sample image of the user refers to a face image corresponding to a user currently using the online education system, and the face sample image may be obtained in a variety of ways, for example, by capturing a video of a user who is listening to the online education system at a fixed time interval, or by selecting from video information obtained by a terminal device.
The feature extraction model is used to extract feature points of an eye region in a face sample image, and the present embodiment does not limit the feature point positioning manner of the face image to be recognized, for example, the position of the eye feature points in the face image to be recognized may be determined by an asm (active Shape model) algorithm, an aam (active application model) algorithm, or based on a dlib face detection algorithm, and an eye feature training set is established. And inputting the eye feature training set into a convolutional neural network model (CNN model) for training to obtain eye feature points in the face image to be recognized.
In some preferred embodiments, the image data for training the convolutional neural network model (CNN model) is preferably continuous frame image data, and the continuous frame image data are grouped in sequence in advance; wherein each group of image data comprises N continuous frames of images, N > 1; and positioning eye feature points of each group of image data, and extracting the eye features in each frame of image through a preset convolutional neural network model. In some preferred embodiments, N-10, and correspondingly, the length of the eye feature list is 10. For the received continuous frame number image data, every 10 frame images are analyzed as one group in turn. It should be noted that the image data with continuous frames may be video data, the video data is divided into a plurality of sub-video data with fixed length, and then the target frame image of each sub-video data is analyzed and screened through the facial image quality as the facial image to be recognized, so that the calculation amount can be reduced, and the recognition efficiency can be improved.
Further, after the face sample image is input into a pre-configured feature extraction model, an eye feature point set is obtained, the eye feature point set refers to a feature point set corresponding to an eye in the face sample image, and the eye feature point set comprises at least two feature points.
It should be noted that when the face sample image is obtained, it is necessary to determine whether a face exists in the image to be recognized through a corresponding face detection algorithm. The face detection algorithm may include a template matching method, a principal component analysis method, a texture analysis method, a spatial face grayscale method, and the like, which is not limited herein.
S102: and calculating the distance information between the characteristic points according to the position information corresponding to the at least two characteristic points.
In this embodiment, after the feature extraction model outputs the at least two feature points, the direct distance information of the at least two feature points is calculated according to the pre-configured spatial coordinates. The distance information between the at least two feature points refers to a distance between two corresponding feature points defined in advance, for example, the at least two feature points are a feature point corresponding to an upper left eyelid and a feature point corresponding to a lower left eyelid, and the distance information includes a spatial distance between the feature point corresponding to the upper left eyelid and the feature point corresponding to the lower left eyelid. Of course, the distance information may include distances between the plurality of features, depending on the pre-configured settings. It should be noted that, the way of calculating the distance information between the at least two feature points may be implemented by using an algorithm that can be implemented in the prior art, and this document is not limited further.
In some possible implementations, the set of eye feature points includes a left upper eyelid feature point, a left lower eyelid feature point, a right upper eyelid feature point, and a right lower eyelid feature point. Correspondingly, the distance information includes position information corresponding to the left upper eyelid feature point and the left lower eyelid feature point, position information of the right upper eyelid feature point and the right lower eyelid feature point, position information of the left upper eyelid feature point and the right upper eyelid feature point, and position information of the left lower eyelid feature point and the right lower eyelid feature point. Of course, more distance information can be obtained according to the permutation and combination, and the four distance information are preferred in the present application, and the specific embodiment is as follows:
the distance information comprises a first distance, a second distance, a third distance and a fourth distance;
correspondingly, the calculating the distance information between the feature points according to the position information corresponding to the feature points includes:
acquiring position information corresponding to the left upper eyelid feature point, the left lower eyelid feature point, the right upper eyelid feature point and the right lower eyelid feature point;
calculating a first distance according to the position information of the upper left eyelid feature point and the lower left eyelid feature point;
calculating a second distance according to the position information of the right upper eyelid feature point and the right lower eyelid feature point;
calculating a third distance according to the position information of the left upper eyelid feature point and the right upper eyelid feature point;
and calculating a fourth distance according to the position information of the left lower eyelid feature point and the right lower eyelid feature point.
In this embodiment, according to the position information corresponding to the left upper eyelid feature point, the left lower eyelid feature point, the right upper eyelid feature point, and the right lower eyelid feature point, the spatial distance between two feature points can be calculated, which is preferably the first distance, the second distance, the third distance, and the fourth distance. It should be noted that, the left upper eyelid feature point and the sitting eyelid feature point are preferably the intersection points of the center point of the eye and the left upper eyelid and the left lower eyelid in the vertical direction, that is, the intersection points of the left upper eyelid and the left lower eyelid and the perpendicular bisector of the eye, that is, the intersection points correspond to the left upper eyelid feature point and the left lower eyelid feature point. Then, the first distance corresponds to a distance between the intersection points of the upper left eyelid and the lower left eyelid with the perpendicular bisector of the eye. Preferably the right eye setting is also the same as the left eye.
In some possible embodiments, the positions corresponding to the left upper eyelid feature point, the left lower eyelid feature point, the right upper eyelid feature point, and the right lower eyelid feature point may also be other points, such as the left upper eyelid feature point and the left lower eyelid feature point, which are all set to be translated by a preset distance in the horizontal direction from the left eye angle to the right eye angle, that is, the positions corresponding to the left upper eyelid feature point and the left lower eyelid feature point.
S103: and determining the current emotion of the user based on the comparison result of the distance information and the corresponding standard value.
In this embodiment, referring to the above-mentioned embodiments, the distance information includes a first distance, a second distance, a third distance and a fourth distance, and the standard values corresponding to the distance information also include four, which may be a first standard value corresponding to the first distance, a second standard value corresponding to the second distance, a third standard value corresponding to the third distance and a fourth standard value corresponding to the fourth distance. And after the distance information is acquired, automatically calling a pre-acquired standard value, carrying out corresponding comparison with the distance information, and confirming the current emotional state of the user according to a comparison result. For example, when the comparison result of the first distance and the first criterion is a first comparison result, and the emotion corresponding to the first comparison result is anger, the current emotional state of the user is anger.
In some possible embodiments, the obtaining manner of the standard value may be pre-entered, and the user may be prompted to be guided to enter the standard value. The specific steps can be as follows: after logging in the online education system, the user is prompted to input a standard value, the user is prompted to perform expression management, facial images are obtained in a smiling or common emotion mode, then the facial images are subjected to feature extraction, the spatial distance between each feature point is obtained, and the spatial distance is defined as the standard value corresponding to the user. In addition, the standard value can be given a certain weight according to the actual situation so as to improve the accuracy of emotion judgment.
In some possible embodiments, the determining the current emotion of the user based on the comparison result between the distance information and the corresponding standard value includes:
when the first distance is smaller than a first standard value and the second distance is smaller than a second standard value, outputting a first comparison result;
when the third distance is smaller than a third standard value and the fourth distance is smaller than a fourth standard value, outputting a second comparison result;
determining that the current emotion of the user is a state of overcoming a poverty when the number of the first comparison results exceeds a preset number threshold value within a preset time period;
and determining that the current emotion of the user is a thinking state when the number of the second comparison results exceeds a preset number threshold value within a preset time period.
In this embodiment, when the first distance is smaller than the first standard value and the second distance is smaller than the second standard value, that is, the distance between the upper left eyelid and the lower left eyelid and the distance between the upper right eyelid and the lower right eyelid are both smaller than the standard values corresponding to the conventional expressions of the user, it may be preliminarily judged that the user may be in a state of getting stuck, and of course, in order to avoid erroneous judgment, the occurrence number of the result (that is, the first comparison result) in a preset time period is set to be greater than the preset number, and it is determined that the user is in a state of getting stuck. And when the third distance is smaller than a third standard value and the fourth distance is smaller than a fourth standard value, namely the distance between the upper left eyelid and the upper right eyelid and the distance between the lower left eyelid and the lower right eyelid are reduced, the user may be in a frown and may be in a thinking state, and similarly, the occurrence frequency of the result (i.e., the second comparison result) in a preset time period is set to be larger than the preset frequency, and the user is determined to be in the thinking state.
It should be noted that, different distance information can be set according to different face rules to correspond to the face rules, so that the current state of the user can be determined, that is, other technical schemes for determining the emotional state according to the irregular face are also within the protection scope of the present application.
It should be noted that the application scene of the teaching teacher client is preferably on-line education, the emotion states of students can be captured in time, corresponding prompts corresponding to the emotion states are set, the teaching teacher is reminded to carry out supervision responsibility in time, the teaching mode is adjusted in time, and the teaching quality is improved.
In some preferred embodiments, before calculating the distance information between the feature points according to the position information corresponding to the at least two feature points, the method further includes:
and searching a corresponding relation list corresponding to the current user, and acquiring a pre-input standard value corresponding to the user.
In some other possible embodiments, before the inputting the sampled image of the face of the user into the pre-configured feature extraction model to obtain the set of eye feature points, the method further includes:
acquiring video information of a current user;
and determining the face image with the maximum image quality value in the video information as the face sampling image.
In this embodiment, the acquisition of the face sample image is performed by confirming and acquiring in the video information acquired by the terminal. The video information is the video information of the current user in the state of listening to the lesson. The image quality value is calculated according to a pre-configured calculation method. For example, a CW clustering algorithm (Chinese _ Whisper) may be used to cluster the face feature vectors in the face feature list. The CW clustering algorithm searches the category and carries out clustering by constructing an undirected graph, taking each face as a node in the undirected graph, taking the similarity between the faces as an edge between the nodes and iteratively searching the weight accumulation sum of the similarity corresponding to one node. The method specifically comprises the following steps:
the determining that the face image with the largest image quality value in the video information is the face sampling image specifically includes:
s201: and calculating the face image turning degree, the face image definition, the face image brightness and the face image size of each frame of the face image in the video information.
The face image turning degree refers to the rotation degree of a face of a user relative to a horizontal plane of a terminal camera device, the face image definition refers to the image definition of a face image, the image brightness refers to the brightness of a current frame, and the face image size refers to the area proportion of the face image in the current frame.
S202: and calculating the image quality value of each frame of the face image according to the face turning degree, the face image definition, the face image brightness and the face image size and preset and one-to-one corresponding weight values.
The image quality value scores the quality of the face image of the current frame in four dimensions of turnover degree, brightness, definition and area size. Meanwhile, in order to realize the accuracy of scoring, corresponding weights can be configured one by one according to actual requirements, the values of the weights can depend on the empirical values of multiple tests, and the scheme is not further limited.
In some preferred embodiments, the formula of the image quality value may be S ═ a × q1+ b × q2+ c × q3+ d × q4, where a, b, c, and d respectively represent the face turning degree, the face image sharpness, the face image brightness, and the face image size, q1, q2, q3, and q4 respectively represent weights of the face turning degree, the face image sharpness, the face image brightness, and the face image size, and a, b, c, and d may be positive numbers or negative numbers, may be in a predetermined interval, and may be defined according to actual needs.
S203: and sequencing the image quality values of the face images of each frame, and determining the face image corresponding to the image quality value at the top of the sequencing as the face sampling image.
The higher the score of the image quality value is, the higher the quality of the current face image is, the face image corresponding to the image quality value in the front of the ranking is taken as the face image with the highest quality, the face image is defined as a face sampling image, and the accuracy of subsequent emotion recognition is improved.
Compared with the prior art, the implementation mode of the invention has the following beneficial effects: the method comprises the steps of inputting a face sampling image of a user into a pre-configured feature extraction model to obtain an eye feature point set, wherein the eye feature point set comprises at least two feature points, the feature extraction model is a trained convolutional neural network model, then calculating distance information between the feature points according to position information corresponding to the at least two feature points, and finally determining the current emotion of the user based on a comparison result of the distance information and a corresponding standard value. The emotion state of the current user is judged by performing feature decomposition on the face sampling image of the user, so that the technical problem that the emotion state of the user cannot be judged visually in the traditional online education is solved, a teacher giving lessons can directly acquire the emotion state of the user, corresponding lesson adjustment is performed, and higher-quality online education is realized.
Corresponding to the emotion recognition device based on online education described in the embodiment of the emotion recognition method based on online education described above, fig. 3 shows a block diagram of the emotion recognition device based on online education provided in the embodiment of the present application, and for convenience of explanation, only the parts related to the embodiment of the present application are shown.
Referring to fig. 3, an emotion recognition apparatus 300 based on online education, the apparatus 300 comprising:
an obtaining module 301, configured to input a face sampling image of a user into a preconfigured feature extraction model, and obtain an eye feature point set, where the eye feature point set includes at least two feature points, and the feature extraction model is a trained convolutional neural network model;
a calculating module 302, configured to calculate distance information between the feature points according to the position information corresponding to the at least two feature points;
a determining module 303, configured to determine a current emotion of the user based on a comparison result between the distance information and the corresponding standard value.
In some other possible embodiments, the set of eye feature points includes a left upper eyelid feature point, a left lower eyelid feature point, a right upper eyelid feature point, and a right lower eyelid feature point.
In some other possible embodiments, the distance information includes a first distance, a second distance, a third distance, and a fourth distance;
correspondingly, the calculating module 302 is further specifically configured to:
acquiring position information corresponding to the left upper eyelid feature point, the left lower eyelid feature point, the right upper eyelid feature point and the right lower eyelid feature point;
calculating a first distance according to the position information of the upper left eyelid feature point and the lower left eyelid feature point;
calculating a second distance according to the position information of the right upper eyelid feature point and the right lower eyelid feature point;
calculating a third distance according to the position information of the left upper eyelid feature point and the right upper eyelid feature point;
and calculating a fourth distance according to the position information of the left lower eyelid feature point and the right lower eyelid feature point.
In some other possible embodiments, the determining module 303 is specifically configured to:
when the first distance is smaller than a first standard value and the second distance is smaller than a second standard value, outputting a first comparison result;
when the third distance is smaller than a third standard value and the fourth distance is smaller than a fourth standard value, outputting a second comparison result;
determining that the current emotion of the user is a state of overcoming a poverty when the number of the first comparison results exceeds a preset number threshold value within a preset time period;
and determining that the current emotion of the user is a thinking state when the number of the second comparison results exceeds a preset number threshold value within a preset time period.
In some other possible embodiments, the apparatus 300 further includes:
and the searching module is used for searching the corresponding relation list corresponding to the current user and acquiring the pre-input standard value corresponding to the user.
In some other possible embodiments, the apparatus 300 further includes:
the second acquisition module is used for acquiring the video information of the current user;
and the second determining module is used for determining the face image with the maximum image quality value in the video information as the face sampling image.
In some other possible embodiments, the second determining module is specifically configured to:
calculating the face image turning degree, the face image definition, the face image brightness and the face image size of each frame of face image in the video information;
calculating the image quality value of each frame of the face image according to the face turning degree, the face image definition, the face image brightness and the face image size and preset and one-to-one corresponding weight;
and sequencing the image quality values of the face images of each frame, and determining the face image corresponding to the image quality value at the top of the sequencing as the face sampling image.
It should be noted that, for the information interaction, execution process and other contents between the above-mentioned devices/modules, because the same concept is based on, the specific functions and technical effects of the embodiment of the emotion recognition method based on online education in the present application can be referred to in detail in the section of the embodiment of the emotion recognition method based on online education, and details are not described here.
It will be apparent to those skilled in the art that, for convenience and brevity of description, the above-mentioned division of the function modules is merely illustrated, and in practical applications, the above-mentioned function distribution may be performed by different function modules according to needs, that is, the internal structure of the emotion recognition apparatus 300 based on online education is divided into different function modules to perform all or part of the above-mentioned functions. Each functional module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional modules are only used for distinguishing one functional module from another, and are not used for limiting the protection scope of the application. For the specific working process of each functional module, reference may be made to the corresponding process in the embodiment of the emotion recognition method based on online education, which is not described herein again.
Fig. 4 is a schematic structural diagram of an electronic device 400 according to a third embodiment of the present application. As shown in fig. 4, the electronic device 400 includes: a processor 402, a memory 401, and a computer program 403 stored in the memory 401 and executable on the processor 402. The number of the processors 402 is at least one, and fig. 4 takes one as an example. The processor 402, when executing the computer program 403, implements one of the above-described steps of emotion recognition based on online education, i.e., the steps shown in fig. 1 or fig. 2.
The specific implementation process of the electronic device 400 can be seen in the above embodiment of the emotion recognition method based on online education.
Illustratively, the computer program 403 may be partitioned into one or more modules/units that are stored in the memory 401 and executed by the processor 402 to accomplish the present application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 403 in the terminal device 400.
The electronic device 400 may be a desktop computer, a notebook, a palm computer, a main control device, or other computing devices, or may be a camera, a mobile phone, or other devices having an image acquisition function and a data processing function, or may be a touch display device. The electronic device 400 may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that fig. 4 is merely an example of an electronic device 400 and does not constitute a limitation of electronic device 400 and may include more or fewer components than shown, or combine certain components, or different components, e.g., electronic device 400 may also include input-output devices, network access devices, buses, etc.
The Processor 402 may be a CPU (Central Processing Unit), other general-purpose Processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage 401 may be an internal storage unit of the electronic device 400, such as a hard disk or a memory. The memory 401 may also be an external storage device of the terminal device 400, such as a plug-in hard disk, SMC (Smart Media Card), SD (Secure Digital Card), Flash Card, or the like provided on the electronic device 400. Further, the memory 401 may also include both an internal storage unit and an external storage device of the electronic device 400. The memory 401 is used for storing an operating system, application programs, a boot loader, data, and other programs, such as program codes of the computer program 403. The memory 401 may also be used to temporarily store data that has been output or is to be output.
Embodiments of the present application further provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the steps in the embodiment of the emotion recognition method based on online education.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the embodiment of the emotion recognition method based on online education according to the present application can be implemented by a computer program, which can be stored in a computer readable storage medium, and when being executed by a processor, the computer program can implement the steps in the embodiment of the emotion recognition method based on online education. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing apparatus/terminal apparatus, a recording medium, computer Memory, ROM (Read-Only Memory), RAM (Random Access Memory), electrical carrier wave signal, telecommunication signal, and software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In certain jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and patent practice.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A method for emotion recognition based on online education, the method comprising:
inputting a face sampling image of a user into a pre-configured feature extraction model to obtain an eye feature point set, wherein the eye feature point set comprises at least two feature points, and the feature extraction model is a trained convolutional neural network model;
calculating distance information between the characteristic points according to the position information corresponding to the at least two characteristic points;
and determining the current emotion of the user based on the comparison result of the distance information and the corresponding standard value.
2. The method of claim 1, wherein the set of eye feature points comprises a left upper eyelid feature point, a left lower eyelid feature point, a right upper eyelid feature point, and a right lower eyelid feature point.
3. The method of claim 2, wherein the distance information comprises a first distance, a second distance, a third distance, and a fourth distance;
correspondingly, the calculating the distance information between the feature points according to the position information corresponding to the feature points includes:
acquiring position information corresponding to the left upper eyelid feature point, the left lower eyelid feature point, the right upper eyelid feature point and the right lower eyelid feature point;
calculating a first distance according to the position information of the upper left eyelid feature point and the lower left eyelid feature point;
calculating a second distance according to the position information of the right upper eyelid feature point and the right lower eyelid feature point;
calculating a third distance according to the position information of the left upper eyelid feature point and the right upper eyelid feature point;
and calculating a fourth distance according to the position information of the left lower eyelid feature point and the right lower eyelid feature point.
4. The method according to claim 3, wherein the determining the current emotion of the user based on the comparison result between the distance information and the corresponding standard value comprises:
when the first distance is smaller than a first standard value and the second distance is smaller than a second standard value, outputting a first comparison result;
when the third distance is smaller than a third standard value and the fourth distance is smaller than a fourth standard value, outputting a second comparison result;
determining that the current emotion of the user is a state of overcoming a poverty when the number of the first comparison results exceeds a preset number threshold value within a preset time period;
and determining that the current emotion of the user is a thinking state when the number of the second comparison results exceeds a preset number threshold value within a preset time period.
5. The method according to claim 1, wherein before calculating the distance information between the feature points according to the position information corresponding to the at least two feature points, the method further comprises:
and searching a corresponding relation list corresponding to the current user, and acquiring a pre-input standard value corresponding to the user.
6. The method according to any one of claims 1-5, wherein before inputting the sampled image of the user's face into a pre-configured feature extraction model to obtain the set of eye feature points, the method further comprises:
acquiring video information of a current user;
and determining the face image with the maximum image quality value in the video information as the face sampling image.
7. The method according to claim 6, wherein the confirming that the face image with the largest image quality value in the video information is the face sample image specifically comprises:
calculating the face image turning degree, the face image definition, the face image brightness and the face image size of each frame of face image in the video information;
calculating the image quality value of each frame of the face image according to the face turning degree, the face image definition, the face image brightness and the face image size and preset and one-to-one corresponding weight;
and sequencing the image quality values of the face images of each frame, and determining the face image corresponding to the image quality value at the top of the sequencing as the face sampling image.
8. An emotion recognition apparatus based on online education, characterized in that the apparatus comprises:
the system comprises an acquisition module, a feature extraction module and a feature extraction module, wherein the acquisition module is used for inputting a face sampling image of a user into a pre-configured feature extraction model and acquiring an eye feature point set, the eye feature point set comprises at least two feature points, and the feature extraction model is a trained convolutional neural network model;
the calculation module is used for calculating distance information between the characteristic points according to the position information corresponding to the at least two characteristic points;
and the determining module is used for determining the current emotion of the user based on the comparison result of the distance information and the corresponding standard value.
9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program which, when executed by a processor, implements a method of emotion recognition based on online education as claimed in any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor when executing the computer program implements a method of emotion recognition based on online education as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110080001.1A CN112784733A (en) | 2021-01-21 | 2021-01-21 | Emotion recognition method and device based on online education and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110080001.1A CN112784733A (en) | 2021-01-21 | 2021-01-21 | Emotion recognition method and device based on online education and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112784733A true CN112784733A (en) | 2021-05-11 |
Family
ID=75757684
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110080001.1A Pending CN112784733A (en) | 2021-01-21 | 2021-01-21 | Emotion recognition method and device based on online education and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112784733A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115937961A (en) * | 2023-03-02 | 2023-04-07 | 济南丽阳神州智能科技有限公司 | Online learning identification method and equipment |
| CN116682159A (en) * | 2023-06-07 | 2023-09-01 | 广东辉杰智能科技股份有限公司 | Automatic stereo recognition method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110084258A (en) * | 2018-02-12 | 2019-08-02 | 成都视观天下科技有限公司 | Face preferred method, equipment and storage medium based on video human face identification |
| CN110110593A (en) * | 2019-03-27 | 2019-08-09 | 广州杰赛科技股份有限公司 | Face Work attendance method, device, equipment and storage medium based on self study |
| CN110119673A (en) * | 2019-03-27 | 2019-08-13 | 广州杰赛科技股份有限公司 | Noninductive face Work attendance method, device, equipment and storage medium |
| CN110263633A (en) * | 2019-05-13 | 2019-09-20 | 广州烽火众智数字技术有限公司 | The personnel that are involved in drug traffic based on space time correlation detect method for early warning, system and storage medium |
| CN110399836A (en) * | 2019-07-25 | 2019-11-01 | 深圳智慧林网络科技有限公司 | User emotion recognition methods, device and computer readable storage medium |
| CN112183238A (en) * | 2020-09-10 | 2021-01-05 | 广州大学 | Method and system for detecting attention of remote education |
-
2021
- 2021-01-21 CN CN202110080001.1A patent/CN112784733A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110084258A (en) * | 2018-02-12 | 2019-08-02 | 成都视观天下科技有限公司 | Face preferred method, equipment and storage medium based on video human face identification |
| CN110110593A (en) * | 2019-03-27 | 2019-08-09 | 广州杰赛科技股份有限公司 | Face Work attendance method, device, equipment and storage medium based on self study |
| CN110119673A (en) * | 2019-03-27 | 2019-08-13 | 广州杰赛科技股份有限公司 | Noninductive face Work attendance method, device, equipment and storage medium |
| CN110263633A (en) * | 2019-05-13 | 2019-09-20 | 广州烽火众智数字技术有限公司 | The personnel that are involved in drug traffic based on space time correlation detect method for early warning, system and storage medium |
| CN110399836A (en) * | 2019-07-25 | 2019-11-01 | 深圳智慧林网络科技有限公司 | User emotion recognition methods, device and computer readable storage medium |
| CN112183238A (en) * | 2020-09-10 | 2021-01-05 | 广州大学 | Method and system for detecting attention of remote education |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115937961A (en) * | 2023-03-02 | 2023-04-07 | 济南丽阳神州智能科技有限公司 | Online learning identification method and equipment |
| CN116682159A (en) * | 2023-06-07 | 2023-09-01 | 广东辉杰智能科技股份有限公司 | Automatic stereo recognition method |
| CN116682159B (en) * | 2023-06-07 | 2024-02-02 | 广东辉杰智能科技股份有限公司 | Automatic stereo recognition method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12099577B2 (en) | Object recognition method and apparatus, electronic device, and readable storage medium | |
| EP3989104A1 (en) | Facial feature extraction model training method and apparatus, facial feature extraction method and apparatus, device, and storage medium | |
| CN109359539B (en) | Attention assessment method and device, terminal equipment and computer readable storage medium | |
| CN109345553B (en) | Palm and key point detection method and device thereof, and terminal equipment | |
| CN110674664A (en) | Visual attention recognition method and system, storage medium and processor | |
| EP3933708A2 (en) | Model training method, identification method, device, storage medium and program product | |
| CN113850238B (en) | Document detection method and device, electronic equipment and storage medium | |
| EP3879454A2 (en) | Method and apparatus for evaluating image relative definition, device and medium | |
| CN112101123B (en) | Attention detection method and device | |
| CN110298569B (en) | Learning evaluation method and device based on eye movement recognition | |
| CN112784733A (en) | Emotion recognition method and device based on online education and electronic equipment | |
| WO2022126917A1 (en) | Deep learning-based face image evaluation method and apparatus, device, and medium | |
| CN111666820A (en) | Speaking state recognition method and device, storage medium and terminal | |
| CN111986117A (en) | System and method for correcting arithmetic operation | |
| CN112087590A (en) | Image processing method, device, system and computer storage medium | |
| CN117992344A (en) | UI (user interface) automation test assertion statement generation method, device and equipment | |
| CN113283383A (en) | Live broadcast behavior recognition method, device, equipment and readable medium | |
| CN110119459A (en) | Image data retrieval method and image data retrieving apparatus | |
| CN116110058A (en) | Virtual human interaction method and system based on handwriting digital recognition | |
| CN113850239B (en) | Multi-document detection method and device, electronic equipment and storage medium | |
| CN113850805B (en) | Multi-document detection method and device, electronic equipment and storage medium | |
| CN115171042A (en) | Student classroom behavior identification method, device, terminal equipment and medium | |
| CN115019396A (en) | Learning state monitoring method, device, equipment and medium | |
| CN112232166A (en) | Artificial intelligence-based lecturer dynamic evaluation method and device, and computer equipment | |
| CN111507555A (en) | Human body state detection method, classroom teaching quality evaluation method and related device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |