TW202145131A - Video processing method and device, electronic equipment and storage medium - Google Patents

Abstract

The invention relates to a video processing method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a video, wherein at least part of video frames in the video comprise a target object; according to the video, detecting at least one kind of learning behaviors of the target object in the teaching course watching process; and when it is detected that the target object executes the at least one type of learning behavior, generating learning state information according to the video frame at least partially containing the at least one type of learning behavior and/or the duration of executing the at least one type of learning behavior by the target object.

Description

Video processing method and device, electronic device and computer-readable storage medium

The present invention claims the priority of the Chinese patent application with the application number of 202010442733.6 and the application title of "video processing method and device, electronic equipment and storage medium" submitted to the Chinese Patent Office on May 22, 2020, the entire contents of which are incorporated by reference in the present invention.

The present invention relates to the field of computer vision, and in particular, to a video processing method and device, an electronic device and a computer-readable storage medium.

In the teaching process, because teachers need to concentrate on teaching, it is difficult for institutions or teachers to grasp the status of students' lectures, and parents cannot understand their children's performance in school. Whether students are actually in class, whether they are listening carefully, and how well they interact in class cannot be quantitatively assessed.

Therefore, how to grasp the learning status of each student in the teaching process while ensuring the quality of teaching has become an extremely unsolved problem at present.

The present invention proposes a solution for video information processing.

According to an aspect of the present invention, a video processing method is provided, comprising:

Acquiring video information, wherein at least part of the video frames in the video information includes a target object; according to the video information, detect at least one type of learning behavior of the target object in the process of watching the teaching course; In the case of at least one type of learning behavior, the learning status message is generated according to a video frame at least partially including the at least one type of learning behavior and/or the duration of the target object performing the at least one type of learning behavior. According to an aspect of the present invention, a video processing apparatus is provided, comprising:

a video acquisition module for acquiring video, wherein at least some of the video frames in the video contain a target object;

a detection module, configured to detect at least one type of learning behavior of the target object in the process of watching the teaching course according to the video;

The generating module is used for, when it is detected that the target object performs at least one type of learning behavior, according to a video frame that at least partially contains the at least one type of learning behavior and/or the target object performs the at least one type of learning behavior. Duration to generate learning status messages.

According to an aspect of the present invention, an electronic device is provided, comprising:

a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the above video processing method.

According to an aspect of the present invention, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned video processing method is implemented.

According to an aspect of the present invention, a computer program is provided, including computer-readable codes, when the computer-readable codes are executed in an electronic device, the processor in the electronic device executes the above-mentioned video processing method.

In the embodiment of the present invention, when it is detected that the target object has at least one type of learning behavior, a video frame containing the learning behavior can be used to generate an intuitive learning state message, and a quantified learning state can be generated according to the duration of the learning behavior. The above methods can flexibly obtain learning status information with evaluation value, which is convenient for teachers or parents and other relevant personnel and institutions to effectively and accurately grasp the learning status of students.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. Other features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the drawings.

Various exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the drawings. The same reference numerals in the figures denote elements with the same or similar function. Although various aspects of the embodiments are shown in the drawings, the drawings are not necessarily to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only an association relationship to describe associated objects, indicating that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three situations. In addition, the term "at least one" herein refers to any combination of any one of a plurality or at least two of a plurality, for example, including at least one of A, B, and C, and may mean including those composed of A, B, and C. Any one or more elements selected in the collection.

In addition, in order to better illustrate the present invention, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present invention may be practiced without certain specific details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present invention.

FIG. 1 shows a flowchart of a video processing method according to an embodiment of the present invention. The method can be applied to a video processing apparatus, and the video processing apparatus can be a terminal device, a server, or other processing equipment. The terminal device may be User Equipment (UE), mobile device, user terminal, terminal, mobile phone, wireless phone, Personal Digital Assistant (PDA), handheld device, computing device, in-vehicle device, mobile phone, etc. wearable devices, etc. In one example, the data processing method can be applied to a cloud server or a local server, and the cloud server can be a public cloud server or a private cloud server, which can be flexibly selected according to actual conditions.

In some possible implementations, the video processing method can also be implemented by the processor calling computer-readable instructions stored in the memory.

As shown in FIG. 1, in a possible implementation manner, the video processing method may include:

Step S11 , acquiring video information, wherein at least some of the video frames in the video information include the target object.

Step S12, according to the video, detect at least one type of learning behavior of the target object in the process of watching the teaching course.

Step S13, when it is detected that the target object performs at least one type of learning behavior, generate learning status information according to at least part of the video frames including at least one type of learning behavior and/or the duration of the target object performing at least one type of learning behavior.

Among them, the target object can be any object whose learning status information is obtained, that is, an object with learning status evaluation requirements, and its specific implementation form can be flexibly determined according to the actual situation. In a possible implementation, the target object can be students, such as primary school students, middle school students or college students; Seniors studying in senior colleges, etc.

In this embodiment of the present invention, the video may be the video recorded by the target object in the course of watching the teaching course, wherein the implementation form of the teaching course is not limited, and may be a pre-recorded course video, a live course or a teacher's live course Courses taught, etc.; at least part of the video frames in the video may contain the target object, that is, the appearance of the target object in the recorded video can be flexibly determined according to the actual situation. In a possible implementation manner, the target object may be in the video all the time, and in a possible implementation manner, the target object may also not appear in the video frame at certain times or in certain time periods.

The scene in which the target object watches the teaching course can be flexibly determined according to the actual situation. In a possible implementation, this scene can be an online scene, that is, the target object watches the teaching course through online education methods such as online classrooms, etc.; In the implementation method, this scene can also be an offline scene, that is, the target object watches the teaching course taught by the teacher on the spot through the traditional face-to-face teaching method, or the target object watches in a specific teaching place such as a classroom through video or other teaching methods. Teaching courses played in the form of multimedia, etc.

The specific implementation form of the video message can be flexibly determined according to the application scenario of the video message processing method. In a possible implementation manner, the video may be real-time video, such as the real-time video recorded by the target object in the process of learning in the online class, or the video captured by the target object through the camera deployed in the classroom when the target object is in the classroom. real-time video, etc.; in a possible implementation, the video can also be recorded video, for example, after the target object learns in the online classroom, the recorded playback video of the target object’s learning, or the target object is in the classroom after class, through deployment The complete classroom learning video captured by the camera in the classroom, etc.

For ease of description, the following embodiments of the invention take the video being recorded in real time by the target object in the process of learning in an online classroom as an example to describe the video processing process. Video processing procedures in other application scenarios can be flexibly expanded with reference to subsequent embodiments of the invention, and details are not described herein again.

After acquiring the video according to the above-mentioned embodiments of the invention in step S11, in step S12, at least one type of learning behavior of the target object in the process of watching the teaching course can be detected. The type and quantity of the detected learning behaviors can be flexibly determined according to the actual situation, and are not limited to the following invention embodiments. In a possible implementation manner, the learning behavior performed by the target object may include at least one of the following behaviors: for example, performing at least one target gesture, expressing target emotion, paying attention to the display area of the teaching course, generating at least one interactive behavior with other objects, Absence in at least some of the video frames in the video, eyes closed, eye contact in the presentation area of the teaching session, etc.

Among them, the target gesture may be some preset gestures that may be generated by the target object in the process of watching the teaching course, and the specific implementation form can be flexibly set according to the actual situation. For details, please refer to the subsequent embodiments of the invention, which will not be described here. Expand.

The target emotion can be some emotions that the target object reflects on the real feeling of the teaching course in the process of watching the teaching course, and its specific realization form can also be flexibly set according to the actual situation, and will not be expanded here.

Focusing on the display area of the teaching course can reflect the attention of the target object in the process of watching the teaching course, wherein the specific area range of the display area can be flexibly set according to the actual situation, and is not limited to the following invention embodiments. In a possible implementation manner, the display area may be the display area of the teaching course video in the online classroom. For example, during the process of online learning by terminal devices such as computers, mobile phones, or tablet computers, the display area may be the display area of these terminal devices. The screen that plays the teaching course, etc.; in a possible implementation, the display area can be the teaching area of the teacher in the offline classroom, such as the podium or the blackboard in the classroom.

At least one interactive behavior with other objects can be the learning-related interaction generated by the target object with other objects related to the teaching course during the course of watching the teaching course. The realization form of other objects can be flexibly determined according to the actual situation. In a possible implementation manner, other objects may be teaching objects, such as teachers, etc. In a possible implementation manner, other objects may also be learning objects other than the target object in the teaching process, such as the target object's classmates, etc.; The interaction behavior with other objects can change flexibly according to different objects. In a possible implementation, when the other objects are teachers, the interaction with other objects can include receiving rewards sent by teachers, such as In a possible implementation, when the other object is the teacher, the interaction with other objects may include answering the teacher's question or speaking according to the teacher's call, etc. , in a possible implementation manner, in the case that other objects are classmates, the interaction with other objects may include group mutual assistance, group discussion or group study and so on.

Not appearing in at least part of the video frames in the video may be that the learning object has left the teaching course at certain moments or certain time periods. For example, during the online learning process of the target object, he may temporarily leave the current online for personal reasons. Learning equipment, or out of the shooting range of the current online learning equipment, etc.

Eye-closing can be the eye-closing operation performed by the target object in the process of watching the teaching course. The eye contact in the display area of the teaching course can be the display area of watching the teaching course. Correspondingly, according to the target object in the video The situation of eye contact in the display area of the course, the situation of the target object not watching the display area of the teaching course, etc. can be further determined.

Through the various learning behaviors mentioned in the above embodiments of the invention, comprehensive and flexible behavior detection can be performed on the learning process of the target object, so as to improve the comprehensiveness and accuracy of the learning status information obtained according to the detection, and to be more flexible and accurate. Grasp the learning status of the target object.

Specifically, in step S12, which type or which types of detections are performed on the various learning behaviors in the above embodiments of the invention can be flexibly set according to the actual situation. In a possible implementation manner, the various learning behaviors mentioned in the above embodiments of the invention may be detected at the same time, and the specific detection methods and processes can be found in the following embodiments of the invention, which will not be expanded here.

When it is detected that the target object performs at least one type of learning behavior, the learning status message may be generated according to a video frame at least partially including the at least one type of learning behavior and/or the duration of the target object performing at least one type of learning behavior. The specific implementation form of the learning status information can be flexibly determined according to the type of learning behavior and the corresponding operation. In a possible implementation, when the learning state information is generated according to at least part of the video frames containing at least one type of learning behavior, the learning state information may include information composed of video frames; in a possible implementation, in the When the target object performs at least one type of learning behavior for the duration, the learning status message may be a digital data message; in a possible implementation manner, the learning status message may also include both video frame messages and data messages. One form of information; in a possible implementation, the learning state information may also include information of other states, and so on. Specifically, for how to generate the learning status information and the implementation form of the learning status information, reference may be made to subsequent embodiments of the invention, which will not be described here.

In the embodiment of the present invention, when it is detected that the target object has at least one type of learning behavior, a video frame containing the learning behavior can be used to generate an intuitive learning state message, and a quantified learning state can be generated according to the duration of the learning behavior. The above methods can flexibly obtain learning status information with evaluation value, which is convenient for teachers or parents and other relevant personnel and institutions to effectively and accurately grasp the learning status of students.

It has been mentioned in the above-mentioned embodiments of the invention that the video information can be the video information recorded by the target object in the process of watching the teaching course, and the scene in which the target object watches the teaching course can be flexibly determined according to the actual situation. Therefore, correspondingly, the video information is obtained in step S11. The method can also be flexibly changed according to different scenarios. In a possible implementation manner, in the case that the scene where the target object watches the teaching course is an online scene, that is, in the case that the target object can watch the teaching course through the online classroom, the method of acquiring the video information may include: if the video information processing device is connected to the target object The equipment for the online learning of the target object is the same device, and the video capture process of the target object watching the teaching course can be collected through the equipment of the target object for online learning; if the video processing device and the equipment for the online learning of the target object are different devices, the device can The device for online learning by the target object collects video information in the process of the target object watching the teaching course, and transmits it to the video processing device in real time and/or in non-real time. In a possible implementation manner, when the scene where the target object watches the teaching course is an offline scene, that is, when the target object participates in face-to-face teaching or watches the video of the teaching course in a specific teaching scene, the way to obtain the video can be as follows: Including: collecting the video of the target object by deploying offline image acquisition equipment (such as ordinary cameras, shooting devices deployed in response to security requirements, etc.). Further, if the image capture device deployed offline can perform video processing, that is, can be used as a video processing device, then the process of acquiring video in step S11 has been completed; if the offline image capture device can not perform video processing, then The video captured by the image capturing device deployed offline can be transmitted to the video processing device in real time and/or non-real time.

As described in the above embodiments of the invention, the manner of detecting the learning behavior of the target object in step S12 can be flexibly determined according to the actual situation. In a possible implementation manner, step S12 may include:

Step S121, performing target object detection on the video to obtain a video frame including the target object.

Step S122, performing at least one type of learning behavior detection on the video frame containing the target object.

It can be seen from the above embodiments of the invention that, in a possible implementation manner, a video frame including a target object in the video may be determined by performing target object detection on the video. After determining which video frames contain the target object, at least one type of learning behavior detection may be performed on the target objects in the video frames containing the target object.

The method for detecting the target object can be flexibly determined according to the actual situation, and is not limited to the following embodiments. In a possible implementation manner, the target object in the video may be detected by means of face detection or face tracking. In a possible implementation manner, after the video frame is detected by means of face detection or face tracking, multiple objects may be detected. For example, screening is performed, and one or more objects are selected as target objects. The specific screening method can be flexibly set according to the actual situation, which is not limited in the embodiment of the present invention.

In a possible implementation manner, after the video frame containing the target object is obtained, at least one type of learning behavior detection may be performed on the video frame containing the target object through step S122. The implementation manner of step S122 can be flexibly changed according to different learning behaviors. For details, please refer to the following embodiments of the invention, which will not be expanded here. In the case of needing to detect multiple types of learning behaviors of the target object, a combination of multiple methods can be used to realize the detection of multiple types of learning behaviors at the same time.

In some possible implementation manners, after the target object detection is performed on the video, the learning behavior detection of the target object in the process of watching the teaching course can be completed. That is, it can be determined that the learning behavior mentioned in the above embodiments of the invention does not occur in at least part of the video frames in the video by performing target object detection on the video. and further according to the video frames in which the target object is not detected, the learning status information is obtained, or the time when the target object does not appear in at least part of the video frames in the video is counted as the learning status information according to the video frames in which the target object is not detected. .

In the embodiment of the present invention, by performing target object detection on the video, a video frame containing the target object is obtained, and at least one type of learning behavior detection is performed on the video frame containing the target object. Through the above process, the target object in the video can be used. Detection, more targeted detection of at least one type of learning behavior of the target object, so that the learning behavior detection is more accurate, and the accuracy and reliability of the subsequent learning status information are further improved.

As described in the above embodiments of the invention, the implementation of step S122 can be flexibly changed according to different learning behaviors. In a possible implementation manner, the learning behavior may include: performing at least one target gesture;

In this case, performing at least one class of learned behavior detection on video frames containing target objects may include:

Detecting at least one target gesture on the video frame containing the target object;

When it is detected that the number of consecutive video frames containing at least one target gesture exceeds the first threshold, recording at least one frame in the video frames containing the target gesture as a gesture start frame;

In the video frames after the gesture start frame, when the number of consecutive video frames not including the target gesture exceeds the second threshold, recording at least one frame in the video frames not including the target gesture as the gesture end frame;

The number of times and/or the time that the target object in the video performs at least one target gesture is determined according to the number of gesture start frames and gesture end frames.

It can be seen from the above embodiments of the invention that in the case that the learning behavior includes performing at least one target gesture, the learning behavior detection performed on the video frame of the target object may include target gesture detection.

The gestures specifically included in the target gesture can be flexibly set according to the actual situation, and are not limited to the following inventive embodiments. Exemplarily, the target gesture includes one or more of a raise hand gesture, a like gesture, an OK gesture, and a victory gesture.

In a possible implementation manner, the target gesture may include learning-related gestures reflected by the target object according to the listening situation during the course of watching the teaching course, such as a hand-raising gesture for answering a question, a gesture of raising a hand for answering a question, responding to the teaching content or the teacher Like gestures to express appreciation (thumbs up, etc.), OK gestures to express understanding or approval of the teaching content, and victory gestures (such as Yeah gestures, etc.) to interact with the teacher.

Specifically, the manner of detecting at least one target gesture on the video frame including the target object can be flexibly determined according to the actual situation, and is not limited to the following inventive embodiments. In a possible implementation manner, the detection of the target gesture can be realized by the relevant algorithm of gesture recognition, for example, the key points of the hand of the target object in the video frame or the image area corresponding to the hand detection frame can be recognized, and the key points of the hand Gesture detection is performed on the image area corresponding to the point or the hand detection frame, and based on the gesture detection result, it is determined whether the target object is performing the target gesture. In a possible implementation manner, the detection of the target gesture can be achieved through a neural network with a gesture detection function. The specific structure and implementation method of the neural network with gesture detection function can be flexibly set according to the actual situation. A neural network that can detect multiple gestures at the same time to detect target gestures; in a possible implementation, video frames containing target objects can also be input to multiple neural networks with a single gesture detection function. , to realize the detection of multiple target gestures.

In the process of detecting the target gesture by any of the above-mentioned embodiments of the invention, if the number of consecutive video frames containing at least one target gesture is detected to exceed the first threshold, the target gesture may be selected from the continuous video frames containing the target gesture. At least one frame is the gesture start frame. The number of the first thresholds can be flexibly set according to the actual situation, and the number of the first thresholds corresponding to different target gestures can be the same or different. The corresponding first threshold is set to 7, then in the case where the number of consecutive video frames containing the hand-raising gesture is detected not less than 6, at least one frame can be selected from the video frames containing the hand-raising gesture as the gesture of the hand-raising gesture. The start frame, when the number of consecutive video frames in which the like gesture is detected is not less than 7, at least one frame may be selected from the video frames containing the like gesture as the gesture start frame of the like gesture. In a possible implementation manner, in order to facilitate the detection of target gestures, the first thresholds corresponding to different target gestures may be set to the same value, and in an example, the number of first thresholds may be set to 6.

The selection method of the gesture start frame can also be flexibly set according to the actual situation. In a possible implementation, the first frame of the detected continuous video frames containing the target gesture can be used as the gesture start frame of the target gesture. In a possible implementation manner, in order to reduce the error of gesture detection, a frame after the first frame in the detected continuous video frames containing the target gesture may also be used as the gesture start frame of the target gesture.

After the gesture start frame is determined, the gesture end frame may be determined from the video frame after the gesture start frame, that is, the end time of the target gesture in the gesture start frame is determined. The specific determination manner can be flexibly selected according to the actual situation, and is not limited to the following inventive embodiments. In a possible implementation manner, in the video frame after the gesture start frame is detected, if the number of consecutive video frames that do not include the target gesture in the gesture start frame exceeds the second threshold, the target gesture will not be included in the case. At least one of the consecutive video frames is recorded as the gesture end frame. The value of the second threshold can also be flexibly set according to the actual situation, and the values of the second threshold corresponding to different target gestures can be the same or different. The specific setting method can refer to the first threshold, which will not be repeated here. In an example, the value of the second threshold corresponding to different target gestures may be the same, for example, it may be set to 10, that is, after the gesture start frame, if it is detected that 10 consecutive frames do not include the target gesture in the gesture start frame, you can The target object is considered to have finished performing the target gesture. In this case, at least one frame can be selected as the gesture end frame from the continuous video frames that do not contain the target gesture, and the selection method can also refer to the gesture start frame. In an example, the continuous video frame that does not contain the target gesture can be The last frame in the frames is used as the gesture end frame; in an example, a frame before the last frame in the continuous video frames that do not contain the target gesture can also be used as the gesture end frame. In a possible implementation, if after the gesture start frame is detected, there is a video frame or frames that do not contain the target object, then one or some video frames that do not contain the target object can also be used as Gesture end frame.

After the gesture start frame and the gesture end frame are determined, the number of gesture start frames and gesture end frames included in the video frame can be used to determine the number of times the target object performs one or some target gestures, and further, it can be determined Duration of performing a certain or certain target gesture, etc. Which content related to the target gesture is specifically determined can be flexibly determined according to the requirement of learning the status information in step S13 . For details, please refer to the subsequent embodiments of the invention, which will not be expanded here.

By detecting at least one target gesture on the video frame containing the target object, and determining the gesture start frame and the gesture end frame according to the detection situation, the number of times and/or the time that the target object in the video performs at least one target gesture is further determined. The process can comprehensively and accurately detect the gestures fed back by the target object in the video according to the learning state, thereby improving the comprehensiveness and accuracy of the subsequent learning state information, and then accurately grasping the learning state of the target object.

In one possible implementation, the learning behavior may include: expressing the target emotion;

In this case, performing at least one class of learned behavior detection on video frames containing target objects may include:

Perform expression detection and/or smile detection on video frames containing target objects;

In the case where it is detected that the target object in the video frame shows at least one first target expression or the smile value detection result exceeds the target smile value, the detected video frame is used as the first detection frame;

In a case where it is detected that the number of consecutive first detection frames exceeds the third threshold, it is determined that the target object produces the target emotion.

The target emotion may be any emotion set according to actual needs, for example, it may be a happy emotion indicating that the target object is focusing on learning, or an annoying emotion indicating that the target object is in a poor learning state. The following embodiments of the invention are described by taking the target emotion as a happy emotion as an example, and the case where the target emotion is other emotions may be expanded accordingly with reference to subsequent invention embodiments.

It can be seen from the above embodiments of the invention that, in the case where the learning behavior includes expressing the target emotion, the detection of the learning behavior of the target object can be realized through expression detection and/or smile value detection. In a possible implementation, the detection of the learning behavior of expressing the target emotion can be realized only through expression detection or smile value detection. In a possible implementation, the target can be jointly determined through expression detection and smile value detection. Whether the subject exhibits the target emotion. Subsequent embodiments of the invention are described by jointly determining whether the target object expresses the target emotion through expression detection and smile value detection.

The expression detection may include detecting the expression displayed by the target object, such as detecting which expression the target object displays. The specific expression division can be flexibly set according to the actual situation. In a possible implementation, the expression can be divided into happy , calmness, etc.; while the smile value detection can include detecting the smile intensity of the target object, such as how much the target object smiles, and the result of the smile value detection can be fed back through numerical values, for example, the smile value can be detected. The result is set to be between [0, 100], the higher the value, the higher the intensity or amplitude of the smile of the target object. The specific expression detection and smile value detection methods can be flexibly determined according to the actual situation. Any method that can detect the expression or smile degree of the target object can be used as a corresponding detection method, and is not limited to the following invention embodiments. In a possible implementation manner, the facial expression detection of the target object can be realized through an expression recognition neural network, and in a possible implementation manner, the smile value detection of the target object can be realized through a smile value detection neural network. Specifically, the structures and implementations of the facial expression recognition neural network and the smile value detection neural network are not limited in the embodiments of the present invention. Any neural network that can realize the expression recognition function through training and any neural network that can realize the smile value detection function through training Any neural network can be applied to the embodiments of the present invention. In a possible implementation manner, expression detection and smile value detection can also be realized by detecting the key points of the face and the key points of the mouth of the target object in the video.

Specifically, in the case of the detection result achieved by the expression detection and the smile value detection, it is determined that the target object generates the target emotion, and the realization method can be flexibly set according to the actual situation. In a possible implementation, it can be considered that the target object in the video frame is considered to be the target object in the video frame when it is detected that the target object shows at least one first target expression, or the result of the smile value detection exceeds the target smile value. The target emotion is expressed, in this case, the video frame can be used as the first detection frame. The specific expression type of the first target expression can be flexibly set according to the actual situation, and is not limited to the following inventive embodiments. In a possible implementation manner, happiness may be used as the first target expression, that is, all video frames in which the expression of the detected target object is happy may be used as the first detection frame. In a possible implementation manner, both happy and calm may be used as the first target expressions, that is, video frames in which the detected expressions of the target object are happy or calm may be used as the first detection frames. Similarly, the specific value of the target smile value can also be flexibly set according to the actual situation, which is not specifically limited here. Therefore, in a possible implementation manner, the video frame in which the detection result of the smile value exceeds the target smile value may also be used as the first detection frame.

In a possible implementation manner, it may be determined that the target object generates the target emotion when a certain video frame is detected as the first detection frame. In a possible implementation manner, in order to improve the detection accuracy and reduce the influence of the detection error on the learning behavior detection result, the target object may be determined when the number of consecutive first detection frames exceeds the third threshold. Generate a target emotion. Wherein, a video frame sequence in which each frame in the continuous video frames is the first detection frame may be regarded as the continuous first detection frame. The number of the third thresholds can be flexibly set according to the actual situation, and its value can be the same as the first threshold or the second threshold, or it can be different. When the 6 frames are all the first detection frames, it can be considered that the target object produces the target emotion.

Further, after it is determined that the target object generates the target emotion, one frame can also be selected from the consecutive first detection frames as the target emotion start frame, and then after the target emotion start frame, the expression of the target object is not detected for 10 consecutive frames. The first target expression, or the detection result of the smile value of the target object in 10 consecutive frames does not exceed the third threshold, or when the target object cannot be detected in a certain frame or several frames, the end frame of the target emotion can be further determined, and then The number of times and/or the time that the target object generates the target emotion is determined according to the target emotion start frame or the target emotion end frame, and the specific process can refer to the corresponding process of the target gesture, which will not be repeated here.

By performing expression detection and/or smile value detection on the video frame containing the target object, and determining the first detection frame according to the results of the expression detection and smile value detection, the number of consecutive first detection frames detected exceeds the number of the first detection frame. In the case of three thresholds, it is determined that the target object produces the target emotion. Through the above process, the emotion of the target object in the learning process can be flexibly determined based on the target object's expression and degree of smile, so that the target object's learning process can be more comprehensively and accurately perceived. Emotional state during the process to generate more accurate learning state information.

In a possible implementation manner, the learning behavior may include: paying attention to the display area of the teaching course;

In this case, performing at least one class of learned behavior detection on video frames containing target objects may include:

Perform expression detection and face angle detection on video frames containing target objects;

When it is detected that the target object in the video frame shows at least one second target expression and the face angle is within the range of the target face angle, the detected video frame is used as the second detection frame;

When it is detected that the number of consecutive second detection frames exceeds the fourth threshold, it is determined that the target object pays attention to the display area of the teaching course.

The implementation form of the display area of the teaching course may refer to the above-mentioned embodiments of the invention, which will not be repeated here.

It can be seen from the above embodiments of the invention that in the case where the learning behavior includes paying attention to the display area of the teaching course, the learning behavior detection of the target object can be realized through expression detection and face angle detection. In a possible implementation manner, the detection of the learning behavior of paying attention to the display area of the teaching course can also be realized only by detecting the face angle. Subsequent embodiments of the invention are described by using expression detection and face angle detection to determine whether the target object pays attention to the display area of the teaching course as an example, and other implementation methods can be expanded correspondingly with reference to the subsequent invention embodiments, which will not be repeated here. .

The implementation of expression detection may refer to the above embodiments of the invention, which will not be repeated here; the face angle detection may be to detect the orientation angle of the face and the like. The specific face angle detection method can be flexibly determined according to the actual situation. Any method that can detect the face angle of the target object can be used as the detection method for face angle detection, and is not limited to the following invention embodiments. In a possible implementation manner, the face angle detection of the target object can be realized through a face angle detection neural network. Specifically, the structure and implementation manner of the face angle detection neural network are not limited in the embodiments of the present invention, and any neural network that can realize the function of face angle detection through training can be applied to the embodiments of the present invention. In a possible implementation manner, the face angle of the target object may also be determined by detecting key points of the face of the target object in the video. The form of the angle of the face that can be detected by face angle detection can also be flexibly determined according to the actual situation. In a possible implementation, the target can be determined by detecting the yaw angle and pitch angle of the face of the target object. The face angle of the object.

Specifically, in the case of the detection result of expression detection and face angle detection, it is determined that the target object pays attention to the display area of the teaching course, and the implementation method can be flexibly set according to the actual situation. In a possible implementation, it may be considered that in the case where the target object in the video frame is detected to display at least one second target expression, and the detected face angle is within the range of the target face angle, the video frame is considered to be in the The target object of , pays attention to the display area of the teaching course, in this case, the video frame can be used as the second detection frame. The specific expression type of the second target expression can be flexibly set according to the actual situation, which may be the same as the first target expression mentioned in the above-mentioned embodiment of the invention, or may be different from the first target expression mentioned in the above-mentioned embodiment of the invention. It is not limited to the following inventive examples. In a possible implementation manner, calmness may be used as the second target expression, that is, the video frames whose expression of the detected target object is calm and the face angle is within the target face angle range may be used as the second detection frame. In a possible implementation, all expressions other than other expressions can be used as the second target expressions, that is, the detected face angle of the target object can be within the range of the target face angle, and the expressions are not “other” video frames , both as the second detection frame. Similarly, the specific range value of the target face angle range can also be flexibly set according to the actual situation, which is not specifically limited here. In a possible implementation manner, the target face angle range may be static. In an example, the overall position that the teacher may move to during teaching (such as the podium area where the teacher is in the offline scene, etc.) may be used as the The target face angle range; in one example, the fixed area that the target object may pay attention to during the course of watching the teaching course (such as the display screen that the target object pays attention to in the online scene, etc.) can be used as the target face angle range. In a possible implementation manner, the target face angle range may also be dynamic. In an example, the target face angle range may be flexibly determined according to the current position of the teacher's movement during the teaching, that is, the target face angle range may be flexibly determined as the teacher moves , to dynamically change the value of the target face angle range.

In a possible implementation manner, it may be determined that the target object pays attention to the display area of the teaching course when a certain video frame is detected as the second detection frame. In a possible implementation manner, in order to improve the detection accuracy and reduce the influence of the detection error on the learning behavior detection result, the target object may be determined when the number of consecutive second detection frames detected exceeds the fourth threshold. Pay attention to the display area of the teaching course. Wherein, a video frame sequence in which each frame of the continuous video frames is a second detection frame may be regarded as a continuous second detection frame. The number of the fourth threshold may be a number flexibly set according to the actual situation, and its value may be the same as or different from the first threshold, the second threshold or the third threshold. In an example, the number of the fourth threshold may be 6, That is, when it is detected that 6 consecutive frames are all second detection frames, it can be considered that the target object pays attention to the display area of the teaching course.

Further, after it is determined that the target object pays attention to the display area of the teaching course, one frame can be selected from the second consecutive detection frames as the attention start frame, and after the attention start frame, the expression of the target object is not detected for 10 consecutive frames. It is the second target expression, or the face angle of the target object in 10 consecutive frames is not within the range of the target face angle, or when the target object cannot be detected in a certain frame or several frames, the attention end frame can be further determined. Then, the number of times and/or the time that the target object pays attention to the teaching course display area is determined according to the attention start frame or the attention end frame. The specific process can refer to the corresponding process of the target gesture and target emotion, which will not be repeated here.

By performing expression detection and face angle detection on the video frame containing the target object, and determining the second detection frame according to the results of the expression detection and the face angle detection, so that the number of consecutive second detection frames detected exceeds the number of the first detection frame. In the case of four thresholds, it is determined that the target object pays attention to the display area of the teaching course. Through the above process, it can be flexibly determined whether the target object pays attention to the display area of the teaching course based on the expression and face angle of the target object, so as to be more comprehensive and accurate. Perceive the concentration of the target object in the learning process, and generate more accurate learning status information.

In a possible implementation manner, the learning behavior may further include: generating at least one interactive behavior with other objects. For the implementation manner of the interactive behavior, reference may be made to the foregoing embodiments of the invention, and details are not described herein again. In this case, the method of detecting the interactive behavior of the video frame containing the target object can be flexibly determined according to the actual situation. In the case of sending a small red flower, or speaking according to the teacher's online class roll call, the detection method of the interactive behavior can be directly based on the signals transmitted by other objects to determine whether the target object produces interactive behavior. In a possible implementation, if the interactive behavior is offline, for example, when the target object is called by the teacher to speak in the classroom, the method of detecting whether the target object has interactive behavior may include: The target action can be identified to determine whether the target object has interactive behavior. The target action can be flexibly set according to the actual situation of the interactive behavior. For example, the target action can include standing up and speaking, or the face is facing other objects and the speaking time exceeds a certain time value, etc.

In a possible implementation manner, the learning behavior may also include not appearing in at least part of the video frames in the video, in this case, step S12 may include:

Performing target object detection on the video, obtaining a video frame containing the target object, and using the video frame other than the video frame containing the target object in the video as the video frame in which the target object is not detected;

When the number of video frames in which the target object is not detected exceeds the preset number of video frames, detecting the learning behavior includes: not appearing in at least part of the video frames in the video.

The method of performing target object detection on video information is detailed in the foregoing embodiments of the invention, and details are not described herein again. In a possible implementation manner, in addition to the video frames containing the target object, each video frame in the video may also have video frames that do not contain the target object, so these video frames that do not contain the target object may be regarded as the undetected target. The object's video frames, and if the number of video frames of the target object is not detected exceeds the preset number of video frames, it is confirmed that the learning behavior of "not appearing in at least some of the video frames in the video" is detected. The preset number of video frames can be flexibly set according to the actual situation. In a possible implementation, the preset number of video frames can be set to 0, that is, when the video contains video frames for which the target object is not detected, it is recognized that the target object is not detected. In order to detect that this learning behavior does not occur in at least some of the video frames in the video, in a possible implementation manner, the preset number of video frames can also be a number greater than 0, and the specific setting can be flexibly determined according to the actual situation.

In a possible implementation manner, the learning behavior may also include closing eyes. In this case, the learning behavior detection method may be eye closing detection. The specific process of eye closing detection may be flexibly set according to the actual situation. It is realized by the neural network of the closed eye detection function. In an example, it is also possible to determine whether the target object has closed eyes by detecting key points in the eye and the eyeball. For example, when the key point in the eyeball is detected, determine The eyes of the target object are open; when only the key points of the eyes are detected, and the key points in the eyeball are not detected, the eyes of the target object are determined to be closed. In a possible implementation manner, the learning behavior may also include eye contact in the display area of the teaching course. In this case, the learning behavior detection method may refer to the process of paying attention to the display area of the teaching course in the above-mentioned embodiment of the invention , the specific detection method can be changed flexibly. For example, the target object can be detected with eyes closed and face angle at the same time, and the video frame with the face angle within the target face angle range and no closed eyes can be used as the third detection frame, and then in the first detection frame. When the number of three detection frames exceeds a certain set threshold, it is determined that the target object is making eye contact in the display area of the teaching course.

After the detection of at least one type of learning behavior of the target object is achieved through any combination of the various implementation manners of the above embodiments of the invention, the learning status information may be generated by step S13 when it is detected that the target object performs at least one type of learning behavior . The specific implementation manner of step S13 is not limited, and can be flexibly changed according to the actual situation of the detected learning behavior, and is not limited to the following inventive embodiments.

It can be seen from the actual content of step S13 in the above embodiment of the invention that in the process of generating the learning status information in step S13, there may be the following generation methods. For example, the learning status information can be generated according to video frames containing at least one type of learning behavior. ; or generate learning status information according to the duration of at least one type of learning behavior performed by the target object; or combine the above two situations, generate a part of learning status information according to the video frame containing at least one type of learning behavior, and according to the target object. The duration for which the subject performs at least one type of learning behavior to generate another type of learning status information. In the case where the learning status information can be generated according to the video frame of the learning behavior, and the learning status information can be generated according to the duration of the target object performing at least one type of learning behavior, which kind of learning status information is generated according to which type of learning behavior. , the mapping method can be flexibly set according to the actual situation. In a possible implementation, some positive learning behaviors may correspond to the process of generating learning status information according to video frames containing the learning behaviors, such as performing at least one target gesture on the target object, showing a positive target emotion, In the case of paying attention to the display area of the teaching course and generating at least one interactive behavior with other objects, the learning status message can be generated according to the video frame containing the above learning behavior; in a possible implementation, some negative learning behavior, such as the target object does not appear in at least part of the video frame in the video, closes eyes, or does not make eye contact in the display area of the teaching course, etc., the learning status message can be generated according to the duration of the above learning behavior.

In a possible implementation manner, the learning status information is generated according to at least part of the video frames containing at least one type of learning behavior, which may include:

Step S1311, acquiring video frames containing at least one type of learning behavior in the video, as a set of target video frames;

Step S1312, performing face quality detection on at least one video frame in the target video frame set, and taking a video frame with a face quality greater than a face quality threshold as the target video frame;

Step S1313, generating a learning status message according to the target video frame.

The video frame containing at least one type of learning behavior may be a video frame in which the target object is detected to perform at least one type of behavior in the process of learning behavior detection, such as the first detection frame, the second detection frame mentioned in the above embodiment of the invention The detection frame and the third detection frame, etc., or the video frame containing the target gesture between the gesture start frame and the gesture end frame, etc.

After determining the video frames containing at least one type of learning behavior, how to obtain the target video frame set can be flexibly determined. In a possible implementation manner, each video frame containing each type of learning behavior can be obtained separately according to the type of learning behavior, so as to form a target video frame set for each type of learning behavior; in a possible implementation manner, it is also possible to According to the type of learning behavior, the partial frames containing each type of learning behavior are obtained, and then based on the partial frames of each type of learning behavior, the target video frame set of this type of learning behavior is obtained, and the selection method of which partial frames are selected can be flexible. Decide.

After the target video frame set corresponding to the learning behavior is obtained, step S1312 may be used to select and obtain the target video frame from the target video frame set. It can be seen from step S1312 that, in a possible implementation manner, the video frames in the target video frame set can be subjected to face quality detection, and then the video frames whose face quality is greater than the face quality threshold are used as target video frames.

The detection method of the face quality can be flexibly set according to the actual situation, and is not limited to the following invention embodiments. In a possible implementation, the video frame can be determined by performing face recognition on the face in the video frame. The integrity of the face in the video frame is used to determine the face quality; in a possible implementation, the face quality can also be determined based on the clarity of the face in the video frame; in a possible implementation, it can also be based on the video frame. The integrity, clarity, brightness and other parameters of the face are used to comprehensively judge the quality of the face in the video frame; in a possible implementation, it can also be obtained by inputting the video frame into the face quality neural network. The face quality in the video frame, the face quality neural network can be obtained by training a large number of face pictures containing the face quality score and annotation, and its specific implementation form can be flexibly selected according to the actual situation, which is not limited in the embodiment of the present invention. .

The specific value of the face quality threshold can be flexibly determined according to the actual situation, which is not limited in this embodiment of the present invention. In a possible implementation, different face quality thresholds may be set for each type of learning behavior; in a possible implementation, the same face threshold may also be set for each type of learning behavior. In a possible implementation, the face quality threshold can also be set as the maximum face quality in the target video frame set. In this case, the video with the highest face quality under each type of learning behavior can be directly set. frame as the target video frame.

In some possible implementations, there may be some video frames that contain multiple types of learning behaviors at the same time. In this case, the manner of processing the video frames containing multiple types of learning behaviors can be flexibly changed according to the actual situation. In a possible implementation manner, these video frames can be respectively attributed to each type of learning behavior, and then selected from the video frame set corresponding to each type of learning behavior according to step S1312 to obtain the target video frame; in a possible In an implementation manner, a video frame containing multiple types of learning behaviors at the same time can also be directly selected as the target video frame.

After the target video frame is determined through any of the above embodiments, step S1313 may be used to generate a learning status message according to the target video frame. The implementation manner of step S1313 can be flexibly selected according to the actual situation. For details, please refer to the following embodiments of the invention, which will not be expanded here.

In the embodiment of the present invention, a video frame containing at least one type of learning behavior in the video frame is obtained as a target video frame set, so that according to the target video frame set of each type of learning behavior, a video frame with higher face quality is selected as the target. video frame, and then generate the learning status message according to the target video frame. Through the above process, the generated learning status information can be based on the information obtained from the video frames with higher face quality and including learning behavior, and has higher accuracy, so that the learning of the target object can be more accurately grasped state.

As described in the above embodiments of the invention, the implementation manner of step S1313 can be flexibly changed. In a possible implementation, step S1313 may include:

use at least one of the target video frames as a learning status message; and/or,

Identifying the region where the target object is located in at least one target video frame, and generating a learning status message based on the region where the target object is located.

It can be seen from the above embodiments of the invention that, in a possible implementation manner, at least one frame in the target video frames can be directly used as the learning state information, and in an example, the obtained target video frames can be further selected, The selection can be random or under certain conditions, and then the selected target video frame can be directly used as the learning status message; in an example, each obtained target video frame can also be directly used as the learning status message .

In a possible implementation manner, the region where the target object is located in the target video frame may be further identified, so as to generate the learning status message according to the region where the target object is located. The manner of identifying the target object area is not limited in the embodiments of the present invention, and in a possible implementation manner, it may be implemented by the neural network with the target object detection function mentioned in the above embodiments of the present invention. After the region of the target object in the target video frame is determined, the target video frame can be further processed correspondingly to obtain learning status information. The processing method can be flexibly determined. In one example, the image of the area where the target object is located in the target video frame can be used as the learning state information; Render the background area, such as adding other stickers, or adding mosaics to the background area, or replacing the image of the background area, etc., to obtain the learning status information that does not display the current background of the target object, so that the target object can be better. Privacy protection, and rendering methods such as stickers can also be used to increase the diversity and beauty of learning status information.

By using at least one frame in the target video as the learning status information, and/or generating the learning status information according to the area where the target object is located in the target video frame, the learning status information finally obtained can be made more flexible by the above method, so that it is possible to According to the needs of the target object, the learning status information that is more prominent for the target object, or the learning status information that protects the privacy of the target object is obtained.

The above embodiments of the invention can be combined arbitrarily to obtain learning status information generated based on video frames containing learning behaviors. For example, Table 1 shows the learning status information generation rules according to an embodiment of the present invention. learning behavior Detection rules data processing Image Processing image upload raise hand gesture Hand raising is detected in N frames in consecutive M frames of the same face track (frame sequence) The time stamp of the first frame where the raised hand was detected is used as the start time of the raised hand. If the hand-raising is not detected in consecutive Z frames, or the face track is lost, the timestamp of the frame where the hand-raising was last detected is used as the hand-raising end time. Start time and end time, as a raise of hands event. From the start of raising the hand to the end of raising the hand, among all the frames in which the raised hand is detected, take the original image or cutout with the highest face quality score and higher than the quality score threshold of the wonderful moment The encoded MJPEG is uploaded to the background. Upload Y sheets every X minutes. Like gesture Thumbs up is detected in N frames in consecutive M frames of the same face track (frame sequence) Ditto From the beginning to the end of the gesture liking, among all the frames where the gesture liking is detected, take the original image or cutout with the highest face quality score and higher than the quality score threshold of the wonderful moment The encoded MJPEG is uploaded to the background. Upload Y sheets every X minutes. OK gesture Gestures are detected in N frames in consecutive M frames of the same face track (frame sequence) OK Ditto From the beginning of the gesture OK to the end, among all the frames where the gesture OK is detected, take the original image or cutout with the highest face quality score and higher than the quality score threshold of the wonderful moment The encoded MJPEG is uploaded to the background. Upload Y sheets every X minutes. victory gesture Gestures are detected in N frames in consecutive M frames of the same face track (frame sequence) Yeah Ditto From the beginning to the end of the gesture Yeah, in all the frames where the gesture Yeah is detected, take the original image or cutout with the highest face quality score and higher than the quality score threshold of the wonderful moment The encoded MJPEG is uploaded to the background. Upload Y sheets every X minutes. Interactive Behavior-Teacher Sends Little Red Flowers SDK (software development kit) calls the rendering of the red flower sticker The rendering start time is used as the start time, and the rendering end time is used as the end time none none Interactive Behavior - The teacher speaks by name SDK calls the rendering of speech stickers Ditto none none show happiness There are N frames in consecutive M frames of the same face track (frame sequence), and it is detected that the expression is happy or the smile value is higher than a certain threshold The timestamp of frame 1 is used as the start time. If no happiness or smile value higher than a certain threshold is detected in consecutive Z frames, or the face track is lost, the time stamp of the frame with the happy or smile value higher than a certain threshold is finally detected as the end time. Start time and end time as a happy event. From the beginning to the end, among all the frames with a happy expression or a smile value higher than a certain threshold, take the original image or cutout with the highest face quality score and higher than the quality score threshold of the wonderful moment and upload it The encoded MJPEG is uploaded to the background. Upload Y sheets every X minutes. Focus on the display area of the teaching course (Concentration) There are N frames in consecutive M frames of the same face track (frame sequence), the expression is detected to be calm and the headpose (face angle) is within the viewing threshold range The timestamp of frame 1 is used as the start time. If no calmness is detected in consecutive Z frames and the headpose is within the viewing threshold range, or the face track is lost, the timestamp of the frame where calmness is finally detected and the headpose is within the viewing threshold range is used as the end time. Start time and end time, as a concentration event. From the beginning to the end, among all the frames with the average detected and the headpose within the viewing threshold range, take the original image or cutout with the highest face quality score and higher than the quality score threshold of the wonderful moment and upload it The encoded MJPEG is uploaded to the background. Upload Y sheets every X minutes.

Table 1 Learning status message generation rules

Among them, M, N, X, Y, and Z are all positive integers, and the specific values can be set according to actual needs. In addition, the parameters such as M in different rows in Table 1 may be the same or different, and the above parameters such as M are only used as a schematic illustration, and are not used as a limitation on the content of the present invention.

Among them, the wonderful moment is the moment corresponding to the target object's active learning behavior. It can be seen from Table 1 that, in an example, the target object can be detected to perform target gestures such as raising his hand, generate the target emotion of happiness, or concentrate on the display area of the teaching course, and interact with the teacher such as roll call speech, etc. In the case of school behavior, certain data processing is performed on the video, and after the data processing, further image processing is performed on the video frame, so as to obtain the target video frame as the learning status message.

In a possible implementation manner, the learning status information is generated according to the duration of at least one type of learning behavior performed by the target object, which may include:

Step S1321, when it is detected that the time when the target object performs at least one type of learning behavior is not less than a time threshold, record the duration of at least one type of learning behavior;

In step S1322, the duration corresponding to at least one type of learning behavior is used as the learning status message.

The time threshold may be a certain value flexibly set according to the actual situation, and the time thresholds of different types of learning behaviors may be the same or different. When it is detected that the target object performs a certain type of learning behavior within a certain period of time, the time when the target object performs these learning behaviors can be counted, so as to be fed back to the teacher or the parent as a learning status message. The specific statistical conditions and the statistical time under which learning behaviors are implemented can be flexibly set according to the actual situation.

In one possible implementation, the absence of the target object in the video (such as no one in the video, someone in the video frame but not sure if it is the target object, or someone in the shot but not the target object) is detected for more than a period of time. When the timing is long, the target object closes his eyes, or the target object does not watch the display area of the teaching course, the duration of these learning behaviors can be counted and used as learning status information.

In the embodiment of the present invention, when it is detected that the time when the target object performs at least one type of learning behavior is not less than the time threshold, the duration of at least one type of learning behavior is recorded as the learning state information, and through the above process, the learning state can be The information is quantified, and the learning status of the target object can be grasped more intuitively and accurately.

In a possible implementation manner, the video processing method proposed in the embodiment of the present invention may further include:

Rendering a background area in at least a part of the video frame in the video, wherein the background area is an area other than the target object in the video frame.

For the segmentation method of the background area and the rendering method of the background area, reference may be made to the above-mentioned embodiments of the invention, the identification of the area where the target object is located in the target video frame and the rendering process after identification, which will not be repeated here. In the process of rendering the background area, in one example, the rendering can be performed by using a preset general template in the current video processing device; Customized templates, etc. for rendering, for example, other background templates can be called from a cloud server that is not a video processing device to render the background area in the video.

By rendering the background area in at least part of the video frame in the video, on the one hand, the privacy of the target object in the video can be protected, and the possibility of privacy leakage of the target object due to the lack of a suitable video capture position can be reduced. It can enhance the interest of the target object to watch the teaching course process.

In a possible implementation manner, the video processing method proposed in the embodiment of the present invention may further include:

Count the learning status information of at least one target object, and obtain the statistical result of at least one target object;

Based on the statistical results of at least one target object, learning state statistics are generated.

In the embodiment of the present invention, a target object contained in a video may be one or multiple. In addition, the video processing method in the embodiment of the present invention may be used to process a single video, or may be used to process a single video. Multiple videos are processed. Therefore, correspondingly, the learning status information of one target object can be obtained, and the learning status information of multiple target objects can also be obtained. In this case, the learning status information of the at least one target object may be counted to obtain the statistical result of the at least one target object. The statistical result may include not only the learning status information of the target object, but also other information related to the target object's viewing of the teaching course. For example, in a possible implementation manner, before step S12, that is, before performing learning behavior detection on the target object, the check-in data of the target object may also be obtained. The check-in data of the target object may include the identity information of the target object and the check-in time, etc. The specific method of obtaining the check-in data can be flexibly determined according to the actual check-in method of the target object, which is not limited in this embodiment of the present invention.

After the statistical result of the at least one target object is obtained, the statistical data of the learning state may be generated according to the at least one statistical result. Specifically, the generation method and content of the learning state statistical data can be flexibly changed according to the realization form of the statistical results. Please refer to the following embodiments of the invention for details, which will not be expanded here.

In the embodiment of the present invention, the statistical result of at least one target object is obtained by counting the learning status information of at least one target object, so that the statistical data of the learning status is generated according to the statistical result of the at least one target object. Through the above process, it is possible to effectively Comprehensive evaluation of the learning status of multiple target objects makes it easier for teachers to grasp the overall learning situation of the entire classroom, and for other relevant personnel to more comprehensively understand the current learning position of the target objects.

In a possible implementation manner, according to the statistical result of at least one of the target objects, the statistical data of the learning state is generated, including:

According to the category to which the at least one target object belongs, the statistical results of the target objects included in the at least one category are obtained, and the statistical data of the learning status of the at least one category is generated, wherein the category to which the target object belongs includes the courses that the target object participates in and the institution where the target object is registered. and at least one of the devices used by the target; and/or,

The statistical result of the at least one target object is visualized to generate the statistical data of the learning state of the at least one target object.

The category to which the target object belongs may be a category classified according to the identity of the target object. For example, the category to which the target object belongs may include at least one of the courses that the target object participates in, the institution registered by the target object, and the equipment used by the target object. One, in which the course that the target object participates in can be the teaching course watched by the target object mentioned in the above invention embodiment, and the institution registered by the target object can be the educational institution where the target object is located, or the grade or target object where the target object is located. The class in which the object is located, etc., the device used by the target object may be the terminal device used by the target object to participate in the online course in the online scene, and the like.

In this embodiment of the present invention, statistical results of target objects included in at least one category may be obtained according to the category to which the target object belongs, that is, at least one statistical result of the category to which the target object belongs may be aggregated to obtain the statistical results of the category to which the target object belongs. Overall learning status statistics. For example, it can be divided according to the categories of equipment, courses, educational institutions, etc., to obtain the statistical results of different target objects under the same device, the statistical results of different target objects under the same course, and the statistical results of different target objects in the same educational institution. Wait. In an example, these statistical results can also be presented in the form of a report. In an example, the statistical results under each category in the report can include not only the overall learning status information of each target object, but also the specific learning status information of each target object, such as focusing on the teaching course display area. The length of time, the length of time to smile, etc., in addition to other information related to watching the teaching course, such as the check-in time of the target object, the number of check-in times, the match between the target object and the face in the preset database, Check-in equipment, check-in classes, etc.

In addition, the statistical result of the at least one target object can also be visualized to obtain the statistical data of the learning state of the at least one target object. Among them, the visualization processing method can be flexibly determined according to the actual situation, for example, the data can be organized into the form of charts or videos. The content included in the learning status statistics can be flexibly determined according to the actual situation, such as the overall learning status information of the target object, the name of the teaching course watched by the target object, and the specific learning status information of the target object, etc. The specific data included can be determined according to the The actual situation is flexible. In one example, the identity of the target object, the name of the teaching course watched by the target object, the duration of the target object's attention to the teaching course display area, the degree of attention of the target object, and the data between the target object and other target objects The comparison results, the number of interactions of the target object, and the emotions of the target object are organized into a visual report and sent to the target object or other relevant personnel of the target object, such as the target object's parents, etc.

In an example, in addition to pictures and videos, the visualized learning status statistics can contain text content in the form of "The class subject is XX, the student A's concentration time is 30 minutes, and the concentration is concentrated, which is higher than that in the class. 10% of the classmates interacted 3 times and smiled 5 times. I would like to express my praise and I would like to continue to work hard.” Pay attention and adjust your child’s study habits in time” and so on.

In this embodiment of the present invention, by acquiring the category to which the at least one target object belongs, the statistical data of the learning state of the at least one category is generated, and/or the statistical results of the at least one target object are visualized to generate the statistical data of the at least one target object. Learn state statistics. Through the above process, the learning state of the target object can be grasped more intuitively and comprehensively through different data statistics methods.

FIG. 2 shows a block diagram of a video processing apparatus according to an embodiment of the present invention. As shown in the figure, the video processing apparatus 20 may include:

a video acquisition module 21, used for acquiring video, wherein at least some of the video frames in the video contain a target object;

The detection module 22 is used to detect at least one type of learning behavior of the target object in the process of watching the teaching course according to the video;

The generation module 23 is used to generate learning status information according to at least part of the video frames that contain at least one type of learning behavior and/or the duration of the target object to perform at least one type of learning behavior when it is detected that the target object performs at least one type of learning behavior .

In one possible implementation, the learning behavior includes at least one of the following behaviors: performing at least one target gesture, expressing target emotion, paying attention to the display area of the teaching course, generating at least one interactive behavior with other objects, performing at least one part of the video Eyes closed, absent from video frames, and eye contact within the presentation area of the teaching session.

In a possible implementation manner, the detection module 22 is configured to: perform target object detection on the video to obtain a video frame containing the target object; and perform at least one type of learning behavior detection on the video frame containing the target object.

In a possible implementation manner, the learning behavior includes performing at least one target gesture; the detection module 22 is further configured to: detect the at least one target gesture on the video frame containing the target object; When the number of video frames exceeds the first threshold, at least one frame in the video frames containing the target gesture is recorded as the gesture start frame; in the video frames after the gesture start frame, the number of consecutive video frames that do not contain the target gesture In the case of exceeding the second threshold, at least one frame in the video frames that does not contain the target gesture is recorded as the gesture end frame; according to the number of gesture start frames and gesture end frames, determine the number of times that the target object in the video performs at least one target gesture and/or time.

In a possible implementation manner, the learning behavior includes expressing the target emotion; the detection module 22 is further configured to: perform expression detection and/or smile value detection on the video frame containing the target object; in the detected video frame, the target object displays at least A first target expression or smile value detection result exceeds the target smile value situation, the detected video frame is used as the first detection frame; In the case where the number of consecutive first detection frames is detected to exceed the third threshold, determine The target object produces the target emotion.

In a possible implementation manner, the learning behavior includes paying attention to the display area of the teaching course; the detection module 22 is further configured to: perform expression detection and face angle detection on the video frame containing the target object; In the case where at least one second target expression is displayed and the face angle is within the range of the target face angle, the detected video frame is used as the second detection frame; when it is detected that the number of consecutive second detection frames exceeds the fourth threshold In this case, it is determined that the target object pays attention to the display area of the teaching course.

In a possible implementation manner, the generating module 23 is used to: obtain a video frame containing at least one type of learning behavior in the video, as a target video frame set; perform face quality detection on at least one video frame in the target video frame set, The video frame whose face quality is greater than the face quality threshold is used as the target video frame; the learning status message is generated according to the target video frame.

In a possible implementation manner, the generating module 23 is further configured to: use at least one frame in the target video frame as the learning state information; and/or, identify the area where the target object is located in the at least one frame of the target video frame, based on the target video frame The area where the object is located, and the learning status message is generated.

In a possible implementation manner, the detection module 22 is used to: perform target object detection on the video, obtain a video frame containing the target object, and use the video frame other than the video frame containing the target object as the undetected target video frames of the object; in the case that the number of video frames of the target object is not detected exceeds the preset number of video frames, detecting the learning behavior includes: not appearing in at least part of the video frames in the video.

In a possible implementation manner, the generating module 23 is configured to: record the duration of the at least one type of learning behavior when it is detected that the time when the target object performs at least one type of learning behavior is not less than a time threshold; duration as a learning status message.

In a possible implementation manner, the apparatus is further configured to: render a background area in at least part of the video frame in the video, where the background area is an area other than the target object in the video frame.

In a possible implementation manner, the apparatus is further configured to: count the learning state information of the at least one target object to obtain a statistical result of the at least one target object; and generate learning state statistical data according to the statistical result of the at least one target object.

In a possible implementation manner, the apparatus is further configured to: obtain statistical results of the target objects included in the at least one category according to the category to which the at least one target object belongs, and generate statistical data of the learning state of the at least one category, wherein the target object belongs to The category includes at least one of the courses attended by the target object, the institution registered by the target object, and the equipment used by the target object; and/or, the statistical result of the at least one target object is visualized to generate the learning status statistical data of the at least one target object .

In the case of not violating the logic, different embodiments of the present application may be combined with each other, and the descriptions of different embodiments have some emphasis, and for the parts that are not described, reference may be made to the records of other embodiments.

In some embodiments of the present invention, the functions or modules included in the devices provided in the embodiments of the present invention may be used to execute the methods described in the above method embodiments, and the specific implementation and technical effects may refer to the above method embodiments The description, for the sake of brevity, will not be repeated here.

Application Scenario Example

The way students learn is usually that teachers teach and students listen to the class. The class lacks interaction and interest. It is not easy for students to get interested in listening to the class, and it is not possible to form positive incentives for students through their real-time performance. At the same time, institutions or teachers cannot grasp the status of students' class attendance, and parents cannot understand their children's performance in school. Especially due to the impact of the epidemic, students spend a lot of time in online classes. The performance of the interaction cannot be quantified. Therefore, how to effectively grasp the students' learning status has become an extremely unsolved problem.

The application example of the present invention proposes a learning system, which can effectively grasp the learning status of students through the video processing method proposed in the above-mentioned embodiments of the present invention.

FIG. 3 shows a schematic diagram of an application example according to the present invention. As shown in the figure, in an example, the learning system can be composed of three parts: a client, an educational software-as-a-Service (SaaS, Software-as-a-Service) background, and an interactive classroom background. Among them, students watch teaching courses through the client, and the client can include two parts, namely the hardware device for learning (such as the client with Windows system or IOS system and SDK installed in the picture), and the student logging in to the online class application (ie the user APP in the figure). The educational SaaS backend can be a platform built by the server of the educational institution where students are located, and the interactive classroom backend can be a platform built by a server that aggregates data from different educational institutions and maintains data, whether it is an educational SaaS backend or an interactive classroom backend , can exchange data with the client through the API interface. Thus, the generation of learning status information and the generation of learning status statistical data mentioned in the above embodiments of the invention are realized.

In an application example of the present invention, the generation process of the learning status message may include:

The client obtains the learning status information of each student by collecting the video of the students watching the teaching course and processing the collected video. The education SaaS background and the interactive classroom background call the learning status generated in different clients through the API interface. information, and perform statistical processing on the learning status information in any manner mentioned in the above embodiments of the invention to generate learning status statistical data.

In an example, the user terminal processes the collected video, and the process of acquiring the learning status information of each student may include:

A. Obtain the wonderful moments of the students in class (that is, the active learning behavior mentioned in the above-mentioned embodiments of the invention).

In an example, a highlight video collection of students can be produced by defining certain rules, and students' performance can be edited into a short video or some wonderful pictures and provided to parents, so that parents can timely evaluate students' performance in class, and if the effect is effective Well, students may be encouraged to continue taking relevant courses.

In an example, obtaining the highlights of the students can be performed after the students have successfully signed in, and the video or pictures of the highlights will be uploaded to the background or the cloud. At the same time, it is also possible to select whether the students can see the content of the uploaded moments in real time. In one example, the rules for defining a wonderful moment may include: generating at least one target gesture, and the target gesture may include raising a hand, a thumbs up, an OK gesture, a gesture Yeah, etc. If it is detected that a student performs the above gestures within a period of time, the Image or video frame extraction can be performed on videos containing gestures. The target emotion of expressing happiness. If it is detected that the student's expression is happy within a certain period of time, and the smile value reaches a certain target smile value (such as 99 points), there can be a video frame with a happy label or a target smile value. Video frames are extracted from pictures or video frames. Pay attention to the display area of the teaching course. If the student's face orientation is always correct within a certain period of time, that is, the headpose is within a certain threshold range, then pictures or video frames can be extracted from the video within this period of time.

b. The learning situation detection is performed on the learning situation of the students (for the negative learning behavior mentioned in the above embodiments of the invention).

In one example, students may not be on the screen or are not focused, and the data can be pushed to parents in real time through learning situation detection, so that parents can pay attention to their children at the first time and correct their children's bad study habits in a timely manner. Auxiliary supervision.

In an example, the process of student learning situation detection can be carried out after the student has successfully signed in. If no one appears in front of the camera for a long time, does not watch the screen, closes his eyes, etc., it is judged that the person's concentration is low. In this case, the duration of the student's above-mentioned learning behavior can be counted, and it can be used as the result of the learning situation detection to obtain the corresponding learning status data. For specific configuration rules of learning situation detection, reference may be made to the foregoing embodiments of the invention, and details are not described herein again.

Through the above examples of the invention, learning status information including exciting moments and learning situation detection can be obtained. Further, the education SaaS background and the interactive classroom background use the API interface to call the learning status information generated in different clients to generate learning status. The process of statistical data can include:

c. Report generation (that is, generating at least one category of learning status statistics in the above embodiments of the invention).

In one example, the background or cloud API can view students' check-in information and learning status information in different dimensions such as devices, courses, and institutions. The main data indicators can include: check-in time, number of check-ins, and face database (that is, the above invention). The target object in the embodiment is matched with the face in the preset database), check-in equipment, check-in course, focus time, smile time, etc.

D. An analysis report (that is, the visualization process in the above embodiments of the invention generates the statistical data of the learning state of at least one target object).

In one example, the educational SaaS backend or the interactive classroom backend can organize students' performance in online classes into a complete learning situation analysis report. The report explains the students' class status through a visual graphical interface. Further, the background can also select a better situation and push it to parents or teachers, so that it can be used by institutional teachers to analyze the students' situation and gradually assist children to improve their learning behavior.

In addition to the above process, the learning system can also perform background segmentation processing on the student's learning video during the process of the student's learning through the user terminal. In an example, the user terminal may provide a background segmentation function for the situation where the student does not have a location background suitable for live broadcast or is unwilling to display the background image due to privacy protection. In an example, the SDK of the client can support several different background templates, for example, several general templates can be preset. In an example, students can also call customized templates from the interactive classroom background through the client. In an example, the SDK can provide a background template preview interface to the APP on the client side, so that students can preview the customized templates that can be called through the APP; during the class, students can also use the background-segmented stickers on the APP on the client side to be used for Render the live background and, in one example, also trigger a shutdown manually if the students are not satisfied with the sticker. The APP on the client side can report the data of students' use of stickers to the corresponding backend (educational SaaS backend or interactive classroom backend).

The learning system proposed in the application example of the present invention can be extended to other related fields, such as online conferences, in addition to being applicable to online classes.

It can be understood that the above method embodiments mentioned in the present invention can be combined with each other to form a combined embodiment without violating the principle and logic. Due to space limitations, the present invention will not repeat them.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

An embodiment of the present invention further provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above-mentioned method is implemented. The computer-readable storage medium can be a volatile computer-readable storage medium or a non-volatile computer-readable storage medium.

An embodiment of the present invention further provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to perform the above method.

An embodiment of the present invention further provides a computer program, including computer-readable code, when the computer-readable code is executed in an electronic device, the processor in the electronic device executes the method to implement the above method.

In practical applications, the above-mentioned memory may be volatile memory, such as RAM; or non-volatile memory, such as ROM, flash memory, hard disk ( Hard Disk Drive, HDD) or Solid-State Drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

The above-mentioned processor may be at least one of ASIC, DSP, DSPD, PLD, FPGA, CPU, controller, microcontroller, and microprocessor. It can be understood that, for different devices, the electronic apparatus for implementing the above processor function may also be other, which is not specifically limited in this embodiment of the present invention.

The electronic device may be provided as a terminal, server or other form of device.

Based on the same technical concept as the foregoing embodiments, an embodiment of the present invention further provides a computer program, which implements the above method when the computer program is executed by a processor.

FIG. 4 is a block diagram of an electronic device 800 according to an embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant and other terminals.

4, an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, a sensing server component 814, and communication component 816.

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

The memory 804 is configured to store various types of data to support the operation of the electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as static random access memory (SRAM), electronically erasable programmable read only memory (EEPROM), erasable Except Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Disk or Optical Disk.

Power supply assembly 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC) that is configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a calling mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or sent via the communication component 816 . In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules. The peripheral interface modules may be keyboards, mice, buttons, and the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for electronic device 800 . For example, the sensor assembly 814 can detect the open/closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or The position of a component of the electronic device 800 changes, the presence or absence of user contact with the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature of the electronic device 800 changes. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, 3G, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related personnel messages from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication assembly 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), Field Programmable Logic Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions executable by the processor 820 of the electronic device 800 to accomplish the above method.

FIG. 5 is a block diagram of an electronic device 1900 according to an embodiment of the present invention. For example, the electronic device 1900 may be provided as a server. 5, electronic device 1900 includes processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions executable by processing component 1922, such as applications. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) Interface 1958. Electronic device 1900 may operate based on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions executable by the processing component 1922 of the electronic device 1900 to accomplish the above method.

The present invention may be a system, method and/or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk-read-only memory (CD-ROM), digital versatile disc (DVD), memory card, magnetic A sheet, a mechanical coding device, such as a punched card or a raised structure in a groove with instructions stored thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing/processing devices, or downloaded to external computers over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network or external storage device. Networks may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. A network interface card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage stored in each computing/processing device in the medium.

The computer program instructions for carrying out the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source or object code written in any combination, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. Computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server execute on. In the case of a remote computer, the remote computer may be connected to the user's computer via any kind of network - including a local area network (LAN) or wide area network (WAN) - or, it may be connected to an external computer (for example, using the Internet service provider to connect via the Internet). In some embodiments, electronic circuits, such as Programmable Logic Circuits, Field Programmable Logic Gate Arrays (FPGA) or Programmable Logic Arrays (PLA), are personalized by utilizing state personnel information of computer readable program instructions. ), the electronic circuitry can execute computer-readable program instructions to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to the processor of a general purpose computer, special purpose computer or other programmable data processing device to produce a machine for execution of the instructions by the processor of the computer or other programmable data processing device When, means are created that implement the functions/acts specified in one or more of the blocks in the flowchart and/or block diagrams. These computer-readable program instructions may also be stored in a computer-readable storage medium, the instructions causing the computer, programmable data processing device and/or other equipment to operate in a particular manner, so that the computer-readable medium storing the instructions Included is an article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer-readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment, such that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented processes such that instructions executing on a computer, other programmable data processing apparatus, or other device implement the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that contains one or more logic for implementing the specified logic Executable instructions for the function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by dedicated hardware-based systems that perform the specified functions or actions. implementation, or may be implemented in a combination of special purpose hardware and computer instructions.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

20: Video processing device 21: Video acquisition module 22: Detection module 23: Generate Mods 800: Electronics 802: Process component 804: memory 806: Power Components 808: Multimedia Components 810: Audio Components 812: Input/Output Interface 814: Sensor Assembly 816: Communication Components 820: Processor 1900: Electronic equipment 1922: Processing components 1926: Power Components 1932: Memory 1950: Web Interface 1958: Input/Output Interface S11~S13: Steps

The drawings herein are incorporated into and constitute a part of this specification, and these drawings illustrate embodiments consistent with the present invention, and together with the description, serve to explain the technical solutions of the present invention: 1 shows a flowchart of a video processing method according to an embodiment of the present invention; FIG. 2 shows a block diagram of a video processing apparatus according to an embodiment of the present invention; 3 shows a schematic diagram of an application example according to the present invention; FIG. 4 shows a block diagram of an electronic device according to an embodiment of the present invention; and FIG. 5 shows a block diagram of an electronic device according to an embodiment of the present invention.

S11~S13: Steps

Claims (12)

A video processing method, comprising: acquiring a video, wherein at least some of the video frames in the video include a target object; According to the video, detecting at least one type of learning behavior of the target object in the process of watching the teaching course; In the case where it is detected that the target object performs at least one type of learning behavior, a learning state is generated according to a video frame at least partially including the at least one type of learning behavior and/or the duration of the target object performing the at least one type of learning behavior message. The method according to claim 1, wherein the learning behavior includes at least one of the following behaviors: performing at least one target gesture, expressing a target emotion, paying attention to the display area of the teaching course, and generating at least one interactive behavior with other objects , absence in at least some of the video frames in the video, eyes closed, and eye contact within the presentation area of the teaching session. The method according to claim 1 or 2, wherein the detecting at least one type of learning behavior of the target object according to the video includes: performing target object detection on the video to obtain a video frame containing the target object; At least one type of learning behavior detection is performed on the video frame containing the target object. The method of claim 3, wherein, In the case that the learning behavior includes performing at least one target gesture, performing at least one type of learning behavior detection on the video frame containing the target object includes: Perform detection of at least one target gesture on the video frame containing the target object; in the case where the number of consecutive video frames containing the at least one target gesture is detected exceeds a first threshold, the video frame containing the target gesture is detected. At least one frame is recorded as the gesture start frame; in the video frames after the gesture start frame, when the number of consecutive video frames that do not include the target gesture exceeds the second threshold, the video frames that do not include the target gesture will be At least one frame in the frame is recorded as the gesture end frame; according to the number of the gesture start frame and the gesture end frame, determine the number of times and/or the time that the target object in the video performs at least one target gesture; and / or, In the case that the learning behavior includes expressing the target emotion, performing at least one type of learning behavior detection on the video frame containing the target object includes: Expression detection and/or smile value detection is performed on the video frame containing the target object; when it is detected that the target object in the video frame shows at least one first target expression or the result of the smile value detection exceeds the target smile value, the The detected video frame is used as a first detection frame; in the case that the number of consecutive first detection frames is detected to exceed a third threshold, it is determined that the target object produces the target emotion; and / or, When the learning behavior includes paying attention to the display area of the teaching course, performing at least one type of learning behavior detection on the video frame containing the target object includes: Perform expression detection and face angle detection on the video frame containing the target object; when it is detected that the target object in the video frame shows at least one second target expression and the face angle is within the range of the target face angle, The detected video frame is used as the second detection frame; in the case that the number of consecutive second detection frames is detected to exceed a fourth threshold, it is determined that the target object pays attention to the display area of the teaching course. The method according to claim 1 or 2, wherein the generating the learning status message according to at least part of the video frames containing the at least one type of learning behavior includes: Acquire a video frame containing at least one type of learning behavior in the video, as a target video frame set; perform face quality detection on at least one video frame in the target video frame set, and detect the video with the face quality greater than the face quality threshold. frame as a target video frame; according to the target video frame, generate the learning status message; and / or, The generation of learning status information according to the duration of the at least one type of learning behavior performed by the target object includes: In the case where it is detected that the time when the target object performs at least one type of learning behavior is not less than a time threshold, record the duration of at least one type of learning behavior; take the duration corresponding to at least one type of learning behavior as the Learning status messages. The method according to claim 5, wherein the generating the learning status message according to the target video frame includes: Using at least one frame of the target video frames as learning status information; and/or, Identifying the region where the target object is located in at least one frame of the target video frame, and generating the learning status message based on the region where the target object is located. The method according to claim 1 or 2, wherein the detecting at least one type of learning behavior of the target object according to the video includes: Performing target object detection on the video, obtaining a video frame containing the target object, and using the video frame other than the video frame containing the target object in the video as a video frame in which the target object is not detected; When the number of video frames in which the target object is not detected exceeds a preset number of video frames, detecting the learning behavior includes: not appearing in at least part of the video frames in the video. The method according to claim 1 or 2, wherein the method further comprises: Rendering a background area in at least part of the video frame in the video, wherein the background area is an area other than the target object in the video frame; and / or, Counting the learning status information of at least one of the target objects to obtain statistical results of at least one of the target objects; and generating learning status statistical data according to the statistical results of at least one of the target objects. The method according to claim 8, wherein generating the statistical data of the learning state according to the statistical result of at least one of the target objects, comprises: Obtain statistical results of at least one target object included in the category according to the category to which at least one of the target objects belongs, and generate statistical data of the learning state of at least one category, wherein the category to which the target object belongs includes the participation of the target object at least one of the course, the institution in which the target is registered, and the equipment used by the target; and/or, Visually process the statistical result of at least one of the target objects, and generate the statistical data of the learning state of at least one of the target objects. A video processing device, comprising: a video acquisition module for acquiring video, wherein at least some of the video frames in the video contain a target object; a detection module, configured to detect at least one type of learning behavior of the target object in the process of watching the teaching course according to the video; The generating module is used for, when it is detected that the target object performs at least one type of learning behavior, according to a video frame that at least partially contains the at least one type of learning behavior and/or the target object performs the at least one type of learning behavior. Duration to generate learning status messages. An electronic device comprising: processor; memory for storing processor-executable instructions; Wherein, the processor is configured to invoke the instructions stored in the memory to execute the method described in any one of request items 1 to 9. A computer-readable storage medium on which computer program instructions are stored, wherein when the computer program instructions are executed by a processor, the method described in any one of claim 1 to 9 is implemented.

Images