CN112055257A

CN112055257A - Video classroom interaction method, device, equipment and storage medium

Info

Publication number: CN112055257A
Application number: CN201910486292.7A
Authority: CN
Inventors: 金磊; 冀志龙; 张无名
Original assignee: Beijing Xintang Sichuang Educational Technology Co Ltd
Current assignee: Beijing Xintang Sichuang Educational Technology Co Ltd
Priority date: 2019-06-05
Filing date: 2019-06-05
Publication date: 2020-12-08
Anticipated expiration: 2039-06-05
Also published as: CN112055257B

Abstract

The embodiment of the invention provides a video classroom interaction method, a device, equipment and a storage medium, wherein the video classroom interaction method comprises the following steps: acquiring student state data of a classroom; when the student state data meet an interaction triggering threshold, generating an interaction video of a teacher according to classroom scene data at a triggering moment, wherein the triggering moment is the moment when the student state data meet the interaction triggering threshold; and inserting the interactive video after the current playing frame of the video playing in the classroom of the video classroom. The video classroom interaction method, device, equipment and storage medium provided by the embodiment of the invention improve the necessity of interaction, ensure the flexibility of interaction, improve the authenticity and reliability of interaction, and ensure the learning atmosphere and interaction effect of classroom.

Description

Video classroom interaction method, device, equipment and storage medium

Technical Field

The embodiment of the invention relates to the field of computers, in particular to a video classroom interaction method, device, equipment and storage medium.

Background

Along with the development of artificial intelligence technology, AI classroom teachers (i.e. recording video teachers) based on artificial intelligence technology also appear, classroom teaching is realized through recorded videos, namely: video classroom.

In order to ensure the quality of class of a video classroom, some interaction links are usually set to mobilize the learning atmosphere of the video classroom, but the current interaction in the video classroom is directional interaction based on a set node and a recorded video, the number of interaction times in the whole video classroom process is small, the video classroom is not rigid naturally, and the video classroom has a large difference from a real class scene.

Therefore, how to improve the interactive effect of the video classroom becomes a technical problem which needs to be solved urgently.

Disclosure of Invention

The embodiment of the invention provides a video classroom interaction method, device, equipment and storage medium, so as to improve the interaction effect of a video classroom.

In order to solve the above problem, an embodiment of the present invention provides a video classroom interaction method, including:

acquiring student state data of a classroom;

when the student state data meet an interaction triggering threshold, generating an interaction video of a teacher according to classroom scene data at a triggering moment, wherein the triggering moment is the moment when the student state data meet the interaction triggering threshold;

and inserting the interactive video after the current playing frame of the video playing in the classroom of the video classroom.

Optionally, the step of acquiring student status data of a classroom includes:

acquiring all student images of a classroom;

acquiring current posture data of each student according to all the student images;

when the student state data meet an interaction triggering threshold value, generating an interaction video of a teacher according to classroom scene data at a triggering moment comprises the following steps:

and when at least one of the current posture data of the students meets the student posture data interaction triggering threshold, generating the interaction video of the teacher according to the classroom scene data at the triggering moment.

Optionally, the step of acquiring student status data of a classroom further includes:

obtaining the whole concentration degree of the classroom according to all the current posture data of the students in the current postures;

when the student state data meet the interaction triggering threshold, generating the teacher interaction video according to the classroom scene data at the triggering moment, and further comprising the following steps of:

and when the overall concentration degree meets the overall concentration degree interaction triggering threshold value, generating the teacher's interaction video according to the classroom scene data at the triggering moment.

Optionally, the step of obtaining the overall concentration degree of the classroom according to all the current posture data of the students in the current postures comprises:

obtaining the number proportion of the current posture data of different students according to all the current posture data of the students;

multiplying each quantity ratio by the weight of each student current attitude data to obtain each quantity weight ratio;

and calculating the sum of the quantity weight ratios to obtain the overall concentration degree.

acquiring current posture data of each student at each moment within a preset time length before the moment of acquiring all student images;

according to the current posture data of each student at each moment, the concentration degree of each person in the classroom is obtained;

and when at least one personal concentration degree meets the personal concentration degree interaction triggering threshold value, generating the interaction video of the teacher according to the classroom scene data at the triggering moment.

Optionally, the step of obtaining the concentration degree of each person in the classroom according to the current posture data of each student at each moment includes:

according to the current posture data of each student at each moment, obtaining the time proportion of the current posture data of different students in the preset time length;

multiplying each time proportion of the same student by the weight of the current posture data of each student to obtain the time weight proportion of each same student, and calculating the sum of each time weight proportion to obtain the individual concentration degree of the same student until each student is traversed to obtain the individual concentration degree.

Optionally, the step of obtaining current posture data of each student according to the all student images includes:

detecting and identifying a face block diagram of each student in all the student images, and detecting a body block diagram of each student;

selecting one of the face block diagrams of each student to obtain a selected face block diagram, calculating the overlapping degree of the selected face block diagram and each body block diagram to obtain a matched body block diagram of which the overlapping degree accords with the threshold value of the overlapping degree, matching the selected face block diagram with the matched body block diagram until the face block diagrams of each student are traversed, and obtaining the current posture data of each student according to the selected face block diagram and the matched body block diagram which are matched with each other.

and acquiring student state data of the classroom according to a preset acquisition trigger standard.

Optionally, the step of acquiring classroom scene data at the trigger time and generating an interactive video of a teacher includes:

generating audio of a teacher according to the classroom scene data at the trigger moment;

generating a video of a teacher according to the trigger time and the audio;

and synthesizing the audio and the video to obtain the interactive video.

Optionally, the step of generating the audio of the teacher according to the classroom scene data at the trigger time includes:

according to the classroom scene data at the trigger moment, language text data of teachers under the classroom scene data are searched;

generating the speech text data into the audio using speech synthesis techniques.

Optionally, the step of generating a video of a teacher according to the trigger time and the audio includes:

acquiring a classroom playing video frame at the trigger moment;

extracting the action posture of a teacher playing the video frame in the classroom;

generating an action gesture video matched with the audio according to the action gesture;

and generating the video according to the motion posture video.

Optionally, the motion pose comprises a facial motion pose;

the step of generating a motion gesture video matched with the audio according to the motion gesture comprises the following steps:

generating a simultaneous motion gesture video with the same time length as the audio according to the motion gesture;

generating a face action gesture video according to the audio;

and replacing the face video in the simultaneous motion gesture video with the face motion gesture video to obtain a motion gesture video matched with the audio.

To solve the above problem, an embodiment of the present invention further provides an interaction apparatus for a video classroom, including:

the student status data acquisition unit is suitable for acquiring student status data of a classroom;

the interactive video generation unit is suitable for generating an interactive video of a teacher according to classroom scene data at a trigger moment when the student state data meet an interactive trigger threshold, wherein the trigger moment is the moment when the student state data meet the interactive trigger threshold;

and the interactive video inter-cut unit is suitable for inter-cutting the interactive video after the current playing frame of the video playing in the classroom.

To solve the above problem, an embodiment of the present invention further provides an apparatus, including at least one memory and at least one processor; the memory stores a program, and the processor calls the program to execute the video classroom interaction method.

In order to solve the above problem, an embodiment of the present invention further provides a storage medium, where a program suitable for video classroom interaction is stored in the storage medium, so as to implement the video classroom interaction method.

Compared with the prior art, the technical scheme of the invention has the following advantages:

the video classroom interaction method, the video classroom interaction device, the video classroom interaction equipment and the storage medium provided by the embodiment of the invention are used for acquiring classroom student state data; when the student state data meet an interaction triggering threshold, generating an interaction video of a teacher according to classroom scene data at a triggering moment, wherein the triggering moment is the moment when the student state data meet the interaction triggering threshold; and inserting the interactive video after the current playing frame of the video playing in the classroom of the video classroom. According to the video classroom interaction method provided by the embodiment of the invention, the student state data of a classroom is continuously acquired in the video classroom process and is compared with the corresponding interaction trigger threshold, when the interaction trigger threshold is met, the generation of an interactive video is triggered, the interactive video needs to be generated based on classroom scene data at the trigger moment, and after the interactive video is generated, the interactive video is spliced to the current playing frame of the classroom playing video of the video classroom and then played. Therefore, the video classroom interaction method provided by the embodiment of the invention triggers the generation and playing of the interactive video according to the student state data in the video classroom proceeding process, determines whether interaction is needed or not based on the actual student state data, improves the necessity of interaction, can determine the number of times of interaction according to the actual student state data, is not influenced by the preset, and ensures the flexibility of interaction; meanwhile, the interactive video is generated based on the classroom scene data at the trigger moment, and can be closely attached to the classroom scene data, so that the interactive reality and credibility are improved, and the classroom learning atmosphere is ensured; furthermore, the interactive video is played after the current playing frame of the video playing video in the classroom at the triggering moment, which is equivalent to that the interactive video is played after the generation time of the interactive video is delayed backwards at the triggering moment.

Drawings

Fig. 1 is a schematic flow chart of a video classroom interaction method according to an embodiment of the present invention;

fig. 2 is another schematic flow chart of a video classroom interaction method according to an embodiment of the present invention;

fig. 3 is a schematic view of an alternative flow of a step of acquiring current posture data of each student according to all student images in the video classroom interaction method according to the embodiment of the present invention;

fig. 4 is a further flowchart of the video classroom interaction method according to the embodiment of the present invention;

fig. 5 is a block diagram of an interactive device for video classroom provided by the embodiment of the present invention;

fig. 6 is an alternative hardware device architecture of the device provided by the embodiment of the present invention.

Detailed Description

In the prior art, the interaction effect of a video classroom is poor.

In one scheme, the teacher may imagine the classroom response of the student according to the interactive scene set by the recorded video, and then record the following interactive response video, for example: the teacher indicates a word to enable the student to follow the reading, the student is supposed to read correctly, and a response video is recorded, namely the teacher says 'great', so that the interaction effect is achieved.

However, this solution can only realize the overall interaction in the classroom, and cannot perform directional interaction with specific individuals, and if the student does not answer or answers incorrectly during the interaction process, the interaction result may be mismatched, and further the credibility of the AI teacher and the learning atmosphere in the classroom are affected.

In another scheme, some interactive nodes are preset in the recorded video, and the interactive video is recorded according to the interactive mode and interactive content preset in each interactive node for the AI teacher, for example: holding hands and calling to answer questions, and calling students to answer questions by AI classroom teachers according to the set contents and judging the right and wrong.

The method can play a certain interaction role in a classroom, but the interaction content and mode need to be designed in advance, so the interaction times are less, and the interaction cannot be triggered according to the classroom condition.

In order to improve the interaction effect of a video classroom, the embodiment of the invention provides an interaction method and a related device of the video classroom, wherein the interaction method of the video classroom comprises the following steps: acquiring student state data of a classroom; when the student state data meet an interaction triggering threshold, generating an interaction video of a teacher according to classroom scene data at a triggering moment, wherein the triggering moment is the moment when the student state data meet the interaction triggering threshold; and inserting the interactive video after the current playing frame of the video playing in the classroom of the video classroom.

In this way, according to the video classroom interaction method provided by the embodiment of the present invention, in the video classroom progress process, student state data of a classroom is continuously obtained and compared with a corresponding interaction trigger threshold, when the interaction trigger threshold is met, generation of an interaction video is triggered, the interaction video needs to be generated based on classroom scene data at the trigger time, after the interaction video is generated, the interaction video is spliced to the current playing frame of the classroom playing video of the video classroom and then played, that is, in the interactive video generation process, the classroom playing video continues to be played forward, and after the interaction video is generated, the classroom playing video is spliced to the current playing frame of the classroom playing video and then played.

Therefore, according to the video classroom interaction method provided by the embodiment of the invention, the generation and playing of the interactive video are triggered according to the student state data in the video classroom proceeding process, and whether interaction is needed or not is determined based on the actual student state data in the classroom, so that the necessity of interaction is improved; the interaction times can be determined according to the actual student state data, the interaction is not influenced by the preset, and the interaction flexibility is ensured; meanwhile, the interactive video is generated based on the classroom scene data at the trigger moment, so that close fit with the classroom scene data is ensured, the interactive reality and credibility are improved, and the classroom learning atmosphere is ensured; furthermore, the interactive video is played after the current playing frame of the video playing video in the classroom at the triggering moment, which is equivalent to that the interactive video is played after the generation time of the interactive video is delayed backwards at the triggering moment.

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating a video classroom interaction method according to an embodiment of the present invention.

As shown in the figure, the video classroom interaction method provided by the embodiment of the present invention includes the following steps:

step S10: and acquiring student status data of a classroom.

In the process of video classroom, on one hand, the video player plays the classroom playing video of the teacher teaching, and meanwhile, the students watch the classroom playing video for learning. And in the course of the lessons, obtaining the student state data of the class in real time according to the setting.

Specifically, the camera can be used for shooting images of students in class in real time, and then analyzing the mode of student state data in the images to acquire the student state data in class.

The step of acquiring the student status data of the classroom may be performed from the beginning of the class of the video classroom, and in another specific embodiment, the student status data of the classroom may also be acquired according to a preset acquisition trigger standard, for example: the method comprises the steps of triggering the start and stop of obtaining the student status data in the classroom by presetting the starting point and the ending point of obtaining the student status data in the classroom playing video.

Such as: setting a trigger for stopping acquiring the student state data of the classroom at the video position where the teacher starts to explain the knowledge point, so as to stop acquiring the student state data of the classroom; and setting triggering for starting to acquire the student state data of the classroom at the video position of the teacher for answering the question, so as to realize starting to acquire the student state data of the classroom.

The method comprises the steps that student state data of a classroom are obtained according to a preset obtaining triggering standard, so that on one hand, interaction needs under a scene needing high-frequency interaction are guaranteed, and interaction effects are guaranteed; the other convenience is realized, the interaction under the scene that the interaction needs to be reduced is avoided, and the class quality is ensured; meanwhile, the transmission quantity and the operation quantity can be reduced, the requirement on operation equipment is lowered, and the cost is lowered.

Of course, it is easy to understand that, in order to ensure the interaction quality of the whole classroom, the step of acquiring student status data of the classroom may be continuously executed within a predetermined time range to be acquired in the video classroom process; similarly, the video classroom interaction method provided by the embodiment of the invention can be continuously performed within a preset time range needing to be acquired in the video classroom performing process.

In addition, it is understood that, in a video classroom, a plurality of students are generally gathered in a classroom, and therefore, the acquired student status data may include the overall status data of all the students or the status data of a specific student.

Step S11: and judging whether the student state data meets an interaction triggering threshold, if so, executing step S12, and if not, executing step S14.

After the student status data of the classroom is acquired, the student status data of the classroom is compared with a preset interaction trigger threshold, if the requirement of the interaction trigger threshold is met, the step S12 is executed, and if the requirement of the interaction trigger threshold is not met, the step S14 is executed, namely the student status data of the classroom is continuously acquired, and the next judgment is carried out.

That is, no matter whether the student status data meets the interaction triggering threshold, the student status data needs to be continuously acquired, so that the student status data can be ensured within the whole acquisition time, and meanwhile, the subsequent interactive video is generated and played under the condition that the interaction threshold is met.

It will be appreciated that the interaction triggering threshold corresponds to a particular type of student status data, and that the interaction triggering threshold is that of the whole student when the student status data is that of the whole student. The criteria for meeting the interaction trigger threshold are also different based on different student status data types, such as: when the student status data is student posture data, the satisfaction of the interaction trigger threshold value can be the same as a certain posture (such as conversation) in the interaction trigger threshold value; when the student status data is concentration, meeting the interaction trigger threshold may be less than a concentration threshold.

The interactive trigger threshold may be determined in advance, and the acquisition of the interactive trigger threshold may be obtained through a large amount of research, or may be continuously adjusted according to specific application conditions.

Step S12: and generating an interactive video of a teacher according to the classroom scene data of the trigger moment, wherein the trigger moment is the moment when the student state data meets an interactive trigger threshold.

When the student state data meet the interaction triggering threshold, the teacher interaction video needs to be generated according to the classroom scene data at the triggering moment, the specific triggering moment is the moment when the student state data meet the interaction triggering threshold, and if the student state data are acquired in a mode of acquiring all student images, the triggering moment can also be understood as the moment corresponding to all the acquired student images.

In order to ensure the authenticity of the interactive video, the interactive video needs to be generated according to the classroom scene data at the triggering moment. The classroom scene data can include the student status data and teacher status data in the classroom playback video, such as: at this time, the teacher state in the classroom playing video is that the teacher requires students to solve the problem on the book, the student state is that a certain student (which student is identified specifically) does not have the problem and does not solve the problem, and based on the classroom scene, the interactive video required to be generated is as follows: the teacher in the video indicates a video segment that a student should grasp the time to solve the problem.

Step S13: and inserting the interactive video after the current playing frame of the video playing in the classroom of the video classroom.

After the interactive video is obtained, the interactive video needs to be inserted into the classroom playing video for playing, and since a certain time is required from the triggering time to the complete generation of the interactive video, in the above process, the classroom playing video has been further played to a certain frame after the triggering time, that is: and playing the current playing frame after splicing the interactive video into the current playing frame.

Therefore, in the video classroom interaction method provided by the embodiment of the present invention, in the video classroom progress process, student state data of a classroom is continuously obtained and compared with a corresponding interaction trigger threshold, when the interaction trigger threshold is met, generation of an interaction video is triggered, the interaction video needs to be generated based on classroom scene data at the trigger time, and after the interaction video is generated, the interaction video is spliced to the current playing frame of the classroom playing video of the video classroom and then played, that is, in the interactive video generation process, the classroom playing video continues to be played forward, and after the interaction video is generated, the classroom playing video is spliced to the current playing frame of the classroom playing video and then played.

Step S14: and (5) stopping.

When the student state data meets the interaction triggering threshold, the generation of the teacher interaction video is stopped, and it can be understood that the video played in the played classroom is also continuously played.

In order to enable comparison of the student status data with the interactive trigger threshold and thus trigger generation of the interactive video, in one embodiment, the student status data may include student posture data.

Referring to fig. 2, fig. 2 is another flow chart of the video classroom interaction method according to the embodiment of the invention.

step S20: all student images of the classroom are acquired.

First, all student images in a classroom can be shot through the camera and then transmitted to a processor and other processing devices, and the processor acquires all student images in the classroom.

It is understood that all student images refer to images taken of all students in a class while they are in class, such as: when the student goes to class, 10 students are in class, and all student images are images of 10 students.

Step S21: and acquiring the current posture data of each student according to the all student images.

And analyzing all the student images to obtain the current posture data of the students. It is understood that the student current pose data refers to the pose of the student in all student images.

Since all student images include all students in a class, in an embodiment, in order to determine the current posture of each student, please refer to fig. 3, and fig. 3 is an optional flowchart illustrating a step of acquiring current posture data of each student according to all student images in the video classroom interaction method according to the embodiment of the present invention.

Step S311: and detecting and recognizing the face block diagram of each student in all the student images, and detecting the body block diagram of each student.

Because all student images contain all students in class, all faces and positions of the faces in all the student images can be detected by using a face detection algorithm to obtain a face block diagram, and further, the detected faces can be identified by using a face identification algorithm to realize the correspondence between the faces and names of the persons.

Meanwhile, a target detection algorithm is used for detecting the position corresponding to the body of each student in all the student images to obtain a body block diagram.

Step S312: selecting one of the face frame diagrams of each student to obtain a selected face frame diagram, calculating the overlapping degree of the selected face frame diagram and each body frame diagram to obtain a matched body frame diagram of which the overlapping degree accords with the overlapping degree threshold value, and matching the selected face frame diagram with the matched body frame diagram.

One of the face frame diagrams of each student is selected as a selected face frame diagram, and the overlapping degree of the selected face frame diagram and the body frame diagram is calculated. Such as: the area of the selected face frame diagram is A, the area of the overlapping area of the selected face frame diagram and one of the body frame diagrams is B, and the overlapping degree between the selected face frame diagram and the body frame diagram is as follows: B/A.

And calculating each overlapping degree of the selected face frame diagram and each different body frame diagram, comparing each overlapping degree with an overlapping degree threshold value, obtaining a body frame diagram corresponding to the overlapping degree meeting the overlapping degree threshold value, namely a matched body frame diagram, and matching the selected face frame diagram with the matched body frame diagram.

And if a plurality of overlapping degrees meeting the threshold value of the overlapping degrees are obtained, selecting the body frame diagram with the highest overlapping degree as the matching body frame diagram.

Step S313: and judging whether a non-matched face block diagram exists, if so, executing step S312, and if not, executing step S314.

And after matching of a selected face block diagram is completed, judging whether a face block diagram of a student which is not matched exists, if so, further matching is needed, executing a step S312, if not, matching of each face block diagram and each body block diagram is completed, the identity of the face block diagram can be given to the body block diagram, and then current posture data of each student is obtained according to the selected face block diagram and the matched body block diagram which are matched with each other.

Step S314: and obtaining the current posture data of each student according to the selected face block diagram and the matched body block diagram which are matched with each other.

And finishing the recognition of the face block diagram and the matching of each face block diagram and each body block diagram, and obtaining the current posture data of each student according to the selected face block diagram and the matched body block diagram which are matched with each other.

After obtaining the current posture data of each student according to all the student images by the method, further processing is performed, please continue referring to fig. 2, in an embodiment, step S22 in fig. 2 may be performed, in other embodiments, step S25 or step S27 may also be performed, and of course, any one, any two, or all three of step S22, step S25, and step S27 may also be performed.

When step S22 is executed, step S22: and judging whether at least one current posture data of the students meets the interaction triggering threshold of the student posture data, if so, executing step S23, and if not, executing step S210.

After obtaining the current posture data of each student, comparing the current posture data of the student with a student posture data interaction triggering threshold, for example, the student posture data triggering threshold includes: and when one hand lifting posture exists in the current posture data of each student, the current posture data of each student is the same as the student posture data interaction triggering threshold, the student posture data interaction triggering threshold is met, the step S23 is triggered and executed, and if the current posture data of each student are not met, the step S23 is not triggered and executed.

Step S23: generating an interactive video of a teacher according to classroom scene data of a trigger moment, wherein the trigger moment is a moment when the student state data meet an interactive trigger threshold

For details of step S23, please refer to step S12 shown in fig. 1, which is not described herein again.

Step S24: and inserting the interactive video after the current playing frame of the video playing in the classroom of the video classroom.

For details of step S24, please refer to step S13 shown in fig. 1, which is not described herein again.

When step S22 is executed, the current posture data of the student is used as a condition for triggering generation of the interactive video, the current posture data of the student is judged in time, and interaction with a specific individual can be realized on an individual basis.

In another specific embodiment, after obtaining the current posture data of each student, step S25 is executed:

step S25: and obtaining the overall concentration degree of the classroom according to all the current posture data of the students.

In a specific embodiment, the number ratio of the current posture data of different students can be obtained according to all the current posture data of the students; multiplying each quantity ratio by the weight of each student current attitude data to obtain each quantity weight ratio; and calculating the sum of the quantity weight ratios to obtain the overall concentration degree.

Specifically, based on the obtained current posture data of each student, the number of current posture data of different types of students is counted, such as: the writing state, the teacher watching state, the hand lifting state and the like are calculated according to the number of the current posture data of different students, and the ratio of the current posture data of the different students to the number of the whole students is calculated, so that the number ratio of the current posture data of the different students can be obtained.

And multiplying the quantity proportion of the current posture data of different students by the weight of the current posture data of the students determined in advance to obtain the weight proportion of each quantity.

Specifically, the weights of the current posture data of the students may be different according to the classroom scene data, such as: the teacher who plays in the classroom playing video is giving a lecture, so the weight of the state of watching the teacher is relatively high; when the teacher playing the video in the classroom requires the student to write, the weight of the writing state is relatively high.

Of course, it can be understood that the weight of the current posture data of the student can be continuously adjusted along with the increase of the number of the students, so that the calculation result can better meet the requirement of the trigger condition.

After the weight ratios of all the quantities are obtained, the sum of the weight ratios of all the quantities is calculated, and then the overall concentration degree of the students in the overall image of the students can be obtained.

Step S26: and judging that the overall concentration meets the overall concentration interaction triggering threshold, if so, executing step S23, and if not, executing step S210.

After the overall concentration degree is obtained, comparing the overall concentration degree with the overall concentration degree interaction triggering threshold, in a specific embodiment, defining that the overall concentration degree is greater than the overall concentration degree interaction triggering threshold as the overall concentration degree meets the overall concentration degree interaction triggering threshold, and if the overall concentration degree is greater than the overall concentration degree interaction triggering threshold, triggering to execute the step S23, and if the overall concentration degree is less than the overall concentration degree interaction triggering threshold, not triggering to execute the step S23.

Details of step S23 and step S24 are not repeated herein.

With whole concentration degree as the condition that triggers interactive video and generate, realized the assurance to student's the whole state of giving lessons, can realize the interdynamic in classroom based on holistic condition.

In another specific embodiment, after obtaining the current posture data of each student, step S27 is executed:

step S27: and acquiring the current posture data of each student at each moment within a preset time length before the moment of acquiring all the student images.

The current posture data of each student at the moment of acquiring all the student images are obtained, and then the current posture data of each student within a preset time before the moment of acquiring all the student images are further acquired.

Thus, for each student, a plurality of numbers of student current posture data over a period of time are obtained.

Specifically, the predetermined time length may be determined as needed, for example: 10S (seconds), 20S, etc.

Step S28: and obtaining the concentration degree of each person in the classroom according to the current posture data of each student at each moment.

In a specific implementation mode, firstly, according to the current posture data of each student at each moment, the time proportion of the current posture data of different students in the preset time length is obtained; and then multiplying each time ratio of the same student by the weight of the current posture data of each student to obtain the time weight ratio of each same student, and calculating the sum of each time weight ratio to obtain the individual concentration degree of the same student until each student is traversed to obtain the individual concentration degree.

After obtaining the current posture data of each student, obtaining the current posture data of each student at each time before the time of obtaining all the student images, such as: for the same student, obtaining the current posture data of the student at each moment within a preset time length before the moment of obtaining all the student images, and then counting the duration (i.e. the number of existing frames) of the current posture data of each student of the student, such as: the duration of the writing state, the duration of the teacher watching state, the duration of the hand-lifting state and the like, and calculating the time ratio of the current posture data of different students in a preset time length to obtain the time ratio of the current posture data of the students in the preset time length; and after the time ratios are obtained, multiplying the time ratios of the current posture data of different students of the student by the corresponding weights of the current posture data of each student to obtain the time weight ratios of each student, further calculating the sum of the time weight ratios to obtain the personal concentration degree of each student, and traversing all students to obtain the personal concentration degree of all students.

Specifically, the weights of the current posture data of the students may also be different according to the classroom scene data, such as: the teacher who plays in the classroom playing video is giving a lecture, so the weight of the state of watching the teacher is relatively high; when the teacher playing the video in the classroom requires the student to write, the weight of the writing state is relatively high.

Step S29: and judging whether at least one of the individual concentration degrees meets the individual concentration degree interaction triggering threshold value, if so, executing step S23, and if not, executing step S210.

After obtaining the individual concentration degree, comparing the individual concentration degree with the individual concentration degree interaction trigger threshold, in a specific embodiment, defining that the individual concentration degree is greater than the individual concentration degree interaction trigger threshold as the individual concentration degree meets the individual concentration degree interaction trigger threshold, when at least one individual concentration degree is greater than the individual concentration degree interaction trigger threshold, triggering to execute step S23, and if the individual concentration degree is less than the individual concentration degree interaction trigger threshold, triggering not to execute step S23 according to the condition.

Details of step S23 and step S24 are not repeated herein.

Step S210: and (5) stopping.

For details of step S210, please refer to step S14, which is not described herein again.

Of course, in another embodiment, after the personal concentration degree of one classmate is obtained, it is determined whether the obtained personal concentration degree meets the personal concentration degree interaction trigger threshold, and then the steps of obtaining the personal concentration degree of another classmate and determining are performed.

The personal concentration degree is used as a condition for triggering the generation of the interactive video, so that the state of the student in class can be grasped within a period of time, and the classroom interaction can be realized based on the condition of the student in continuous state.

When simultaneously taking the current posture data of the student, the overall concentration degree and the individual concentration degree as conditions for triggering the generation of the interactive video, any one of the conditions satisfies the corresponding triggering condition (at least one of the current posture data of the student satisfies the interactive triggering threshold of the posture data of the student, the overall concentration degree satisfies the interactive triggering threshold of the overall concentration degree and at least one of the individual concentration degree satisfies the interactive triggering threshold of the individual concentration degree respectively), the interactive video can be triggered and generated, when at least two of the conditions satisfy the triggering condition simultaneously, in a specific implementation mode, in order to guarantee the principle of priority of the overall student, the priority order is: the overall concentration degree is prior to the individual concentration degree, the individual concentration degree is prior to the current posture data of the student, in the classroom scene data of the trigger moment according to which the teacher interactive video is generated, the preferentially considered classroom scene data is that the overall concentration degree scene is prior to the individual concentration degree scene, and the individual concentration degree scene is prior to the current posture data scene of the student.

In order to ensure the effect of the generated teacher interaction video, an embodiment of the present invention further provides a video classroom interaction method, please refer to fig. 4, and fig. 4 is a further flowchart of the video classroom interaction method provided in the embodiment of the present invention.

step S40: and acquiring student status data of a classroom.

For details of step S40, please refer to step S10 shown in fig. 1, which is not described herein again.

Step S41: judging whether the student state data meet an interaction triggering threshold value, if so, executing a step S421; if not, step S44 is executed.

After the student status data of the classroom is acquired, the student status data is compared with a preset interaction trigger threshold, if the requirement of the interaction trigger threshold is met, the step S421 is executed, and if the requirement of the interaction trigger threshold is not met, the step S44 is executed. Please refer to step S11 shown in fig. 1 for other details of step S41, which are not described herein again.

Step S421: and generating audio of the teacher according to the classroom scene data at the trigger moment.

According to the classroom scene data of the trigger time, for example: student state data and the state of a teacher in a classroom playing video at the triggering moment are searched for language text data of the teacher under the classroom scene data.

For example: the student state data is that A student is speaking, and the state of teacher in the video played in class at the trigger moment is that teacher looks everywhere and does the question, and the language text data of teacher that needs to obtain this moment is promptly: the students do not speak and pay attention to do the subject.

The language text data of the teacher under the specific classroom scene data can be acquired by searching in the database.

After the language text data of the teacher is obtained, the language text data can be generated into the audio by using a speech synthesis technology, so that the length of the audio can be obtained.

Step S422: and generating a video of the teacher according to the trigger time and the audio.

In a specific embodiment, the video of the teacher may be generated by first obtaining a classroom playing video frame at the trigger time, then extracting an action posture of the teacher playing the classroom playing video frame, generating an action posture video matched with the audio according to the action posture, and generating the video according to the action posture video.

After the audio of the teacher is obtained, the video of the teacher needs to be generated, because the video of the teacher is finally used for generating the interactive video and is inserted into the classroom playing video for playing, the reality of the generated video of the teacher after being spliced with the classroom playing video needs to be ensured, therefore, the classroom playing video frame at the trigger moment can be obtained and used as the basis for generating the video of the teacher, and although the classroom playing video is further played for a period of time when the interactive video is specifically played, the action change in the classroom playing video can be considered to be very small due to short time, and the reality of playing after splicing is not influenced.

After obtaining the video frame played in the classroom at the triggering moment, the motion posture of the teacher can be detected by using a human posture detection algorithm, and the motion posture can be specifically represented by bone points, and the method comprises the following steps: facial skeleton points, body skeleton points, and hand skeleton points.

A motion gesture video is then generated that matches the audio based on the motion gesture, which is understood to include both the same length as the audio and a mouth-shape fit to the audio.

To this end, in one embodiment, a simultaneous motion gesture video may first be generated from the motion gesture for the same length of time as the audio; generating a face action posture video according to the audio; and finally, replacing the face video in the simultaneous action posture video with the face action posture video to obtain an action posture video matched with the audio.

Specifically, a simultaneous motion gesture video with the same time length as the audio is generated according to the motion gesture, the required number of video frames can be determined according to the time length of the audio, then, a character motion modeling method and other methods are used for obtaining the simultaneous motion gesture video, and because the mouth state of the simultaneous motion gesture video is not completely matched with the audio, according to the audio, a human face key point generation algorithm from the audio to a human face key point is adopted for generating the audio into human face key points corresponding to language text data, so that a face motion gesture video is obtained, and finally, the face motion gesture video in the simultaneous motion gesture video is replaced by the face motion gesture video, so that the motion gesture video matched with the audio can be obtained.

And after the action posture video is obtained, the video of the teacher corresponding to the action posture video is obtained, namely the video.

Step S423: and synthesizing the audio and the video to obtain the interactive video.

And after the audio and the video which are matched with each other are obtained, the audio and the video are synthesized to obtain the interactive video.

Step S43: and inserting the interactive video between the classroom video frame at the trigger moment and the next frame of the classroom video frame.

For details of step S43, please refer to step S13 shown in fig. 1, which is not described herein again.

Step S44, stop.

For details of step S44, please refer to step S13 shown in fig. 1, which is not described herein again.

By adopting the video classroom interaction method, the matching with classroom scene data is ensured, the authenticity of the interactive video is also ensured, and the interaction effect of the video classroom is improved.

In the following description of the video classroom interaction apparatus provided by the embodiment of the present invention, the video classroom interaction apparatus described below may be regarded as a functional module architecture that is required to be configured by an electronic device (e.g., a PC) to implement the video classroom interaction method provided by the embodiment of the present invention. The content of the interaction device for video classroom described below can be referred to in correspondence with the content of the interaction method for video classroom described above.

Fig. 5 is a block diagram of an interaction apparatus for video classroom provided in an embodiment of the present invention, where the interaction apparatus for video classroom can be applied to both a client and a server, and referring to fig. 5, the interaction apparatus for video classroom can include:

a student status acquisition unit 100 adapted to acquire student status data of a classroom;

the interactive video generating unit 110 is adapted to generate an interactive video of a teacher according to classroom scene data at a trigger time when the student status data meet an interactive trigger threshold, wherein the trigger time is a time when the student status data meet the interactive trigger threshold;

the interactive video insertion unit 120 is adapted to insert the interactive video after a currently playing frame of a classroom playing video of the video classroom.

Specifically, the student status acquiring unit 100 is adapted to acquire student status data of a classroom according to a preset acquisition trigger standard.

After acquiring student state data of a classroom, the student state data is sent to the interactive video generation unit 110 by the student state acquisition unit 100, the interactive video generation unit 110 firstly judges whether the student state data meets an interactive trigger threshold, and when the student state data meets the interactive trigger threshold, generates an interactive video of a teacher according to classroom scene data at the trigger moment, wherein the trigger moment is the moment when the student state data meets the interactive trigger threshold, and sends the interactive video to the interactive video insertion unit 120, and the interactive video insertion unit 120 inserts the interactive video after a current playing frame of a classroom playing video of the video classroom, so that the interaction of the video classroom is realized.

In particular, the classroom scene data can include the student status data and teacher status data in the classroom playback video, such as: at this time, the teacher state in the classroom playing video is that the teacher requires students to solve the problem on the book, the student state is that a certain student (which student is identified specifically) does not have the problem and does not solve the problem, and based on the classroom scene, the interactive video required to be generated is as follows: the teacher in the video indicates a video segment that a student should grasp the time to solve the problem.

It can be seen that the video classroom interaction device provided in the embodiment of the present invention continuously obtains student status data of a classroom during a video classroom process, compares the obtained student status data with a corresponding interaction trigger threshold, triggers generation of an interaction video when the interaction trigger threshold is met, generates the interaction video based on classroom scene data at a trigger time, and after the interaction video is generated, splices the interaction video to the classroom of the video classroom to play the current playing frame of the video.

Therefore, the video classroom interaction method provided by the embodiment of the invention triggers the generation and playing of the interactive video according to the student state data in the classroom proceeding process, determines whether interaction is needed or not based on the actual student state data, improves the necessity of interaction, can determine the number of times of interaction according to the actual student state data, is not influenced by the preset, and ensures the flexibility of interaction; meanwhile, the interactive video is generated based on the classroom scene data at the trigger moment, and can be closely attached to the classroom scene data, so that the interactive reality and credibility are improved, and the classroom learning atmosphere is ensured; furthermore, the interactive video is played after the current playing frame of the video is played in the classroom at the trigger moment, which is equivalent to that the interactive video is played after the generation time of the interactive video is delayed backwards at the trigger moment.

In one embodiment, the student status acquiring unit 100, adapted to acquire student status data of a classroom, may specifically include: acquiring all student images of a classroom; acquiring current posture data of each student according to all the student images;

the interactive video generating unit 110 is adapted to generate an interactive video of a teacher according to classroom scene data at a trigger time when the student status data satisfy an interactive trigger threshold, where the trigger time is a time when the student status data satisfy the interactive trigger threshold, and may specifically include: and when at least one of the current posture data of the students meets the student posture data interaction triggering threshold, generating the interaction video of the teacher according to the classroom scene data at the triggering moment.

Specifically, the student status acquiring unit 100 is adapted to acquire student status data of a classroom, and may specifically include: detecting and identifying a face block diagram of each student in all the student images, and detecting a body block diagram of each student; selecting one of the face block diagrams of each student to obtain a selected face block diagram, calculating the overlapping degree of the selected face block diagram and each body block diagram to obtain a matched body block diagram of which the overlapping degree accords with the threshold value of the overlapping degree, matching the selected face block diagram with the matched body block diagram until the face block diagrams of each student are traversed, and obtaining the current posture data of each student according to the selected face block diagram and the matched body block diagram which are matched with each other.

The current posture data of the student is used as a condition for triggering the generation of the interactive video, the current posture data of the student is judged very timely, and the interaction of specific individuals can be realized based on individuals.

In another embodiment, the student status acquiring unit 100 is adapted to acquire student status data of a classroom, and may specifically include: obtaining the whole concentration degree of the classroom according to all the current posture data of the students in the current postures; the interactive video generating unit 110 is adapted to generate an interactive video of a teacher according to classroom scene data at a trigger time when the student status data satisfy an interactive trigger threshold, where the trigger time is a time when the student status data satisfy the interactive trigger threshold, and specifically may include: and when the overall concentration degree meets the overall concentration degree interaction triggering threshold value, generating the teacher's interaction video according to the classroom scene data at the triggering moment.

Specifically, the student status acquiring unit 100, adapted to acquire student status data of a classroom, may specifically include: obtaining the number proportion of the current posture data of different students according to all the current posture data of the students; multiplying each quantity ratio by the weight of each student current attitude data to obtain each quantity weight ratio; and calculating the sum of the quantity weight ratios to obtain the overall concentration degree.

In another embodiment, the student status acquiring unit 100 is adapted to acquire student status data of a classroom, and may specifically include: acquiring current posture data of each student at each moment within a preset time length before the moment of acquiring all student images; according to the current posture data of each student at each moment, the concentration degree of each person in the classroom is obtained; the interactive video generating unit 110 is adapted to generate an interactive video of a teacher according to classroom scene data at a trigger time when the student status data satisfy an interactive trigger threshold, where the trigger time is a time when the student status data satisfy the interactive trigger threshold, and specifically may include: and when at least one personal concentration degree meets the personal concentration degree interaction triggering threshold value, generating the interaction video of the teacher according to the classroom scene data at the triggering moment.

Specifically, the student status acquiring unit 100, adapted to acquire student status data of a classroom, may specifically include: according to the current posture data of each student at each moment, obtaining the time proportion of the current posture data of different students in the preset time length; multiplying each time proportion of the same student by the weight of the current posture data of each student to obtain the time weight proportion of each same student, and calculating the sum of each time weight proportion to obtain the individual concentration degree of the same student until each student is traversed to obtain the individual concentration degree.

In a specific embodiment, the interactive video generating unit 110 is adapted to generate an interactive video of a teacher according to classroom scene data at a trigger time when the student status data meets an interactive trigger threshold, where the trigger time is a time when the student status data meets the interactive trigger threshold, and specifically may include: generating audio of a teacher according to the classroom scene data at the trigger moment; generating a video of a teacher according to the trigger time and the audio; and synthesizing the audio and the video to obtain the interactive video.

Specifically, the interactive video generating unit 110 is adapted to search, according to the classroom scene data at the trigger time, language text data of a teacher under the classroom scene data; generating the speech text data into the audio using speech synthesis techniques.

Specifically, the interactive video generating unit 110 is adapted to obtain a classroom playing video frame at the trigger time; extracting the action posture of a teacher playing the video frame in the classroom; generating an action gesture video matched with the audio according to the action gesture; and generating the video according to the motion posture video.

Specifically, the interactive video generating unit 110 is adapted to generate a simultaneous motion gesture video with the same time length as the audio according to the motion gesture; generating a face action gesture video according to the audio; and replacing the face video in the simultaneous motion gesture video with the face motion gesture video to obtain a motion gesture video matched with the audio.

Therefore, matching with classroom scene data is ensured, authenticity of interactive video is ensured, and interactive effect of video classroom is improved

The device provided by the embodiment of the invention can load the program module framework in a program form so as to realize the video classroom interaction method provided by the embodiment of the invention; the hardware device can be applied to an electronic device with specific data processing capacity, and the electronic device can be: such as a terminal device or a server device.

Optionally, fig. 6 shows an optional hardware device architecture of the device provided in the embodiment of the present invention, which may include: at least one memory 3 and at least one processor 1; the memory stores a program which is called by the processor to execute the video classroom interaction method, and in addition, at least one communication interface 2 and at least one communication bus 4; the processor 1 and the memory 3 may be located in the same electronic device, for example, the processor 1 and the memory 3 may be located in a server device or a terminal device; the processor 1 and the memory 3 may also be located in different electronic devices.

As an optional implementation of the disclosure in the embodiment of the present invention, the memory 3 may store a program, and the processor 1 may call the program to execute the video classroom interaction method provided in the above embodiment of the present invention.

In the embodiment of the invention, the electronic equipment can be a tablet computer, a notebook computer and other equipment capable of performing video classroom interaction.

In the embodiment of the present invention, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete mutual communication through the communication bus 4; it is clear that the communication connection of the processor 1, the communication interface 2, the memory 3 and the communication bus 4 shown in fig. 6 is only an alternative;

optionally, the communication interface 2 may be an interface of a communication module, such as an interface of a GSM module;

the processor 1 may be a central processing unit CPU or a Specific Integrated circuit asic (application Specific Integrated circuit) or one or more Integrated circuits configured to implement an embodiment of the invention.

The memory 3 may comprise a high-speed RAM memory and may also comprise a non-volatile memory, such as at least one disk memory.

It should be noted that the above terminal device may further include other devices (not shown) that may not be necessary for the disclosure of the embodiment of the present invention; these other components may not be necessary to understand the disclosure of embodiments of the present invention, which are not individually described herein.

An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and when the instructions are executed by a processor, the method for video classroom interaction may be implemented as described above.

The computer executable instruction stored in the computer readable storage medium provided by the embodiment of the invention triggers the generation and playing of the interactive video according to the student state data in the video classroom proceeding process, determines whether interaction is needed or not based on the actual student state data, improves the necessity of interaction, can determine the number of times of interaction according to the actual student state data, is not influenced by the preset, and ensures the flexibility of interaction; meanwhile, the interactive video is generated based on the classroom scene data at the trigger moment, and can be closely attached to the classroom scene data, so that the interactive reality and credibility are improved, and the classroom learning atmosphere is ensured; furthermore, the interactive video is played after the current playing frame of the video playing video in the classroom at the triggering moment, which is equivalent to that the interactive video is played after the generation time of the interactive video is delayed backwards at the triggering moment.

The embodiments of the present invention described above are combinations of elements and features of the present invention. Unless otherwise mentioned, the elements or features may be considered optional. Each element or feature may be practiced without being combined with other elements or features. In addition, the embodiments of the present invention may be configured by combining some elements and/or features. The order of operations described in the embodiments of the present invention may be rearranged. Some configurations of any embodiment may be included in another embodiment, and may be replaced with corresponding configurations of the other embodiment. It is obvious to those skilled in the art that claims that are not explicitly cited in each other in the appended claims may be combined into an embodiment of the present invention or may be included as new claims in a modification after the filing of the present application.

Embodiments of the invention may be implemented by various means, such as hardware, firmware, software, or a combination thereof. In a hardware configuration, the method according to an exemplary embodiment of the present invention may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, and the like.

In a firmware or software configuration, embodiments of the present invention may be implemented in the form of modules, procedures, functions, and the like. The software codes may be stored in memory units and executed by processors. The memory unit is located inside or outside the processor, and may transmit and receive data to and from the processor via various known means.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Although the embodiments of the present invention have been disclosed, the present invention is not limited thereto. Various changes and modifications may be effected therein by one skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.

Claims

1. An interaction method for a video classroom is characterized by comprising the following steps:

acquiring student state data of a classroom;

2. The method of claim 1, wherein the step of obtaining student status data of a classroom comprises:

acquiring all student images of a classroom;

3. The method of claim 2, wherein the step of obtaining student status data of a classroom further comprises:

obtaining the overall concentration degree of the classroom according to all the current posture data of the students;

4. The method of claim 3, wherein said step of obtaining an overall concentration of said classroom based on all of said student current posture data comprises:

5. The method of claim 2, wherein the step of obtaining student status data of a classroom further comprises:

6. The method of claim 5, wherein the step of obtaining the concentration degree of each person in the classroom according to the current posture data of each student at each time comprises:

7. The video classroom interaction method of any one of claims 2-6, wherein said step of obtaining current pose data for each student based on said all student images comprises:

8. The method for video classroom interaction of any one of claims 1-6, wherein said step of obtaining student status data for a classroom further comprises:

9. The video classroom interaction method of any one of claims 1-6, wherein the step of obtaining classroom scene data at the trigger time and generating the teacher interaction video comprises:

acquiring classroom scene data at a trigger moment, and generating audio of a teacher according to the classroom scene data at the trigger moment;

generating a video of a teacher according to the trigger time and the audio;

and synthesizing the audio and the video to obtain the interactive video.

10. The method for interacting in video classroom according to claim 9, wherein said step of generating teacher's audio according to classroom scene data at the time of said trigger comprises:

11. The method of claim 9, wherein the step of generating a video of the teacher according to the trigger time and the audio comprises:

acquiring a classroom playing video frame at the trigger moment;

and generating the video according to the motion posture video.

12. The video classroom interaction method of claim 11, wherein said action gesture comprises a facial action gesture;

generating a face action gesture video according to the audio;

13. The method for video classroom interaction of any one of claims 1-6, wherein said classroom scene data includes said student status data and teacher status data in said classroom playback video.

14. An interactive device for video classroom, comprising:

15. An apparatus comprising at least one memory and at least one processor; the memory stores a program that the processor calls to perform the video classroom interaction method of any of claims 1-13.

16. A storage medium storing a program suitable for video classroom interaction to implement the video classroom interaction method of any one of claims 1-13.