CN111462554A

CN111462554A - Online classroom video knowledge point identification method and device

Info

Publication number: CN111462554A
Application number: CN202010324019.7A
Authority: CN
Inventors: 张新华; 王朝选; 顾佳槟; 郭弘毅; 申会强; 吴高峰
Original assignee: Zhejiang Lancoo Technology Co ltd
Current assignee: Zhejiang Lancoo Technology Co ltd
Priority date: 2020-04-22
Filing date: 2020-04-22
Publication date: 2020-07-28

Abstract

The application relates to the field of teaching, and discloses a method and a device for identifying knowledge points of online classroom videos, which can accurately identify all knowledge points contained in classroom videos. The method comprises the following steps: determining a knowledge point domain of a classroom video; acquiring a classroom video, wherein the classroom video comprises a teaching courseware part and a blackboard writing part; for the teaching courseware part, dividing the courseware part into a plurality of sections of videos according to the switching time point of the courseware page, and identifying a first set of knowledge points contained in each section of video based on the knowledge point domain; for the blackboard writing part, dividing the blackboard writing part into a plurality of sections of videos according to the blackboard writing updating time point, and identifying a second set of knowledge points contained in each section of video based on the knowledge point domain; dividing the audio corresponding to each video into a plurality of audio segments according to the audio pause points, and identifying a third set of knowledge points contained in each audio segment based on the knowledge point domain; and calculating a set of knowledge points contained in the classroom video according to the first set, the second set and the third set.

Description

Online classroom video knowledge point identification method and device

Technical Field

The application relates to the field of teaching, in particular to an online classroom video knowledge point identification technology.

Background

The online classroom video greatly promotes the propagation of high-quality teaching resources, and is popular with users. With the popularization of class recording and broadcasting products, the number of online class video resources is increased, so that users are difficult to find video resources which are interested in the users from a huge number of video libraries. At present, the recognition accuracy rate has a larger space for scattered fragmentation of recognition results of knowledge points of online classroom videos.

Disclosure of Invention

The application aims to provide a method and a device for identifying knowledge points of an online classroom video, which can accurately identify all knowledge points contained in the classroom video.

The application discloses an online classroom video knowledge point identification method, which comprises the following steps:

determining a knowledge point domain of a classroom video to be identified;

acquiring the classroom video, wherein the classroom video comprises a teaching courseware part and a blackboard writing part;

for the teaching courseware part, dividing the courseware part into a plurality of sections of videos according to the switching time point of the courseware page, and identifying a first set of knowledge points contained in each section of video based on the knowledge point domain;

for the blackboard writing part, dividing the blackboard writing part into a plurality of sections of videos according to the blackboard writing updating time point, and identifying a second set of knowledge points contained in each section of video based on the knowledge point domain;

dividing the audio corresponding to each video into a plurality of audio segments according to audio pause points, and identifying a third set of knowledge points contained in each audio segment based on the knowledge point domain;

and calculating a set of knowledge points contained in the classroom video according to the first set, the second set and the third set.

In a preferred embodiment, before determining the knowledge point domain of the classroom video to be recognized, the method further includes:

the method comprises the steps that a curriculum schedule base and a knowledge point base are constructed in advance, attribute information of a curriculum schedule in the curriculum schedule base comprises a teaching date, a teaching time, a lesson leaving time, a teaching teacher, subjects and classes, and attribute information of knowledge points in the knowledge point base comprises subjects, grades, learning stages and keywords;

the determining of the knowledge point domain of the classroom video to be recognized further comprises:

presetting an automatic recording switch of the classroom video, wherein the starting time and the stopping time of the automatic recording switch respectively correspond to the class-on time and the class-off time in a class schedule of the classroom video;

responding to a recording switch starting signal of the classroom video, and acquiring a curriculum schedule of the classroom video from the curriculum schedule library;

determining subjects, grades and learning stages corresponding to the classroom videos according to the acquired curriculum schedule;

and acquiring the knowledge point domain of the classroom video from the knowledge point library according to the subject, grade and learning stage corresponding to the classroom video.

In a preferred example, the obtaining of the classroom video, where the classroom video includes a teaching courseware part and a blackboard writing part, further includes:

the method comprises the steps that a classroom video is obtained from a classroom video synthesis module, wherein the video synthesis module is used for obtaining a camera shooting video and a teaching computer screen video in a switching mode according to a preset rule, and synthesizing the camera shooting video corresponding to a blackboard writing part of a blackboard and the teaching computer screen video corresponding to a teaching courseware part into the classroom video;

the acquisition the classroom video, after the classroom video includes teaching courseware part and blackboard writing part, still include:

and switching and identifying the knowledge points in the teaching courseware part and the blackboard writing part in the classroom video according to the preset switching condition.

In a preferred embodiment, the switching and identifying the knowledge points in the teaching courseware part and the blackboard writing part in the classroom video according to the preset switching condition further includes:

switching from identifying knowledge points in the blackboard-writing portion to identifying knowledge points in the courseware portion when one of the following conditions is met:

receiving a signal to execute teaching software on a teaching computer for playing a teaching courseware,

a signal is received to turn on the projector executing on the teaching computer,

receiving a signal for operating a keyboard or a mouse on a teaching computer; and the number of the first and second groups,

switching from identifying knowledge points in the courseware section to identifying knowledge points in the blackboard-writing section when one of the following conditions is met:

receiving a signal to execute a shutdown of the tutorial software on the tutorial computer,

receiving a signal to turn off the projector on the teaching computer,

and recognizing that the black board area has occlusion.

In a preferred embodiment, before the teaching courseware section is divided into a plurality of sections of videos according to the courseware page switching time point, the method further includes:

recording the time points of operating a keyboard and a mouse on a teaching computer for the teaching courseware part;

acquiring images of a teaching computer screen of two frames before and after the time point;

performing gray mapping on the two frames of images before and after the image is subjected to difference, and performing binary mapping on the difference image to generate a binary image;

if the sum of the pixel numbers of the binary image is larger than a first preset threshold value, judging that the time point is a courseware page switching time point;

the identifying a first set of knowledge points for each section of video based on the domain of knowledge points, further comprising:

acquiring a teaching courseware page in each video corresponding to the teaching courseware part;

converting the teaching courseware page corresponding to each video into a text;

and identifying the knowledge points in the text according to a preset knowledge point identification algorithm based on the knowledge point domain to obtain the first set.

In a preferred embodiment, the recognizing, based on the knowledge point domain and according to a preset knowledge point recognition algorithm, knowledge points in a text to obtain the first set further includes:

splitting the text of the teaching courseware page corresponding to each video according to words or phrases, matching the split words or phrases with the keywords of the knowledge points in the knowledge point domain, and determining all the keywords and the occurrence frequency of the keywords in the text;

and determining the keywords with the occurrence frequency larger than a second preset threshold as the knowledge points of the text.

In a preferred embodiment, before dividing the blackboard-writing part into a plurality of sections of videos according to the blackboard-writing updating time point, the method further includes:

for the blackboard writing part, respectively carrying out gray mapping on a current image frame and a reference image frame of a video shot by a corresponding camera at preset time intervals, then carrying out difference, and carrying out binary mapping on the difference image to generate a binary image;

if the sum of the number of the pixels of the binary image is smaller than a third preset threshold value, judging that the blackboard writing is not updated, and updating the current image frame into the reference image frame, wherein the initial reference image frame is the first image frame after the camera is started;

if the sum of the number of the pixels of the binary image is larger than a fourth preset threshold value, determining that the blackboard writing is updated or a blackboard area is provided with a shelter;

dividing the binary image into a plurality of region blocks according to connectivity, and calculating and drawing a rectangular block capable of comprising the minimum area of each region block;

if the lower edge of a certain rectangular block is above the lower edge of the blackboard area, judging that the blackboard writing is updated, and if the lower edge of a certain rectangular block is below the lower edge of the blackboard area, judging that the blackboard area has a shelter;

the identifying a second set of knowledge points for each section of video based on the domain of knowledge points, further comprising:

acquiring an updated blackboard writing image frame in each video corresponding to the blackboard writing part;

extracting an effective area in the updated blackboard writing image frame and converting the effective area into a text;

and identifying the knowledge points in the text according to a preset knowledge point identification algorithm based on the knowledge point domain to obtain the second set.

In a preferred embodiment, based on the knowledge point domain, recognizing knowledge points in the text according to a preset knowledge point recognition algorithm to obtain the second set, further comprising:

splitting the text of the effective area of the updated blackboard writing image frame corresponding to each video according to words or phrases, matching the split words or phrases with the keywords of the knowledge points in the knowledge point domain, and determining all the keywords and the occurrence frequency of the keywords in the text;

In a preferred embodiment, before dividing the audio corresponding to each video into a plurality of audio segments according to an audio stop point, the method further includes:

monitoring the volume value of the audio in any unit time length for the audio corresponding to each section of video;

and if the volume value of the audio in a certain unit time length is smaller than a fifth preset threshold, the time point corresponding to the unit time length is a stop point for the teacher to explain, wherein the fifth preset threshold is determined according to the average value of the volume of the audio corresponding to each video.

In a preferred embodiment, the calculating the set of knowledge points included in the classroom video according to the first set, the second set, and the third set further includes:

and intersecting the third set corresponding to each audio clip with the first set or the second set in the corresponding time clip to obtain a set of knowledge points contained in the classroom video.

The application also discloses online classroom video knowledge point identification system includes:

the determining module is used for determining the knowledge point domain of the classroom video to be identified;

the identification module is used for acquiring the classroom video, wherein the classroom video comprises a teaching courseware part and a blackboard writing part, the teaching courseware part is divided into a plurality of sections of videos according to courseware page switching time points, a first set of knowledge points contained in each section of video is identified based on knowledge point domains, the blackboard writing part is divided into a plurality of sections of videos according to blackboard writing updating time points, a second set of knowledge points contained in each section of video is identified based on the knowledge point domains, audio corresponding to each section of video is divided into a plurality of audio segments according to audio pause points, and a third set of knowledge points contained in each audio segment is identified based on the knowledge point domains;

and the calculation module is used for calculating a set of knowledge points contained in the classroom video according to the first set, the second set and the third set.

The application also discloses a device includes:

a camera for shooting a video covering a blackboard area and a teacher's activity area on a podium;

the teaching computer is used for recording a screen for showing the teaching courseware into a video-form teaching courseware part;

the video synthesis module is used for switching and acquiring the video shot by the camera corresponding to the blackboard writing part and the video of the screen of the teaching computer corresponding to the teaching courseware part according to a preset rule and synthesizing the videos into the classroom video;

and the online classroom video knowledge point identification module is used for describing knowledge points in the classroom video according to the foregoing after the classroom video is acquired from the video synthesis module.

a memory for storing computer executable instructions; and the number of the first and second groups,

a processor for implementing the steps in the method as described hereinbefore when executing the computer-executable instructions.

The present application also discloses a computer-readable storage medium having stored therein computer-executable instructions which, when executed by a processor, implement the steps in the method as described above.

Compared with the prior art, the embodiment of the application at least comprises the following advantages and effects:

the method comprises the steps of respectively identifying knowledge points of a teaching courseware part, a blackboard writing part and a teaching audio part in a classroom video, taking a courseware page switching time point as a video branch point of the teaching courseware part, taking a blackboard writing updating time point as a video branch point of the blackboard writing part, dividing an audio corresponding to each video obtained by dividing into a plurality of audio segments according to an audio pause point, identifying the knowledge points contained in each video and the knowledge points contained in each audio segment according to a determined knowledge point domain of the classroom video, and accurately identifying all the knowledge points contained in the classroom video.

Furthermore, a recording switch is arranged on the class video in advance according to the information of the class schedule, the class schedule of the class video is obtained from a class schedule library in response to a starting signal of the recording switch of the class video, the knowledge point domain of the class video is determined according to the subject, the grade and the learning stage in the class schedule, the reliability of the determined knowledge point domain is high, and when the knowledge points contained in each video and the knowledge points contained in each audio clip are identified according to the knowledge point domain, the identification range of the knowledge points is effectively reduced, and the identification efficiency is improved.

Furthermore, when the knowledge point base is constructed, keywords are set for each knowledge point in the knowledge point base, when the knowledge point contained in each video is identified according to the knowledge point domain of the determined classroom video, the effective area in the teaching courseware page or blackboard writing corresponding to each video is converted into a text, then the text is split according to words or phrases, and finally the split words and phrases are matched with the keywords of the knowledge points in the knowledge point domain one by one, so that the identification accuracy and reliability can be improved.

Furthermore, the intersection of the knowledge points corresponding to each audio segment and the knowledge points of each audio segment in the corresponding time segment is taken to obtain the knowledge points contained in the classroom video and the time period thereof, so that the classroom video can be secondarily processed, for example, the video segment corresponding to each audio segment is associated with teaching resources such as courseware, teaching plans, exercises and the like, and the learning efficiency of the user can be improved.

The present specification describes a number of technical features distributed throughout the various technical aspects, and if all possible combinations of technical features (i.e. technical aspects) of the present specification are listed, the description is made excessively long. In order to avoid this problem, the respective technical features disclosed in the above summary of the invention of the present application, the respective technical features disclosed in the following embodiments and examples, and the respective technical features disclosed in the drawings may be freely combined with each other to constitute various new technical solutions (which are considered to have been described in the present specification) unless such a combination of the technical features is technically infeasible. For example, in one example, the feature a + B + C is disclosed, in another example, the feature a + B + D + E is disclosed, and the features C and D are equivalent technical means for the same purpose, and technically only one feature is used, but not simultaneously employed, and the feature E can be technically combined with the feature C, then the solution of a + B + C + D should not be considered as being described because the technology is not feasible, and the solution of a + B + C + E should be considered as being described.

Drawings

Fig. 1 is a schematic flow chart of a method for identifying knowledge points of online classroom video according to a first embodiment of the present application;

FIG. 2 is a schematic diagram of one example of a difference binary map of gray maps of a current image frame and a reference image frame of a video captured by a camera according to the present application;

fig. 3 is a flowchart illustrating a method for identifying knowledge points of an online classroom video according to a second embodiment of the present application;

fig. 4 is a flowchart illustrating a method for identifying points of knowledge in an online classroom video according to a third embodiment of the present application.

Detailed Description

In the following description, numerous technical details are set forth in order to provide a better understanding of the present application. However, it will be understood by those skilled in the art that the technical solutions claimed in the present application may be implemented without these technical details and with various changes and modifications based on the following embodiments.

Description of partial concepts:

an effective area: the blackboard area has a closed area of complete sub-blackboards.

To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.

A first embodiment of the present application relates to a method for identifying points of knowledge in online classroom video, a flow of which is shown in fig. 1, and the method includes the following steps:

in step 101, a knowledge point domain of a classroom video to be identified is determined.

Optionally, before step 101, the method further includes the following steps:

and a curriculum schedule base and a knowledge point base are constructed in advance. The attribute information of the curriculum schedule in the curriculum schedule library comprises the date of lectures, the time of lessons, a teacher giving lessons, subjects and classes. The "keyword" refers to a synonym related to the knowledge point and used for searching for a match, and the "learning stage" refers to the sequence of teaching teachers teaching the knowledge points according to the teaching outline, for example, all the knowledge points of a certain subject in a certain grade may be subdivided into a plurality of learning stages in advance. The attribute information of the knowledge points in the knowledge point base comprises subjects, grades, learning stages and keywords. The knowledge point library stores, in advance, course-related information for all classrooms in a predetermined area (for example, but not limited to, a school area, an area of a regional education office, and the like). The following table 1 shows attribute information corresponding to "knowledge point-oxygen element". It should be noted that the details listed in table 1 are mainly for easy understanding and are not intended to limit the scope of the present application.

TABLE 1

Optionally, step 101 may further include ①②③④, in step ①, presetting an automatic recording switch of the classroom video, the starting time and the stopping time of the automatic recording switch respectively corresponding to the time of getting on class and the time of getting off class in the schedule of the classroom video, then executing step ②, in response to the recording switch starting signal of the classroom video, acquiring the schedule of the classroom video from the schedule library, then executing step ③, determining the subject, the grade and the learning stage corresponding to the classroom video according to the attribute information of the schedule, then executing step ④, acquiring the knowledge point field of the classroom video from the knowledge point library according to the subject, the grade and the learning stage corresponding to the schedule, wherein the knowledge point field may be a set of all knowledge points of a certain subject of a certain grade under a certain learning stage.

TABLE 2

Then, step 102 is entered to obtain the classroom video, wherein the classroom video comprises a teaching courseware part and a blackboard writing part.

Optionally, step 102 may be further implemented as: and acquiring the classroom video from a classroom video synthesis module, wherein the video synthesis module is used for switching and acquiring the camera shooting video and the teaching computer screen video according to a preset rule, and synthesizing the camera shooting video corresponding to the blackboard writing part and the teaching computer screen video corresponding to the teaching courseware part into the classroom video. The camera is used to capture video including a blackboard area, such as but not limited to video covering the blackboard area and teacher's activity area on the podium; the teaching computer is used for recording and playing videos of screens of teaching courseware.

Optionally, after step 102, the following steps are further included:

Optionally, the "switching and identifying the knowledge points in the teaching courseware part and the blackboard-writing part in the classroom video according to the preset switching condition" may further include the following steps:

switching from recognizing the knowledge point in the blackboard-writing part to recognizing the knowledge point in the teaching course part when one of the conditions is satisfied, ① receiving a signal to execute on the teaching computer to turn on teaching software for playing a teaching course, ② receiving a signal to execute on the teaching computer to turn on a projector, ③ receiving a signal to operate a keyboard or a mouse on the teaching computer, and,

switching from identifying knowledge points in the courseware section to identifying knowledge points in the blackboard-writing section occurs when one of the conditions is met ① receiving a signal to turn off the teaching software executing on the teaching computer ② receiving a signal to turn off the projector executing on the teaching computer ③ identifying an occlusion in the blackboard area.

Then, step 103 is entered, for the teaching courseware part, the teaching courseware part is divided into a plurality of sections of videos according to the courseware page switching time point, and a first set of knowledge points contained in each section of video is identified based on the knowledge point domain, and for the blackboard-writing part, the teaching courseware part is divided into a plurality of sections of videos according to the blackboard-writing updating time point, and a second set of knowledge points contained in each section of video is identified based on the knowledge point domain.

For the identification process of the knowledge points of the courseware section in step 103:

in one embodiment (embodiment a), before "for the teaching courseware section, it is divided into a plurality of videos according to the courseware page switching time point" in step 103, the following step ①②③ is further included:

in step ①, for the teaching courseware part, recording the time points of the teaching teacher operating the keyboard and mouse on the teaching computer, obtaining the images of the two frames of teaching computer screen before and after the time point, then step ②, making difference after the gray level imaging of the two frames of images before and after the time point, and performing binary imaging processing on the difference image to generate a binary image, then step ③, if the sum of the pixel number of the binary image is greater than a first preset threshold, determining the time point as a courseware page switching time point, wherein the first preset threshold can be set according to the need.

In one embodiment (embodiment B), the "identifying the first set of knowledge points included in each video based on the knowledge point domain" in step 103 may be further implemented as: firstly, acquiring a teaching courseware page in each video corresponding to the teaching courseware part; then, converting the teaching courseware page corresponding to each video into a text; and then, based on the knowledge point domain, recognizing the knowledge points in the text according to a preset knowledge point recognition algorithm to obtain the first set. Optionally, the "identifying the knowledge points in the text based on the knowledge point domain according to a preset knowledge point identification algorithm to obtain the first set" is further implemented as follows: firstly, splitting a text of the teaching courseware page corresponding to each section of video according to words (such as English subject) or phrases (such as Chinese subject), matching the split words or phrases with keywords of knowledge points in a knowledge point domain, and determining all keywords and the occurrence frequency thereof in the text; and then, determining the keywords with the occurrence frequency larger than a second preset threshold as the knowledge points of the text. Wherein, the second preset threshold value can be set according to the requirement.

To better understand the above "recognition process of knowledge points of a course section", a "video of a 45-minute period corresponding to the course section" is taken as a specific example, wherein the video has 4 pages in a course page (such as PPT format), and the recognition process of knowledge points of the example video includes: firstly, according to the above embodiment a, the courseware page switching time points of the example video are respectively determined as the 6 th minute, the 10 th minute and the 25 th minute, and the example video is divided into the corresponding 4 sections of videos according to the determined time points; then, it is recognized according to the above embodiment B that the knowledge points included in the teaching courseware in the page of 4 sections of video are corresponding to the first sets of B1, B2, B3 and B4, and then it can be known that the teacher is the 1 st section of video 6 minutes before the video, the set of knowledge points of the explained content is B1, the 2 nd section of video 6 minutes to 10 th minute, the set of knowledge points of the explained content is B2, and so on.

The identification process of the knowledge points of the blackboard-writing part in step 103 is as follows:

optionally, before "for the blackboard-writing part, dividing the blackboard-writing part into a plurality of video sections according to the blackboard-writing updating time point" in step 103, the following steps a and b are further included: in the step a, for the blackboard writing part, respectively making the gray level of the current image frame and the reference image frame of the video shot by the corresponding camera image into a difference at preset time intervals, and performing binary imaging processing on the difference image to generate a binary image; and b, executing step b, judging whether the blackboard writing is updated according to the binary image, and determining the blackboard writing updating time point.

Wherein, the preset time can be set according to the situation. In one embodiment, the preset time may be set according to the shortest time between two writing of the blackboard writing by the teacher; for example, if the shortest time between the teacher writing the board twice is t, the preset time may be set to ≦ t. In another embodiment, the preset time may be set empirically; for example, considering that the picture frame rate in the standard video is 25 frames/second, the preset time may be set to 1s, 2s, 3s, and the like.

Optionally, the step b may be further implemented as: comparing the sum of the number of pixels of the binary image with a third threshold and a fourth threshold, if the sum of the number of pixels of the binary image is smaller than a third preset threshold, judging that the blackboard writing is not updated, and updating the current image frame into the reference image frame, wherein the initial reference image frame is the first image frame after the camera is started; if the sum of the number of the pixels of the binary image is larger than a fourth preset threshold value, determining that the blackboard writing is updated or a shelter is arranged in the blackboard area; and then dividing the binary image into a plurality of area blocks according to connectivity, calculating and drawing a rectangular block which can comprise the minimum area of each area block, judging that the blackboard eraser is updated if the lower edge of a certain rectangular block is above the lower edge of the blackboard area, and judging that the blackboard area has a shelter if the lower edge of a certain rectangular block is below the lower edge of the blackboard area. The third preset threshold and the fourth preset threshold may be set as needed. For example, the following steps are carried out: as shown in fig. 2, an example of a binary map of the difference between the gray map of the current image frame and the gray map of the reference image frame of the video captured by the camera is shown, in which the lower edge of the rectangular block corresponding to the first white region block is located above the lower edge of the black region for clear distinction, and the lower edge of the rectangular block corresponding to the second white region block (larger than the area of the first white region block) is located below the lower edge of the black region for movement of the teacher (blocking object).

In one embodiment, the blackboard may be pre-calibrated with the camera in a manual calibration manner to determine the blackboard area before the camera captures the video.

Optionally, the "identifying the second set of knowledge points included in each video based on the knowledge point domain" in step 103 may be further implemented as: firstly, acquiring an image frame of an updated blackboard writing in each video corresponding to the blackboard writing part; then, extracting an effective area in the updated blackboard writing image frame and converting the effective area into a text; and then, based on the knowledge point domain, recognizing the knowledge points in the text according to a preset knowledge point recognition algorithm to obtain the second set.

Optionally, the "extracting the effective area in the updated blackboard-writing image frame" may be further implemented as step A, B, C, D, in step a, converting the image frame from an RGB color space to a L ab color space, extracting a B-channel image in the L ab color space, then performing step B, dividing the B-channel image into a background and a blackboard border according to the gray characteristic of the image, then performing step C, performing binarization processing on the blackboard border to obtain a border binary image, and then performing step D, confirming the effective area in the blackboard area according to the border binary image, and extracting.

Optionally, the steps B and C can be further implemented by firstly determining the size of the B-channel image to be M × N, and the proportion of the number of pixels of the blackboard frame to the whole image to be omega₀Average gray of μ₀The ratio of the number of background pixels to the whole image is omega₁Average gray of μ₁The total average gray scale of the b-channel image is recorded as mu; then, setting a threshold T, wherein the number of pixels in the image with the gray scale value smaller than the threshold T is N0, and the number of pixels with the gray scale value larger than the threshold T is N1, then:

ω₀＝N₀/M×N (1)

ω₁＝N₁/M×N (2)

N₀+N₁＝M×N (3)

ω₀+ω₁＝1 (4)

μ＝ω₀*μ₀+ω₁*μ₁(5)

g＝ω₀(μ₀-μ)²+ω₁(μ₁-μ)²(6)

substituting formula (5) for formula (6) yields the equivalent formula:

g＝ω₀ω₁(μ₀-μ₁)²(7) obtaining a threshold T which maximizes g by adopting a traversal method, wherein the threshold T is an optimal binarization threshold; and then, carrying out binarization processing on the blackboard frame according to the threshold value T to obtain a frame binary image.

The blackboard can be a single-connection blackboard or a sub-blackboard with the same multiple-connection specification. In one embodiment, the blackboard is a multiple sub-blackboard with the same specification, and in this embodiment, the "extracting the effective area in the updated blackboard writing image frame" may be further implemented as: pre-framing a complete sub-blackboard in a shooting picture of the camera, and calculating an image pixel value of the complete sub-blackboard; then executing the steps B and C; then obtaining a blackboard frame form image of the current image frame according to the frame binary image, and supplementing a certain area into a closed area if the lower edge of the area in the blackboard frame form image is a non-closed curve; calculating the image pixel value of each closed area in the blackboard frame form graph; if the absolute value of the difference value between the image pixel value of a certain closed area and the image pixel value of the complete sub-blackboard is smaller than a sixth preset threshold value, identifying the closed area; calculating the color average value of all pixels in each identified closed area and the color average value of all pixels in the black board area; and if the absolute value of the difference value between the color average value of all the pixels in a certain identified closed area and the color average value of all the pixels in the blackboard area is less than a seventh preset threshold value, determining the closed area as the effective area.

Optionally, the "recognizing the knowledge points in the text according to the preset knowledge point recognition algorithm based on the knowledge point domain to obtain the second set" may be further implemented as: firstly, splitting a text of an effective area of the updated blackboard writing image frame corresponding to each video according to words or phrases, matching the split words or phrases with keywords of knowledge points in a knowledge point domain, and determining all the keywords and the occurrence frequency of the keywords in the text; and then, determining the keywords with the occurrence frequency larger than a second preset threshold as the knowledge points of the text. Wherein, the second preset threshold value can be set according to the requirement.

It should be noted that the application can, but is not limited to, convert the blackboard writing in the effective area into text by using an OCR algorithm.

Then, step 104 is entered, the audio corresponding to each video section is divided into a plurality of audio segments according to the audio pause point, and a third set of knowledge points included in each audio segment is identified based on the knowledge point domain.

Optionally, before step 104, the following steps are further included:

for example, the unit time length is 1 second, the average value of the volume of the audio corresponding to a certain section of example video is calculated to be N, for the audio corresponding to the section of example video, the volume value of the audio per second is monitored, and if the volume of the audio in the nth second is less than η × N, the time point corresponding to the nth second is the pause point of the teacher interpretation, wherein η and N are configurable parameters.

Then, step 105 is performed to calculate a set of knowledge points included in the classroom video according to the first set, the second set, and the third set.

Optionally, step 105 is further implemented as: and intersecting the third set corresponding to each audio clip with the first set or the second set in the corresponding time clip to obtain a set of knowledge points contained in the classroom video.

A second embodiment of the present application relates to an online classroom video knowledge point identification system, the structure of which is shown in fig. 3, and the online classroom video knowledge point identification system includes a determination module, an identification module, and a calculation module.

The determining module is used for determining the knowledge point domain of the classroom video to be identified.

Optionally, the system further comprises a building module for building the curriculum schedule base and the knowledge point base. The curriculum schedule attribute information in the curriculum schedule library comprises a curriculum teaching date, a curriculum taking time, a curriculum teaching teacher, subjects and classes. The "keyword" refers to a synonym related to the knowledge point and used for searching for a match, and the "learning stage" refers to the sequence of teaching teachers teaching the knowledge points according to the teaching outline, for example, all the knowledge points of a certain subject in a certain grade may be subdivided into a plurality of learning stages in advance. The knowledge point attribute information in the knowledge point library comprises subjects, grades, learning stages and keywords. The knowledge point library stores, in advance, course-related information for all classrooms in a predetermined area (for example, but not limited to, a school area, an area of a regional education office, and the like). The above table 1 shows attribute information corresponding to "knowledge point-oxygen element". It should be noted that the details listed in table 1 are mainly for easy understanding and are not intended to limit the scope of the present application.

Optionally, the determining module is further configured to preset an automatic recording switch for the classroom video, a start time and a stop time of the automatic recording switch correspond to a time of getting on a class and a time of getting off a class in a curriculum schedule of the classroom video respectively, obtain the curriculum schedule of the classroom video from the curriculum schedule library in response to a start signal of the recording switch for the classroom video, determine a subject, a grade, and a learning stage corresponding to the classroom video according to attribute information of the curriculum schedule, and obtain a knowledge point domain of the classroom video from the knowledge point library according to the subject, the grade, and the learning stage corresponding to the curriculum schedule. The knowledge point domain may be a set of all knowledge points of a certain subject of a certain grade in a certain learning stage. For example, the classroom video includes the information as shown in table 2 above, wherein the recording time corresponds to the learning stage, that is, the learning stage can be determined by the recording time through table lookup, and the knowledge point domain corresponding to the video can be obtained through the learning stage, the grade, and the subject information. It should be noted that the details listed in table 2 are mainly for easy understanding and are not intended to limit the scope of the present application.

The recognition module is used for acquiring the classroom video, wherein the classroom video comprises a teaching courseware part and a blackboard writing part, the teaching courseware part is divided into a plurality of sections of videos according to courseware page switching time points, a first set of knowledge points contained in each section of video is recognized based on knowledge point domains, the blackboard writing part is divided into a plurality of sections of videos according to blackboard writing updating time points, a second set of knowledge points contained in each section of video is recognized based on the knowledge point domains, audio corresponding to each section of video is divided into a plurality of audio segments according to audio pause points, and a third set of knowledge points contained in each audio segment is recognized based on the knowledge point domains.

Optionally, the recognition module is further configured to acquire the classroom video from a video synthesis module, where the video synthesis module is configured to switch between acquiring the camera shooting video and the teaching computer screen video according to a preset rule, and synthesize the camera shooting video corresponding to the blackboard-writing portion and the teaching computer screen video corresponding to the teaching courseware portion into the classroom video. Wherein the camera is used to capture video including a blackboard area, such as but not limited to video covering the blackboard area and the teacher's activity area on the podium, and the teaching computer is used to record video of the screen playing the teaching courseware.

Optionally, the identification module is further configured to switch to identify the knowledge points in the teaching courseware section and the blackboard-writing section in the classroom video according to the preset switching condition, further optionally, the identification module is further configured to switch from identifying the knowledge points in the blackboard-writing section to identifying the knowledge points in the teaching courseware section by ① receiving a signal to execute on a teaching computer to turn on teaching software for playing a teaching, ② receiving a signal to execute on a teaching computer to turn on a projector, ③ receiving a signal to operate a keyboard or a mouse on a teaching computer, and when one of the following conditions is satisfied, switching from identifying the knowledge points in the teaching courseware section to identifying the knowledge points in the blackboard-writing section by ① receiving a signal to execute on a teaching computer to turn off the teaching software, ② receiving a signal to execute on a teaching computer to turn off the projector, ③ identifying that there is an occlusion in the blackboard area.

In an embodiment (embodiment C), the identification module is further configured to record, for the courseware part, a time point when a teacher operates a keyboard and a mouse on a teaching computer, acquire images of two frames of teaching computer screens before and after the time point, perform difference after gray-level imaging on the two frames of images, perform binary imaging on the difference image to generate a binary image, and determine that the time point is a courseware page switching time point if a sum of numbers of pixels of the binary image is greater than a first preset threshold. The first preset threshold value can be set as required.

In one embodiment (embodiment D), the identification module is further configured to obtain a teaching course page in each video corresponding to the teaching course part, convert the teaching course page corresponding to each video into a text, and identify a knowledge point in the text according to a preset knowledge point identification algorithm based on the knowledge point domain to obtain the first set. Optionally, the recognition module is further configured to split a text of the teaching courseware page corresponding to each section of video according to words or phrases, match the split words or phrases with keywords of the knowledge points in the knowledge point domain, determine all keywords in the text and occurrence frequency thereof, and determine the keywords having occurrence frequency greater than a second preset threshold as the knowledge points of the text.

Taking "a 45-minute video corresponding to the teaching session part" as an example, wherein the teaching session page (such as PPT format) in the example video has 4 pages, specifically, the recognition module is configured to determine that the time points of session page switching are respectively 6 th minute, 10 th minute and 25 th minute according to the above embodiment C, and divide the example video into 4 corresponding videos according to the time points, and recognize the knowledge points contained in the teaching session in the 4 videos according to the above embodiment D to obtain the corresponding first sets B1, B2, B3 and B4. Then it can be seen that the teacher was section 1 video 6 minutes before the example video, the set of knowledge points for the content being taught was B1, the section 2 video 6 minutes through 10 minutes, the set of knowledge points for the content being taught was B2, and so on.

Optionally, the identification module is further configured to, for the blackboard writing part, perform difference after graying mapping the current image frame and the reference image frame of the video captured by the corresponding camera at preset time intervals, perform binary mapping processing on the difference image to generate a binary image, determine whether the blackboard writing is updated according to the binary image, and determine a blackboard writing updating time point.

Optionally, the identification module is further configured to compare the sum of the pixel numbers of the binary image of the blackboard-writing portion with a third threshold and a fourth threshold, determine that the blackboard-writing portion is not updated if the sum of the pixel numbers of the binary image is smaller than the third preset threshold, update the current image frame to the reference image frame, where the initial reference image frame is the first image frame after the camera is turned on, determine that the blackboard-writing portion is updated or the blackboard area has a blocking object if the sum of the pixel numbers of the binary image is greater than the fourth preset threshold, divide the binary image into a plurality of area blocks according to connectivity, calculate and draw a rectangular block capable of including a minimum area of each area block, determine that the blackboard-writing portion is updated if a lower edge of a certain rectangular block is above a lower edge of the blackboard area, and determine that the lower edge of a certain rectangular block is below the lower edge of the blackboard area, then the blackboard area is judged to have a shelter. The third preset threshold and the fourth preset threshold may be set as needed. In one embodiment, the blackboard may be pre-calibrated with the camera in a manual calibration manner to determine the blackboard area before the camera captures the video.

For example, fig. 2 shows an example of a binary map of the difference between a gray map of a current image frame and a gray map of a reference image frame of a video captured by a camera, in which a blackboard area is framed by a gray rectangular frame, in which the lower edge of a rectangular block corresponding to a first white area block is above the lower edge of the blackboard area and is caused by blackboard writing update, and the lower edge of a rectangular block corresponding to a second white area block (having a larger area than the first white area block) is below the lower edge of the blackboard area and is caused by movement of a teacher (a blocking object).

Optionally, the identification module is further configured to acquire an updated blackboard writing image frame in each section of video corresponding to the blackboard writing portion, extract an effective area in the updated blackboard writing image frame and convert the effective area into a text, and identify a knowledge point in the text according to a preset knowledge point identification algorithm based on the knowledge point domain to obtain the second set.

Optionally, the identification module is further configured to convert the updated blackboard writing image frame from an RGB color space to an L ab color space, extract a b-channel image in the L ab color space, divide the b-channel image into a background and a blackboard border according to a gray characteristic of the image, perform binarization processing on the blackboard border to obtain a border binary image, and determine and extract an effective area in the blackboard area according to the border binary image.

Optionally, the identification module is further configured to determine that the size of the b-channel image is M × N, and the ratio of the number of pixels of the blackboard border to the whole image is ω₀Average gray of μ₀The ratio of the number of background pixels to the whole image is omega₁Average gray of μ₁The total average gray scale of the b-channel image is recorded as mu, a threshold value T is set, and the number of pixels with the gray scale value smaller than the threshold value T in the image is recorded as N₀The number of pixels having a pixel gray level greater than the threshold T is denoted by N₁Then, there are:

ω₀＝N₀/M×N (1)

ω₁＝N₁/M×N (2)

N₀+N₁＝M×N (3)

ω₀+ω₁＝1 (4)

μ＝ω₀*μ₀+ω₁*μ₁(5)

g＝ω₀(μ₀-μ)²+ω₁(μ₁-μ)²(6)

substituting formula (5) for formula (6) yields the equivalent formula:

g＝ω₀ω₁(μ₀-μ₁)²(7)

and obtaining a threshold value T which enables g to be maximum by adopting a traversal method, wherein the threshold value T is an optimal binarization threshold value, and carrying out binarization processing on the blackboard frame according to the threshold value T to obtain a frame binary image.

The blackboard can be a single-connection blackboard or a sub-blackboard with the same multiple-connection specification. In one embodiment, the blackboard is a multiple sub-blackboard with the same specification, in the embodiment, the recognition module is further configured to select a complete sub-blackboard in the shooting picture of the camera in advance, calculate an image pixel value of the complete sub-blackboard, obtain a blackboard frame shape chart of the updated blackboard writing image frame according to the frame two-value chart after obtaining the frame two-value chart according to the above alternatives, supplement a certain area in the blackboard frame shape chart into a closed area if the lower edge of the area is a non-closed curve, calculate an image pixel value of each closed area in the blackboard frame shape chart, identify the closed area if the absolute value of the difference between the image pixel value of the certain closed area and the image pixel value of the complete sub-blackboard is less than a sixth preset threshold, and calculate a color average value of all pixels in each identified closed area and a color average value of all pixels in the blackboard area, and if the absolute value of the difference value between the color average value of all the pixels in a certain identified closed area and the color average value of all the pixels in the blackboard area is less than a seventh preset threshold value, determining the closed area as the effective area. The sixth preset threshold and the seventh preset threshold may be set as needed.

Optionally, the recognition module is further configured to split a text of the effective area of the updated blackboard writing image frame corresponding to each section of video according to words or phrases, match the split words or phrases with keywords of the knowledge points in the knowledge point domain, and determine all the keywords and occurrence frequencies thereof in the text; and then, determining the keywords with the occurrence frequency larger than a second preset threshold as the knowledge points of the text.

It should be noted that, the present embodiment may, but is not limited to, convert the blackboard writing in the effective area into text by using an OCR algorithm.

Optionally, the identification module is further configured to monitor a volume value of the audio in any unit time length for the audio corresponding to each piece of video, and if the volume value of the audio in a certain unit time length is smaller than a fifth preset threshold, the time point corresponding to the unit time length is a stopping point for the teacher to explain, where the fifth preset threshold is determined according to an average value of the volume of the audio corresponding to each piece of video, for example, the unit time length is 1 second, the identification module calculates an average value of the volume of the audio corresponding to a certain section of example video to be N, and monitors the volume value of the audio in each second for the audio corresponding to the section of example video, and if the volume of the audio in the nth second is smaller than η N, the time point corresponding to the nth second is a stopping point for the teacher to explain, where η and N are configurable parameters.

The calculation module is used for calculating a set of knowledge points contained in the classroom video according to the first set, the second set and the third set.

Optionally, the computing module is further configured to intersect the third set corresponding to each audio segment with the first set or the second set in the corresponding time segment, so as to obtain a set of knowledge points included in the classroom video.

The first embodiment is a method embodiment corresponding to the present embodiment, and the technical details in the first embodiment may be applied to the present embodiment, and the technical details in the present embodiment may also be applied to the first embodiment.

A third embodiment of the present application relates to an apparatus having a structure as shown in fig. 4, and the apparatus includes a camera, a teaching computer, a video composition module, and an online classroom video knowledge point identification module.

The camera is used to capture video including a blackboard area. Alternatively, the camera shooting range may be set as needed. For example, only a blackboard area may be included; for example, it may include a blackboard area and all or most of the area of the teacher's activity on the podium; but is not limited thereto.

The teaching computer is used for recording and playing videos of screens of teaching courseware. Optionally, the teaching computer is further configured to send corresponding signals to the video composition module and the online classroom knowledge point identification module when an action of opening teaching software for playing a teaching courseware, an action of opening a projector, or an action of operating a keyboard or a mouse is monitored, and send corresponding signals to the video composition module and the online classroom knowledge point identification module when an action of closing the teaching software for playing the teaching courseware, an action of closing the projector, or identification of occlusion in the blackboard area is monitored.

The video synthesis module is used for switching and acquiring a camera shooting video and a teaching computer screen video according to a preset rule, and synthesizing the camera shooting video corresponding to the blackboard writing part and the teaching computer screen video corresponding to the teaching courseware part into a classroom video.

And the online classroom video knowledge point identification module is used for acquiring the classroom video from the video synthesis module and identifying knowledge points in the classroom video according to the online classroom video knowledge point identification method of the first embodiment of the application.

It should be noted that, as will be understood by those skilled in the art, the implementation functions of the modules shown in the above embodiment of the online classroom video knowledge point identification system can be understood by referring to the relevant description of the online classroom video knowledge point identification method. The functions of the modules shown in the above embodiment of the online classroom video knowledge point identification system can be realized by a program (executable instructions) running on a processor, and can also be realized by a specific logic circuit. In the embodiment of the present application, the online classroom video knowledge point identification system can be stored in a computer readable storage medium if it is implemented in the form of a software function module and sold or used as an independent product. Based on such understanding, the technical solutions of the embodiments of the present application may be essentially implemented or portions thereof contributing to the prior art may be embodied in the form of a software product stored in a storage medium, and including several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read Only Memory (ROM), a magnetic disk, or an optical disk. Thus, embodiments of the present application are not limited to any specific combination of hardware and software.

Accordingly, the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, the computer-executable instructions implement the method embodiments of the present application. Computer-readable storage media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable storage medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.

In addition, the embodiment of the application also provides an online classroom video knowledge point identification system, which comprises a memory for storing computer executable instructions and a processor; the processor is configured to implement the steps of the method embodiments described above when executing the computer-executable instructions in the memory. The Processor may be a Central Processing Unit (CPU), other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), or the like. The aforementioned memory may be a read-only memory (ROM), a Random Access Memory (RAM), a Flash memory (Flash), a hard disk, or a solid state disk. The steps of the method disclosed in the embodiments of the present invention may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

It is noted that, in the present patent application, relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, the use of the verb "comprise a" to define an element does not exclude the presence of another, same element in a process, method, article, or apparatus that comprises the element. In the present patent application, if it is mentioned that a certain action is executed according to a certain element, it means that the action is executed according to at least the element, and two cases are included: performing the action based only on the element, and performing the action based on the element and other elements. The expression of a plurality of, a plurality of and the like includes 2, 2 and more than 2, more than 2 and more than 2.

All documents mentioned in this application are to be considered as being incorporated in their entirety into the disclosure of this application so as to be subject to modification as necessary. It should be understood that the above description is only a preferred embodiment of the present disclosure, and is not intended to limit the scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present disclosure should be included in the scope of protection of one or more embodiments of the present disclosure.

Claims

1. A method for recognizing online classroom video knowledge points is characterized by comprising the following steps:

determining a knowledge point domain of a classroom video to be identified;

2. The method for identifying knowledge points of online classroom video as claimed in claim 1, wherein before determining the knowledge point domain of the classroom video to be identified, the method further comprises:

3. The method for identifying points of knowledge in online classroom video as in claim 1, wherein said obtaining of said classroom video, said classroom video including a teaching courseware section and a blackboard-writing section, further comprises:

4. The method for identifying knowledge points of online classroom video as claimed in claim 3, wherein said switching and identifying knowledge points in a teaching class part and a blackboard writing part in said classroom video according to said preset switching condition further comprises:

receiving a signal to turn off the projector on the teaching computer,

and recognizing that the black board area has occlusion.

5. The method for identifying points of knowledge in online classroom video as recited in claim 1, wherein said teaching courseware segment is divided into a plurality of sections of video according to courseware page switching time points, further comprising:

6. The method of claim 5, wherein the identifying the knowledge points in the text according to a predetermined knowledge point identification algorithm based on the knowledge point domain to obtain the first set further comprises:

7. The method for identifying points of knowledge in online classroom video as recited in claim 1, wherein before the blackboard-writing portion is divided into a plurality of sections of video according to the blackboard-writing update time point, the method further comprises:

8. The method of claim 7, wherein the second set is obtained by recognizing the knowledge points in the text according to a predetermined knowledge point recognition algorithm based on the knowledge point domain, further comprising:

9. The method for identifying knowledge points of online classroom video as recited in claim 1, wherein before the audio corresponding to each video section is divided into a plurality of audio segments according to audio pause points, the method further comprises:

10. The method for identifying knowledge points of online classroom video according to any one of claims 1-9, wherein said computing the set of knowledge points contained in said classroom video based on said first set, said second set, and said third set further comprises:

11. An apparatus, comprising:

an online classroom video knowledge point identification module, configured to identify a knowledge point in a classroom video according to the method claimed in any one of claims 1-10 after the classroom video is obtained from the video composition module.