KR20210144082A - Method of detecting valuable sections of video lectures, computer program and computer-readable recording medium - Google Patents

Abstract

The present invention relates to a method for detecting an important section of a video lecture comprising: a step of extracting at least one frontal gaze section wherein an instructor continuously stares straight ahead for a predetermined time during a video lecture; a step of calculating a voice analysis result for each of the extracted at least one frontal gaze section; a step of deriving an important section detection model using the voice analysis result and the reference data; and a step of detecting the important section using the derived important section detection model. Therefore, the present invention enables to provide a method for detecting the important section of the video lecture using a model for detecting the important section derived using the lecture characteristics of the instructor.

Description

Method of detecting valuable sections of video lectures, computer program and computer-readable recording medium

The present invention relates to a method for detecting an important section of a video lecture, and more particularly, to a method for detecting an important section of a video lecture based on the lecture characteristics of an instructor.

Video lectures are not limited by time and space, but also substantially subsidize or replace the existing face-to-face lectures due to economic efficiency.

Learners who study using video lectures are not only passively watching a given lecture, but are more actively using video lectures by using various playback control functions.

In particular, learners want to quickly find important places related to the lecture topic, difficult places to review, interesting places, etc., and move to or save the found places. At this time, the methods that learners can use to move to the desired place within the video include moving back and forth for a set amount of time, moving the indexed table of contents chapter by verse, and using the mouse to move to the desired position. Most of these are insufficient to satisfy the needs of learners.

Recently, in the field of artificial intelligence, research on how to search for important information in video lectures using machine learning is being conducted. There is a disadvantage in that information such as a vocabulary dictionary, lecture subtitles, and learner feedback required for extraction is additionally required in advance.

On the other hand, in the learning satisfaction of adult learners with respect to video lectures, the presence or absence of the instructor does not significantly affect the learning satisfaction. However, in video lectures for elementary, middle and high school students, it is known that the lecturer's appearance itself gives students a sense of teaching reality and improves learning satisfaction or learning effect. For this reason, most of the video lectures for elementary, middle and high school students are produced with lecturers appearing.

In the present inventor, the instructor artificially adjusts the atmosphere or flow of the lecture to more effectively convey the lecture topic. In particular, when explaining the content to be emphasized, it is clearly different from when explaining the part that is not in order to emphasize it. It was found that the lecture characteristics of the instructor were distinguished.

Therefore, in the case of detecting an important section in a video lecture in which an instructor appears, it was expected that it would be effective to be based on the lecture characteristics of the instructor instead of based on keyword-oriented data.

KR 1205388 B1 KR 0593837 B1

Accordingly, an object of the present invention is to provide a method for detecting important sections of a video lecture based on the lecture characteristics of the instructor, instead of based on keyword-oriented data.

Another object of the present invention is to provide a method for detecting important sections of a video lecture using a critical section detection model derived using lecture characteristics of an instructor.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

The object is, according to a first aspect of the present invention,

extracting at least one frontal gaze section in which the instructor continuously gazed at the front for a predetermined time during the video lecture;

calculating a voice analysis result for each of the extracted at least one frontal gaze section;

deriving an important section detection model using the voice analysis result and reference data; and

Including the step of detecting a critical section using the derived critical section detection model,

It is achieved by the method of detecting critical sections of video lectures.

In this case, the voice analysis result preferably includes the pitch of the instructor's voice and the intensity of the instructor's voice.

In addition, the reference data is an identification value by an expert,

The step of deriving the critical section detection model is,

It may be by machine learning using the voice analysis result and the reference data as training data, and the machine learning is the critical section by performing logistic regression analysis using the voice analysis result and the reference data as training data. Preferably by determining the optimal parameter values of the detection model.

Furthermore, the step of extracting the at least one frontal gaze section,

It may be to extract at least one frontal gaze section in which the instructor continuously gazes straight ahead for 5 seconds or more during a video lecture using a predetermined face recognition program.

In addition, the calculating of the result of the voice analysis includes calculating a representative value for each section of the pitch of the instructor's voice and the intensity of the instructor's voice using a predetermined voice analysis program for each of the extracted at least one frontal gaze section. and can be standardized.

In this case, the representative value for each section is preferably one of an average value, a maximum value, a minimum value, and a median value.

Said object is also according to a second aspect of the invention,

It is achieved by a computer-readable recording medium recording a program for performing the above method.

Furthermore, the object is also according to a third aspect of the invention,

It is achieved by a computer program stored in a medium for executing the above method through combination with hardware.

According to the method for detecting important sections of a video lecture as described above, there is an advantage of providing a method of detecting important sections of a video lecture based on the lecture characteristics of the instructor instead of based on keyword-oriented data.

In addition, according to the method for detecting important sections of a video lecture of the present invention, there is an advantage of providing a method for detecting important sections of a video lecture using a model for detecting important sections derived using the lecture characteristics of the instructor.

1 is a flowchart of a method for detecting an important section of a video lecture according to the present invention.
2 is a view showing an example of extraction of the frontal gaze section according to an embodiment of the present invention.
3 is a diagram illustrating an example of calculating the instructor's voice pitch information according to an embodiment of the present invention.
4 is a diagram illustrating an example of calculating intensity information of an instructor's voice according to an embodiment of the present invention.

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the contents described in the accompanying drawings. However, the present invention is not limited or limited by the exemplary embodiments. The same reference numerals provided in the respective drawings indicate members that perform substantially the same functions.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

When a part "includes" a certain element throughout the specification, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

As mentioned in the description of the background of the invention, the instructor artificially adjusts the atmosphere or flow of the lecture in order to more effectively convey the subject of the lecture. It shows the characteristics of the lecturer's lecture that is clearly different from the case where he explains the parts that are not.

At this time, the lecture characteristics of the instructor when explaining the content to be emphasized are: first, that the instructor's face looks straight ahead in order to focus the learner (frontal gaze characteristic), and secondly, the voice The point is that it controls pitch and intensity (a characteristic of using voice).

In video lectures, direct eye contact is impossible because the learning takes place with the instructor and the learner separated. However, the instructor can use a shooting technique called frontal gaze to achieve the effect of eye contact, and the learner who sees this can feel as if the instructor is making eye contact with him and giving a direct explanation, so the learner's concentration can be improved. .

In addition to video lectures, instructors tend to speak louder by raising the tone in the areas emphasized to induce the learner's concentration or to improve the delivery of important content. It means that the displayed intensity increases. This characteristic of the instructor's voice used in the important section is significantly different from when explaining the section other than the important section, and can be an important criterion for judging whether it is an important section or not.

The method for detecting important sections of a video lecture according to the present invention was developed to identify the lecture characteristics of the instructor in consideration of this point, and to derive a model capable of detecting the important sections of the video lecture using the characteristics.

1 is a flowchart of a method for detecting an important section of a video lecture according to the present invention.

1 , the method for detecting important sections of a video lecture according to the present invention essentially includes a step of extracting a frontal gaze section (S100), a step of calculating a voice analysis result (S200), and a step of deriving an important section detection model (S300) do. Of course, a critical section detection step of detecting a critical section using the derived critical section detection model may also be included.

First, in the step of extracting the frontal gaze section ( S100 ), at least one frontal gaze section in which the instructor continuously gazes at the front for a predetermined time during the video lecture is extracted.

More preferably, using a predetermined face recognition program, it is possible to select and extract all of the at least one frontal gaze section in which the instructor continuously stares at the front for 5 seconds or more during the video lecture.

In this case, the predetermined face recognition program may be implemented by borrowing an existing face recognition program as it is or through programming using a predetermined program, for example, using python (v3.7.1) and openCV (v4.1.1). can be implemented.

The extraction of the frontal gaze section uses the face recognition program implemented in this way to recognize the instructor’s face when the video is played, and the frontal recognition information consisting of three data corresponding to the recognition start time, recognition end time, and frontal gaze time. can be realized by generating and outputting it in the form of a text file, as shown in FIG. 2 .

Here, the duration of the frontal gaze, which is a criterion that can be regarded as an important section, can be determined empirically or experimentally.

For example, in order to determine the duration of the frontal gaze of the important section, the present inventor first extracted 552 frontal exam sections from 8 arbitrarily selected SAT math video lectures, and then passed the verification of 3 experts with more than 10 years of experience as a math instructor. 37 important sections were selected and the duration of the frontal gaze was investigated. As a result of the investigation, as shown in Table 1 below, the longest duration was 59.56 seconds, and the shortest duration was 5.07 seconds, and there was no important section having a duration of less than 5 seconds.

Section number Duration (seconds) Section number Duration (seconds) Section number Duration (seconds) 1-D 27.9 6-C 59.56 8-16 5.81 1-G 15.97 6-E 5.94 8-20 6.81 2-B 12.08 6-H 13.15 8-28 17.88 2-C 6.87 6-I 5.84 8-30 13.15 2-D 10.08 7-A 6.37 8-31 5.87 2-E 5.61 7-B 6.44 8-32 5.07 2-F 17.82 7-G 25.56 8-33 10.01 2-G 25.56 7-H 17.88 8-34 5.87 3-C 14.61 7-O 5.17 8-35 11.85 4-D 5.17 8-5 8.21 8-36 5.34 5-C 5.87 8-8 9.14 8-37 7.31 5-D 11.64 8-9 9.68 8-39 14.68 5-F 6.14 8-10 5.24

Next, the reason for the frontal gaze was investigated by classifying 552 frontal gaze sections extracted from the above randomly selected 8 SAT math video lectures based on the duration of 5 seconds. As a result of the investigation, as shown in Table 2 below, the section of the frontal gaze with a duration of less than 5 seconds did not contain content that could be judged to be an important section, as shown in Table 2 below.

Frontal gaze section with duration less than 5 seconds Frontal gaze section with a duration of 5 seconds or longer Reason for face-to-face period writing poem Conceptual explanation when writing a problem Explanation of problem analysis and interpretation method when graphing Explain the key clues to the problem formula writing poetry Explanation of how to catch a variable at the time of calculation Suggest a key strategy for problem solving When the problem condition is indicated Key Terms Explained word meaning Solving process summary Explanation of intent

As such, according to the results of the two surveys, in the 8 arbitrarily selected SAT math video lectures, the duration of the frontal gaze in the important section was determined to be 5 seconds or more. Of course, depending on which field of video lecture is selected, the threshold value of the duration of the face-to-face gaze in the important section may be less than or greater than 5 seconds.

Accordingly, a section having a frontal gaze duration greater than or equal to a predetermined threshold value is extracted as a frontal gaze section.

Next, in the voice analysis result calculation step (S200), a voice analysis result is calculated for each of the at least one frontal gaze section extracted above. In this case, the voice analysis result may include the pitch of the instructor's voice and the intensity of the instructor's voice.

More preferably, for each of the at least one frontal gaze section extracted above, a representative value for each section of the pitch of the instructor's voice and the intensity of the instructor's voice can be calculated and standardized using a predetermined voice analysis program. .

As mentioned above, the instructor tends to speak loudly by raising the tone in the part emphasized to induce the learner's concentration or to improve the delivery of important content. This means that the intensity representing Therefore, the part with a high pitch or high intensity of the instructor's voice has a very high probability of being an important section, so it may be desirable to actively utilize it in deriving an important section detection model.

In this case, the predetermined voice analysis program may be implemented by borrowing an existing voice analysis program as it is or through programming using a predetermined program, for example, it may be implemented using praat(v6.1).

The pitch of the voice among the voice analysis results is, using the voice analysis program implemented in this way, the voice pitch information of the corresponding section, such as in FIG. For example, it may be provided as an average value, a maximum value, a minimum value, and a median value of the pitch. At this time, as shown in (a) of FIG. 3 , a window showing the voice waveform and the pitch graph of the corresponding section may be provided together.

In this case, in the case of performing voice analysis using an audio file of a video lecture rather than a video lecture, the step of first extracting the audio file from the video lecture may be further included.

Of course, this voice pitch information can be provided for all sections of the video lecture separated by a predetermined interval, but advantageously in terms of time and cost, it can be provided only for each of the at least one frontal gaze section extracted above .

Here, at least one of the average value, the maximum value, the minimum value, and the median value of the pitch provided for the corresponding section is defined as a representative value for each section, and this can be treated as a part of the voice analysis result. The average value that best represents it may be desirable.

Similarly, the intensity of the voice among the voice analysis results is the intensity of the corresponding section, as shown in FIG. 4(b), for a predetermined section of the video lecture and/or the audio file of the video lecture using the voice analysis program implemented in this way. Information, for example, may be provided as an average value, a maximum value, and a minimum value of the intensity. In this case, as shown in (a) of FIG. 4 , a window showing the voice waveform and the intensity graph of the corresponding section may be provided together.

Of course, this voice intensity information may be provided for all sections of the video lecture separated by a predetermined interval, but advantageously in terms of time and cost, it may be provided only for each of the at least one frontal gaze section extracted above .

Here, at least one of the average value, the maximum value, and the minimum value of the intensity provided for the corresponding section can be defined as a representative value for each section and treated as a part of the voice analysis result. An average value expressed may be preferable.

In order to determine whether a corresponding section is an important section from the representative value for each section, it is necessary to convert it into a relative value for comparison. This is referred to as standardization, and Equation (1) can be used, for example.

(1)

(One)

Here, X is the representative value of the corresponding section, m is the average value of the representative values for each section of the entire video lecture, and σ is the standard deviation.

This is a value indicating how far the representative value of the corresponding section is from the average. For example, if the standardized value of the representative value of the pitch in a section is positive, the instructor’s voice tone in that section was higher than the average voice tone in the video. can be judged Similarly, if the standardized value of the representative value of the intensity of a certain section is positive, it can be determined that the instructor's voice in that section was larger than the average voice in the video.

Next, in the step of deriving a critical section detection model ( S300 ), a critical section detection model is derived using the voice analysis result and reference data. In this case, the reference data may be an identification value by an expert. The identification value may be 0 or 1, for example, 1 if it is an important interval, and 0 if it is not an important interval, which may be assigned by an expert. In this case, the expert may be a professional with personality and/or may be a result of an impersonal computer program corresponding to the identification result of the professional with personality.

More preferably, the derivation of the critical section detection model may be performed by machine learning using the speech analysis result and the reference data as training data, wherein the machine learning comprises the speech analysis result and the It is preferable to perform logistic regression analysis using the reference data as training data to determine the optimal parameter value of the critical interval detection model.

Logistic regression analysis is the most commonly used analysis method to predict the probability that a result will fall into 0 or 1. The critical section detection model according to the present invention also basically uses a logistic function as in Equation (2) below to determine whether it is a critical section (1) or not (0).

(2)

(2)

As a variable for determining whether or not the corresponding section is an important section, in the present invention, as mentioned above, the calculated voice analysis result is used for each of the extracted at least one frontal gaze section. In other words, the (normalized) representative value of the voice pitch and the (normalized) representative value of the voice intensity are defined as independent variables of the linear regression equation as in Equation (3) below.

(3)

(3)

where the variable p is a representative value of the (normalized) voice pitch, i is the representative value of the (normalized) voice intensity, ω _p and ω _i are the weights for the variables p and i, respectively, and b is the bias value of the linear regression equation. means (bias).

Therefore, using Equations (2) and (3), a regression function representing the probability that the result value is determined to be 1 (important section) can be obtained as in Equation (4) below.

(4)

(4)

The result value of the regression function S(f(p,i)) is a probability value between 0 and 1, and means the probability of determining the corresponding section as 1, which is an important section. Therefore, if the final result value is not 0, rounding, rounding, and rounding may be performed to have one of 1. For example, in the present invention, rounding is performed using flooring.

(5)

(5)

In addition, a cost function equation such as Equation (6) using the maximum likelihood estimation method is used to obtain _{the optimal parameter values (ω p} , ω _i and b) that minimize the error in determining the critical section.

(6)

(6)

Here, m is the number of sample intervals required for model generation, k is the standard interval number, and y is the expert identification value assigned to the corresponding interval.

Calculations of Equations (4) to (6) above may be implemented by borrowing an existing program as it is or by programming using a predetermined program, for example, using weka (v3.8), an open source software. It can be processed to obtain the optimal values of the parameters (ω _p , ω _i and b) of the linear regression equation.

For example, in the present invention, _{in order to obtain the optimal parameter (ω p} , ω _i and b) values of the linear regression equation, 507 frontal images are selected from among the above 8 arbitrarily selected 8 SAT math video lectures. After extracting the gaze section, 80 sections with a duration of 5 seconds or longer were again extracted as sample sections.

Next, a voice file is extracted from each video lecture, and an expert identification value is assigned to each of the 80 extracted corresponding sample sections as 1 if it is an important section and 0 if it is not an important section, and logistic using the open source software weka. A regression analysis was performed. As such, the optimal parameter information derived as a result of logistic regression analysis _{is 1.6288 for ω p} , 1.5533 for ω _i , and 1.2254 for b.

Accordingly, the critical section detection model to which the parameter information derived through logistic regression analysis is substituted is as shown in Equation (7) below.

(7)

(7)

Of course, as mentioned above, the optimal parameter information derived may vary depending on which field of video lecture is selected.

Finally, a critical section can be detected using the critical section detection model derived according to the present invention. Hereinafter, the performance evaluation result of the critical section detection model derived as described above will be described.

The performance evaluation uses the cross-validation method, which is a method of randomly extracting and verifying the entire data for training and testing. Here, accuracy, precision, recall, and F-measure were obtained through 10-fold cross-validation for all 80 data sets.

First, looking at the accuracy of the critical section detection model, an error occurred in the detection of 6 sections out of 80 sections. .

In addition, with reference to Table 3 below, which summarizes the results of comparing the detection results of the critical section detection model with the expert identification values, precision, recall, and F-measure are obtained.

Critical section detection model result value total One 0 expert identification One 23 5 28 0 One 51 52 total 24 56 80

Precision refers to the ratio of actual critical intervals among the intervals detected by the detection model as critical intervals. So the precision here is 95.83% (=23/24*100). Recall is the ratio of what the critical section detection model correctly detects as a critical section among the actual critical sections. Therefore, the recall here is 82.14% (=23/28*100). The F-measure is the harmonic mean of precision and recall, here 88.5% (=2*95.83*82.14/(95.83+82.14)).

If training data are obtained from more video lectures to find the optimal parameters of the critical section detection model according to the present invention, it is expected that performance such as detection accuracy and precision of critical sections can be improved.

Meanwhile, the methods according to an embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computing devices and recorded in a computer-readable recording medium.

The program instruction form may be collectively referred to as software, which may include a computer program, code, instructions, or a combination of one or more of these, and configures the processing device to operate as desired. may be configured or may independently or collectively instruct the processing device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. may be permanently or temporarily embody in The software may be distributed over networked computing devices, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and used by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

In summary, according to the method of detecting important sections of a video lecture of the present invention, instead of based on keyword-oriented data, a method of detecting important sections of a video lecture based on the lecture characteristics of the instructor is provided.

More specifically, according to the method for detecting important sections in a video lecture of the present invention, a method for detecting important sections in a video lecture using a critical section detection model derived using lecture characteristics of an instructor is provided.

Accordingly, the detected important section can be used to automatically insert a bookmark therein so that the learner can use it, or generate an event in the corresponding section to increase the learner's concentration. In addition, by extracting the detected important sections, it makes it possible to use the video lectures more effectively, such as producing a highlight video or a promotional video.

Terms used in this specification are generally intended to be "open-ended" terms, particularly in claims (eg, the body of claims) (eg, "comprising" means "including but not limited to"). , "have" should be construed as "have at least more" and "comprise" be interpreted as "including but not limited to") It is expressly recited in the claims, and in the absence of such recitation, no such intent is understood.

Only specific features of the invention have been shown and described herein, and various modifications and variations will occur to those skilled in the art. It is therefore to be understood that the claims are intended to cover changes and modifications that fall within the spirit of the present invention.

Claims (9)

extracting at least one frontal gaze section in which the instructor continuously gazed at the front for a predetermined time during the video lecture;
calculating a voice analysis result for each of the extracted at least one frontal gaze section;
deriving an important section detection model using the voice analysis result and reference data; and
Including the step of detecting a critical section using the derived critical section detection model,
A method of detecting important sections of video lectures. The method of claim 1,
The voice analysis result includes the pitch of the instructor's voice and the intensity of the instructor's voice,
A method of detecting important sections of video lectures. 3. The method of claim 2,
The reference data is an identification value by an expert,
The step of deriving the critical section detection model is,
By machine learning using the voice analysis result and the reference data as training data,
A method of detecting important sections of video lectures. 4. The method of claim 3,
The machine learning is by performing logistic regression analysis on the voice analysis result and the reference data as training data to determine the optimal parameter value of the critical section detection model,
A method of detecting important sections of video lectures. 5. The method of claim 4,
The step of extracting the at least one frontal gaze section,
Characterized in extracting at least one frontal gaze section, in which the instructor continuously gazed at the front for more than 5 seconds during the video lecture using a predetermined face recognition program,
A method of detecting important sections of video lectures. 6. The method of claim 5,
The step of calculating the voice analysis result includes calculating, for each of the extracted at least one frontal gaze section, a representative value for each section of the pitch of the instructor's voice and the intensity of the instructor's voice using a predetermined voice analysis program, characterized by standardizing,
A method of detecting important sections of video lectures. 7. The method of claim 6,
The representative value for each section is characterized in that one of an average value, a maximum value, a minimum value, and a median value,
A method of detecting important sections of video lectures. A computer-readable recording medium recording a program for performing the method of any one of claims 1 to 7. A computer program stored in a medium for executing the method of any one of claims 1 to 7 through combination with hardware.

Images