KR20220053441A - Method, apparatus and computer program for evaluating lecture video using neural network - Google Patents

Abstract

Disclosed is a neural network learning method for evaluating a lecture video. The method comprises: a step of obtaining a plurality of lecture videos and lecture evaluation information for each of the plurality of lecture videos; a step of performing pre-processing of the acquired lecture video; a step of obtaining lecture information on the lecture video; a step of extracting a feature vector by analyzing the pre-processed lecture video; a step of generating learning data by labeling lecture information, a feature vector, and lecture evaluation information on the lecture video; and a step of training a neural network for evaluating the lecture video based on the learning data.

Description

Method, apparatus and computer program for evaluating lecture video using neural network}

The present invention relates to a method, an apparatus, and a computer program for evaluating a learner's expected satisfaction with a lecture video of a lecturer, lecture appropriateness, etc. using a neural network.

Further, the present invention relates to a method, an apparatus and a computer program for training the above-described neural network.

Due to the impact of COVID-19, which started in China and spread around the world at the end of 2019, there was a big change in human lifestyle, etc. It has rapidly shifted to a consumption culture, that is, a non-face-to-face consumption culture.

In particular, as educational institutions are also implementing non-face-to-face lectures, the method in which lecturers upload recorded lecture videos using real-time online classes using interactive video communication platforms or video streaming services has become the majority of teaching methods.

However, existing educators who are not familiar with non-face-to-face classes have difficulties in finding effective teaching methods for online classes, which may lead to a decrease in learner interest and lower academic achievement.

The above-described conventional problem can be solved if the learner's satisfaction, interest, academic achievement, etc. can be predicted and evaluated in advance with respect to the lecture video of the lecturer.

Accordingly, an object of the present invention is to provide a method, apparatus, and computer program for evaluating the expected satisfaction of the learner with respect to the lecture video of the lecturer, the appropriateness of the lecture, etc. using a neural network.

Another object of the present invention is to provide a method, an apparatus, and a computer program for learning the above-described neural network.

A method for learning a neural network for evaluating a lecture image according to an embodiment of the present invention for achieving the above object includes obtaining a plurality of lecture images and lecture evaluation information for each of the plurality of lecture images, the obtained performing pre-processing of the lecture image, obtaining lecture information for the lecture image, analyzing the pre-processed lecture image to extract a feature vector, the lecture information, the feature vector, and the lecture evaluation information It includes the steps of generating learning data by labeling the lecture image, and training a neural network for evaluating the lecture image based on the learning data.

The extracting of the feature vector may include extracting a first feature vector by analyzing visual data related to the appearance of the lecturer from the lecture image, analyzing auditory data related to the lecturer's voice from the lecture image. The method further includes extracting a second feature vector and extracting a third feature vector by analyzing lecture data related to lecture content from the lecture image, wherein the generating of the learning data includes the extracted first feature vector; The second feature vector and the third feature vector may be labeled on the lecture image.

In addition, the extracting of the first feature vector may include extracting a first feature vector including a feature value of at least one of a movement of a lecturer, an expression of the lecturer, and an emotion of the lecturer.

In addition, the extracting of the second feature vector may include extracting a second feature vector including a feature value of at least one of a tone, a size, a rhythm, a tone, and a speed of the lecturer's voice.

Also, it is possible to extract a third feature vector including a feature value of at least one of the difficulty of the vocabulary used in the lecture content of the lecturer, the richness of the vocabulary, the number of repetitions of the lecture keyword, the length of the sentence, and the duration of the lesson.

In addition, the lecture information may include at least one of a lecture purpose of the lecture video, a lecture subject, a lecture atmosphere, a lecture subject, and a lecture method.

On the other hand, the method for evaluating a lecture image based on a neural network according to an embodiment of the present invention for achieving the above object includes learning a neural network through a neural network learning method to evaluate the lecture image, and a lecture image to be evaluated and receiving lecture information on the lecture image, and calculating expected lecture evaluation information by inputting the lecture information and lecture image into the learned neural network.

The calculating of the expected lecture evaluation information may include extracting a first feature vector by analyzing visual data related to the appearance of the lecturer from the lecture image to be evaluated, and from the lecture image to be evaluated. extracting a second feature vector by analyzing auditory data related to the lecturer's voice; extracting a third feature vector by analyzing the lecture content of the lecturer from the lecture image to be evaluated; and the lecture information and the first and calculating predicted lecture evaluation information by inputting the first to third feature vectors into the neural network.

On the other hand, the neural network-based lecture image evaluation apparatus according to an embodiment of the present invention for achieving the above object is a neural network model storage unit for storing a neural network learned through a neural network learning method in order to evaluate a lecture image, an evaluation target is and an input unit for receiving a lecture image to be received, receiving lecture information on the lecture image, and an AI processor for calculating expected lecture evaluation information by inputting the lecture information and the lecture image to the learned neural network.

In addition, in the program stored in the computer-readable recording medium according to an embodiment of the present invention for achieving the above object, a program code for executing the above-described neural network learning method may be recorded.

In addition, in the program stored in the computer-readable recording medium according to an embodiment of the present invention for achieving the above object, a program code for executing the method for evaluating the above-described neural network-based lecture image may be recorded. .

According to the present invention, the lecturer can evaluate his/her lecture video in advance and use it as feedback data for learning a more effective lecture delivery method in an online class.

In addition, by replacing the subject who evaluates the lecture video of the lecturer with an artificial intelligence rather than a human, it is possible to ensure that the lecture evaluation of the lecturer is conducted in a fair and objective manner.

In addition, a lecturer preparing for a newly opened lecture can record the lecture video and input it into the neural network, thereby confirming the learners' expected lecture evaluation in advance.

1 is an exemplary diagram illustrating a neural network-based lecture image evaluation system according to an embodiment of the present invention.
2 and 3 are flowcharts illustrating a neural network learning method for evaluating a lecture image according to an embodiment of the present invention.
4 is a flowchart illustrating a method of evaluating a lecture image based on a neural network according to an embodiment of the present invention.
5 is an exemplary diagram illustrating a user interface according to an embodiment of the present invention.
6 is an exemplary diagram illustrating a neural network for evaluating a lecture image according to an embodiment of the present invention.
7 is a block diagram showing the configuration of a lecture image evaluation apparatus according to an embodiment of the present invention.

The following is merely illustrative of the principles of the invention. Therefore, those skilled in the art will be able to devise various devices that, although not explicitly described or shown herein, embody the principles of the present invention and are included within the spirit and scope of the present invention. In addition, all conditional terms and examples listed herein are, in principle, expressly intended only for the purpose of understanding the concept of the present invention, and it should be understood that they are not limited to the specifically enumerated embodiments and states as such. do.

Moreover, it is to be understood that all detailed description reciting the principles, aspects, and embodiments of the invention, as well as specific embodiments, are intended to cover structural and functional equivalents of such matters. It should also be understood that such equivalents include not only currently known equivalents, but also equivalents developed in the future, i.e., all devices invented to perform the same function, regardless of structure.

Thus, for example, the block diagrams herein are to be understood as representing conceptual views of illustrative circuitry embodying the principles of the present invention. Similarly, all flowcharts, state transition diagrams, pseudo code, etc. may be tangibly embodied on computer-readable media and be understood to represent various processes performed by a computer or processor, whether or not a computer or processor is explicitly shown. should be

The functions of the various elements shown in the figures including a processor or functional blocks represented by similar concepts may be provided by the use of dedicated hardware as well as hardware having the ability to execute software in association with appropriate software. When provided by a processor, the functionality may be provided by a single dedicated processor, a single shared processor, or a plurality of separate processors, some of which may be shared.

In addition, the clear use of terms presented as processor, control, or similar concepts should not be construed as exclusively referring to hardware having the ability to execute software, and without limitation, digital signal processor (DSP) hardware, ROM for storing software. It should be understood to implicitly include (ROM), RAM (RAM) and non-volatile memory. Other common hardware may also be included.

In the claims of this specification, a component expressed as a means for performing the function described in the detailed description includes, for example, a combination of circuit elements that perform the function or software in any form including firmware/microcode, etc. It is intended to include all methods of performing the functions of the device, coupled with suitable circuitry for executing the software to perform the functions. Since the present invention defined by these claims is combined with the functions provided by the various enumerated means and in a manner required by the claims, any means capable of providing the functions are equivalent to those contemplated from the present specification. should be understood as

The above objects, features and advantages will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily implement the technical idea of the present invention. There will be. In addition, in the description of the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

Hereinafter, various embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is an exemplary diagram illustrating a neural network-based lecture image evaluation system 1000 according to an embodiment of the present invention.

Referring to FIG. 1 , the neural network-based lecture image evaluation system 1000 includes a user 10 requesting evaluation of a lecture image and a lecture image evaluation apparatus 100 for evaluating a corresponding lecture image according to a lecture image evaluation request. can do.

The user 10 may request an evaluation of the lecture image from the lecture image evaluation apparatus 100 . Here, the user 10 is a subject who performs a lecture in the corresponding lecture image, and may be a lecturer, an instructor, a teacher, an educator, etc. (hereinafter, collectively referred to as a "lecturer"), or a supervisor who evaluates the lecture image of the lecturer. .

The lecture image evaluation apparatus 100 may evaluate the lecture image based on the neural network at the request of the lecturer 10 , and provide the lecture evaluation result to the lecturer 10 . Here, the neural network is learned to evaluate a lecture image, and is learned by the lecture image evaluation apparatus 100 or is learned from an external server or device other than the lecture image evaluation apparatus 100, and then the lecture image evaluation apparatus 100 can be provided on

Hereinafter, a neural network learning method for evaluating a lecture image will be described with reference to FIG. 2 .

2 is a flowchart illustrating a neural network learning method for evaluating a lecture image according to an embodiment of the present invention.

Referring to FIG. 2 , the neural network learning method for evaluating lecture images may acquire a plurality of lecture images and lecture evaluation information evaluated thereon ( S100 ). For example, in the acquiring step ( S100 ), a plurality of lecture images and lecture evaluation information for each of the plurality of lecture images may be obtained through an online lecture platform.

Here, the online lecture platform is a platform that provides lecture images to learners through online, and the online lecture platform may provide real-time lecture images or recorded lecture images to learners.

In addition, the lecture video may be a video for a predetermined time or longer.

In addition, the lecture evaluation information may refer to an evaluation of the lecture video left by the learner who watched the lecture video on the corresponding online lecture platform. For example, the lecture evaluation information may include lecture evaluation scores (star points, numbers, etc.) for a lecture video left by a learner who watched the lecture video, or lecture evaluation text.

Here, the lecture evaluation score may include a learner's satisfaction score, a lecture adequacy score indicating how appropriate the lecture is for the purpose of the lecture to the learner, and the like.

In addition, the lecture evaluation text is, for example, "the lecturer was unable to concentrate on the lecture because there was a lot of movement", "the lecturer's voice speed was slow and boring." " There may be few opinions of the learner about the lecture, etc.

Next, pre-processing of the acquired lecture image may be performed (S200). Specifically, in the pre-processing step S200, the eyes, nose, and mouth of the lecturer's face are pre-processed in the lecture image to be clear, and the lecturer's voice may be separated from the background sound to be pre-processed so that the lecturer's voice is emphasized. In addition, the pre-processing step S200 may pre-process so that text related to lecture content is emphasized in the lecture image. In addition, in the pre-processing step (S200), it is possible to remove parts irrelevant to the lecture ability of the lecturer, such as the break time irrelevant to the lecture, and the learner, not the instructor, makes a presentation and asks a question.

Then, lecture information on the lecture image may be obtained (S300). Here, the lecture information relates to various pieces of information about the corresponding lecture image, and may be information input by a worker who creates and manages learning data. However, according to another implementation example, lecture information may be automatically provided through learning of an AI processor, which will be described later.

Specifically, lecture information includes the lecture purpose of the lecture video (eg, university entrance exam lectures, lectures by qualified professionals, etc.), lecture subjects (eg elementary school students, middle school students, high school students, college students, adults, etc.), and lecture atmosphere (eg, For example, rigorous, serious, humorous, confident, positive, etc.), lecture subjects (eg, Korean, math, English, scientific research, social studies, second language, etc.) ) may be included.

Next, a feature vector may be extracted by analyzing the pre-processed lecture image (S400). Here, the feature vector may relate to the appearance of the lecturer, the lecturer's voice, and lecture contents.

In this regard, it will be further described with reference to FIG. 3 .

Referring to FIG. 3 , in the step of extracting the feature vector of the neural network learning method for evaluating the lecture image ( S400 ), the first feature vector may be extracted by analyzing visual data related to the appearance of the lecturer from the lecture image ( S410 ). ).

Here, the visual data is data that can be visually recognized in relation to the lecturer when the learner watches the lecture video, and the first feature vector is the lecturer's movement (eg, the frequency of the gesture, the size of the gesture, the direction, etc.) , corresponding to at least one of an expression (e.g., expressionless, smiling, crying, angry, surprised, etc.) and emotion (e.g., happiness, anger, contempt, disgust, fear, sadness, surprise, etc.) It may be a feature value.

That is, in the step of extracting the first feature vector ( S410 ), the first feature vector including at least one feature value among the movement of the lecturer, the expression of the lecturer, and the emotion of the lecturer may be extracted by analyzing the lecture image. Here, the extraction of the first feature vector may be performed using a neural network trained to extract a feature vector corresponding to at least one of the lecturer's movement, the lecturer's expression, and the lecturer's emotions from the lecture image.

In addition, a second feature vector may be extracted by analyzing auditory data related to the lecturer's voice from the lecture video (S420). Here, the auditory data is data that the learner can perceive as auditory when viewing the lecture video, and the second feature vector may be a feature value corresponding to at least one of tone, size, rhythm, tone, and speed of the lecturer's voice. .

That is, in the step of extracting the second feature vector ( S420 ), a second feature vector including at least one feature value among the tone, size, rhythm, tone, and speed of the lecturer's voice may be extracted by analyzing the lecture video. . Here, the extraction of the second feature vector may be performed using a neural network trained to extract a feature vector corresponding to at least one of tone, size, rhythm, tone, and speed of the lecturer's voice from the lecture image.

In addition, a third feature vector may be extracted by analyzing the lecture content of the lecturer from the lecture video (S430). Here, the lecture content relates to the content that the lecturer wants to deliver to the learner through the lecture, and the third feature vector is the difficulty of the vocabulary used in the lecture content, the richness of the vocabulary, the number of repetitions of the lecture keyword, the length of the sentence, and the class It may be a feature value corresponding to at least one of the durations.

That is, the step of extracting the third feature vector ( S430 ) is performed by analyzing at least one of the difficulty of the vocabulary used in the lecture content, the richness of the vocabulary, the number of repetitions of the lecture keyword, the length of the sentence, and the duration of the lesson from the lecture video. A third feature vector related to the lecture content of the lecturer may be extracted. Here, the extraction of the third feature vector is a neural network trained to extract a feature vector corresponding to at least one of the lecturer's vocabulary difficulty, vocabulary richness, number of repetitions of lecture keywords, sentence length, and class duration from the lecture image. can be performed using

Again, referring to FIG. 2 , learning data may be generated by labeling the input lecture information, the extracted feature vector, and the lecture evaluation information on the corresponding lecture image ( S500 ). Here, the extracted feature vector may mean the extracted first feature vector, the second feature vector, and the third feature vector.

Then, the neural network may be trained using the generated learning data (S600).

Here, the neural network is a statistical model that mimics the neural network of biology, and may have problem-solving ability through a learning process.

The parameters of such a neural network can be adjusted through a learning process using learning data.

Here, the neural network may be trained by a deep learning method. The deep learning method refers to a machine learning algorithm that attempts high-level abstraction through a combination of several nonlinear transformation techniques.

For example, the artificial neural network model according to the present invention may be trained using a supervised learning method. In addition, the artificial neural network model according to the present invention can learn through unsupervised learning for discovering a judgment criterion by self-learning using learning data without guidance. In addition, the artificial neural network model according to the present invention can be learned through reinforcement learning using feedback on whether the result of the situation determination according to learning is correct. In addition, the artificial neural network model according to the present invention can be trained using a learning algorithm including an error back-propagation method or a gradient decent method.

Here, the neural network includes a Bayes Neural Network (BNN), a Convolution Neural Network (CNN), a Recurrent Neural Network (RNN), and a Deep Neural Network (DNN). It may be formed in one of the structures.

Such a neural network may be learned by analyzing correlations between various input features and output values (eg, lecture evaluation information) using learning data including lecture information, feature vectors, and lecture evaluation information.

For example, one learner may feel uncomfortable with the instructor's angry feelings, while another learner may give a positive evaluation because it may be an opportunity to focus on learning. The evaluation of such a course may depend on various factors such as the subject of the lecture, the subject of the lecture, the purpose of the lecture, and the like. Accordingly, the neural network according to an embodiment of the present invention can be learned by analyzing the correlation between the emotion of the lecturer and various input values.

As another example, one learner may feel that the lecture is distracting if there is a lot of movement, and the evaluation score of the lecture may be lowered, while another learner may give a positive evaluation if the lecture is not boring because the lecture is not boring if there is a lot of movement. The evaluation of such a course may depend on various factors such as the subject of the lecture, the subject of the lecture, the purpose of the lecture, and the like. Accordingly, the neural network according to an embodiment of the present invention can be learned by analyzing the correlation between the movement of the lecturer and various input values.

As another example, one learner may give a low lecture evaluation score to an instructor who speaks slowly because it is boring, while another learner may give a high lecture evaluation score because they can understand the lesson step by step. The evaluation of such a course may depend on various factors such as the subject of the lecture, the subject of the lecture, the purpose of the lecture, and the like. Accordingly, the neural network according to an embodiment of the present invention can be learned by analyzing the correlation between the lecturer's speaking speed and various input values.

As another example, some learners may give a low lecture evaluation score because the teacher's expressionlessness interferes with forming a positive relationship with the lecturer, whereas some learners may not be very concerned about it. The evaluation of such a course may depend on various factors such as the subject of the lecture, the subject of the lecture, the purpose of the lecture, and the like. Accordingly, the neural network according to an embodiment of the present invention can be learned by analyzing the correlation between the facial expression of the lecturer and various input values.

As another example, one learner may give a low lecture evaluation score to an instructor with a rich vocabulary because it makes lectures difficult, while another learner may give a high lecture evaluation score to the lecturer because it improves the quality of education. The evaluation of such a course may depend on various factors such as the subject of the lecture, the subject of the lecture, the purpose of the lecture, and the like. Accordingly, the neural network according to an embodiment of the present invention can be learned by analyzing the correlation between the lecturer's vocabulary richness and various input values.

Meanwhile, the neural network learned using the learning data may be trained as a neural network model that receives lecture information and feature vectors and outputs expected lecture evaluation information.

Here, the expected lecture evaluation information may include an expected lecture evaluation score obtained by quantifying the evaluation of the lecture image and an expected lecture evaluation feedback obtained by evaluating the lecture image.

Here, the expected lecture evaluation score may be a numerical value of how well the lecture was conducted relative to the expected satisfaction of the learner or the purpose of the lecture, and the expected lecture evaluation feedback provides information on factors that affected the expected evaluation score. may include

Meanwhile, the above-described neural network learning method may be performed by the AI processor 150 of the lecture image evaluation apparatus 100 to be described later, and the neural network learned by the neural network learning method may be used to evaluate the lecture image.

Next, a method of evaluating a neural network-based lecture image will be described with reference to FIG. 4 .

4 is a flowchart illustrating a method of evaluating a lecture image based on a neural network according to an embodiment of the present invention.

Referring to FIG. 4 , the neural network-based method of evaluating the lecture image of the lecture image evaluation apparatus 100 may train the neural network through the neural network learning method for evaluating the lecture image ( S1000 ). Here, the step (S1000) of learning the neural network may include the above-described steps (S100) to (S600).

Next, the lecture image evaluation apparatus 100 may receive an evaluation target lecture image from the lecturer 10 (S2000). Specifically, the lecture image evaluation apparatus 100 may receive an evaluation target lecture image from the lecturer 10 through a user interface.

In addition, the lecture image evaluation apparatus 100 may receive lecture information on the lecture image from the lecturer 10 ( S3000 ). Specifically, the lecture image evaluation apparatus 100 may receive lecture information on the lecture image from the lecturer 10 through the user interface. In this regard, it will be described with reference to FIG.

Referring to FIG. 5 ( a ), the lecture image evaluation apparatus 100 provides a user interface as shown in FIG. 5 ( a ), and can receive lecture images and lecture information from the user 10 through the user interface. . Here, the user interface may include a lecture image upload area for evaluation, an area in which the uploaded lecture image is displayed, an area for receiving lecture information, and the like.

Next, the lecture image evaluation apparatus 100 may calculate expected lecture evaluation information by inputting lecture information and lecture images received from the user 10 into the learned neural network ( S4000 ).

Specifically, the neural network may analyze the lecture image to extract a first feature vector including a feature value of at least one of the lecturer's movement, the lecturer's facial expression, and the lecturer's emotion. Also, the neural network may analyze the lecture image to extract a second feature vector including a feature value of at least one of tone, size, rhythm, tone, and speed of the lecturer's voice. In addition, the neural network extracts a third feature vector including a feature value of at least one of the difficulty of the vocabulary used in the lecture content, the richness of the vocabulary, the number of repetitions of the lecture keyword, the length of the sentence, and the class duration from the lecture video. can

In addition, the lecture image evaluation apparatus 100 may calculate expected lecture evaluation information by inputting lecture information received from the user 10 and the extracted first to third feature vectors into the neural network.

An example of such a neural network according to an embodiment of the present invention will be described with reference to FIG. 6 .

6 is an exemplary diagram illustrating a neural network for evaluating a lecture image according to an embodiment of the present invention.

Referring to FIG. 6 , the neural network 30 according to an embodiment of the present invention may be configured as a Convolution Neural Network (CNN) model including layers that perform a plurality of convolution operations.

When the lecture image and lecture information are input to the learned neural network 30 , the feature values may be emphasized through convolution while passing through the inner layers of the neural network 30 .

Various feature values are output in the form of a new feature map through operation with a filter determined for each convolution layer, and the final feature map generated through iterative operation for each layer is a fully-connected layer ) and can be flattened. Estimated lecture evaluation information may be output based on the flattened characteristic information.

Next, the lecture image evaluation apparatus 100 may display the calculated expected lecture evaluation information (S5000). Specifically, the lecture image evaluation apparatus 100 may display expected lecture evaluation information through a user interface in order to provide expected lecture evaluation information to the user 10 . In this regard, it will be described with reference to FIG.

Referring to FIG. 5B , the lecture image evaluation apparatus 100 may provide a user interface as shown in FIG. 5B , and may provide expected lecture evaluation information to the user 10 through the user interface. Here, the expected satisfaction of the learner with respect to the lecture video and the appropriateness of the lecture may be displayed on the user interface, and a lecture keyword obtained by evaluating the lecture image as a keyword may also be displayed.

In addition, expected lecture evaluation feedback may be displayed in relation to the lecturer's facial expression, emotion, vocabulary, and the like.

Hereinafter, the configuration of the lecture image evaluation apparatus 100 will be described with reference to FIG. 7 .

7 is a block diagram showing the configuration of the lecture image evaluation apparatus 100 according to an embodiment of the present invention.

Referring to FIG. 7 , the lecture image evaluation apparatus 100 may include a display unit 110 , an input unit 120 , a storage unit 130 , a user interface generating unit 140 , and an AI processor 150 .

Various data related to the operation of the lecture image evaluation apparatus 100 may be displayed on the display unit 110 . For example, the display 110 may display a user interface as shown in FIG. 5 .

The input unit 120 may receive various data necessary for the operation of the lecture image evaluation apparatus 100 . For example, the input unit 120 may receive a user input for inputting lecture images and lecture information.

The storage unit 130 may store various data necessary for the operation of the lecture image evaluation apparatus 100 . Specifically, the storage unit 130 is divided into a neural network model storage unit 132 and a motion data storage unit 134 , and various data may be stored therein.

The neural network model storage unit 132 may store a neural network learned to evaluate lecture images.

The motion data storage 134 may store various data necessary for evaluating the lecture image. For example, the motion data storage unit 130 may store a lecture image to be evaluated and lecture information of the corresponding lecture image.

The user interface generator 140 may perform a function of generating a user interface provided to the user 10 . Specifically, the user interface generating unit 140 may generate a user interface for receiving lecture images or lecture information, or may generate a user interface for providing expected lecture evaluation information.

For example, the user interface generator 140 may generate a user interface as shown in FIG. 5 .

The AI processor 150 may perform a function of controlling the overall operation of the lecture image evaluation apparatus 100 . Specifically, the AI processor 150 may perform the above-described neural network learning method and a lecture image evaluation method using the learned neural network.

Specifically, the AI processor 150 may train a neural network for evaluating a lecture image. For example, the AI processor 150 obtains lecture evaluation information that evaluates lecture images and lecture images, performs pre-processing to emphasize the face and voice of the lecturer of the obtained lecture images, and analyzes the pre-processed lecture images. Feature vectors can be extracted. In this case, the feature vector may be the first feature vector, the second feature vector, and the third feature vector described above.

Then, the AI processor 150 generates learning data by labeling the input lecture information, the extracted feature vector, and the lecture evaluation information on the lecture image, and trains a neural network for evaluating the lecture image based on the generated learning data. can In this regard, since it has been described with reference to FIGS. 2 to 3 , a detailed description thereof will be omitted.

Also, the AI processor 150 may calculate expected lecture evaluation information by inputting lecture information and lecture images to the learned neural network. In this regard, since it has been described with reference to FIG. 4, a detailed description thereof will be omitted.

Meanwhile, in the specification and claims, terms such as "first", "second", "third", and "fourth" are used to distinguish between similar elements, if any, and this is not necessarily the case. Used to describe a specific sequence or sequence of occurrences. It will be understood that the terms so used are interchangeable under appropriate circumstances to enable the embodiments of the invention described herein to operate, for example, in sequences other than those shown or described herein. Likewise, where methods are described herein as comprising a series of steps, the order of those steps presented herein is not necessarily the order in which those steps may be performed, and any described steps may be omitted and/or Any other steps not described may be added to the method. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, a second component may also be referred to as a first component.

Also, terms such as "left", "right", "front", "rear", "top", "bottom", "above", "below" in the specification and claims are used for descriptive purposes, It is not necessarily intended to describe an invariant relative position. It will be understood that the terms so used are interchangeable under appropriate circumstances to enable the embodiments of the invention described herein to operate otherwise than, for example, as shown or described herein. As used herein, the term “connected” is defined as being directly or indirectly connected in an electrical or non-electrical manner. Objects described herein as being "adjacent" to one another may be in physical contact with one another, in proximity to one another, or in the same general scope or area as appropriate for the context in which the phrase is used. The presence of the phrase “in one embodiment” herein refers to the same, but not necessarily, embodiment.

In addition, in the specification and claims, references to 'connected', 'connecting', 'fastened', 'fastening', 'coupled', 'coupled', etc., and various variations of these expressions, refer to other elements directly It is used in the sense of being connected or indirectly connected through other elements.

On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

In addition, the suffixes "module" and "part" for components used in this specification are given or used in consideration of ease of writing the specification, and do not have distinct meanings or roles by themselves.

In addition, the terms used herein are for the purpose of describing the embodiments and are not intended to limit the present invention. As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps.

So far, the present invention has been looked at with respect to preferred embodiments thereof. All embodiments and conditional examples disclosed through this specification have been described with the intention of helping those of ordinary skill in the art to understand the principles and concepts of the present invention, and those skilled in the art It will be understood that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention.

Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

Meanwhile, the method according to various embodiments of the present invention described above may be implemented as a program and provided to a server or devices. Accordingly, each device can download the program by accessing the server or device in which the program is stored.

In addition, the above-described method according to various embodiments of the present invention may be implemented as a program and stored in various non-transitory computer readable media to be provided. The non-transitory readable medium refers to a medium that stores data semi-permanently, rather than a medium that stores data for a short moment, such as a register, cache, memory, and the like, and can be read by a device. Specifically, the various applications or programs described above may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims In addition, various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: lecture image evaluation device 110: display unit
120: input unit 130: storage unit
140: user interface generator 150: AI processor

Claims (11)

In the learning method of a neural network for evaluating a lecture image,
obtaining a plurality of lecture images and lecture evaluation information for each of the plurality of lecture images;
performing pre-processing of the obtained lecture image;
obtaining lecture information on the lecture video;
extracting a feature vector by analyzing the pre-processed lecture image;
generating learning data by labeling the lecture information, the feature vector, and the lecture evaluation information on the lecture image; and
A neural network learning method comprising a; learning a neural network for evaluating a lecture image based on the learning data. The method of claim 1,
Extracting the feature vector comprises:
extracting a first feature vector by analyzing visual data related to the appearance of the lecturer from the lecture image;
extracting a second feature vector by analyzing auditory data related to the lecturer's voice from the lecture image; and
and extracting a third feature vector by analyzing lecture data related to lecture contents from the lecture image.
The generating of the learning data comprises labeling the extracted first feature vector, second feature vector, and third feature vector on the lecture image. 3. The method of claim 2,
The step of extracting the first feature vector,
A method for learning a neural network, comprising extracting a first feature vector including at least one of a movement of a lecturer, an expression of the lecturer, and an emotion of the lecturer. 3. The method of claim 2,
The extracting of the second feature vector comprises extracting a second feature vector including a feature value of at least one of tone, size, rhythm, tone, and speed of the lecturer's voice. 3. The method of claim 2,
A neural network characterized by extracting a second feature vector including a feature value of at least one of the difficulty of the vocabulary used in the lecture content of the lecturer, the richness of the vocabulary, the number of repetitions of the lecture keyword, the length of the sentence, and the duration of the lesson How to learn. The method of claim 1,
The lecture information is a neural network learning method, characterized in that it includes at least one of a lecture purpose of the lecture video, a lecture subject, a lecture atmosphere, a lecture subject, and a lecture method. In a method for evaluating a lecture image based on a neural network,
training a neural network through a neural network learning method to evaluate a lecture image;
receiving a lecture video to be evaluated;
receiving lecture information on the lecture video;
The lecture image evaluation method comprising; inputting the lecture information and the lecture image to the learned neural network to calculate expected lecture evaluation information. 8. The method of claim 7,
Calculating the expected lecture evaluation information includes:
extracting a first feature vector by analyzing visual data related to the appearance of the lecturer from the lecture image to be evaluated;
extracting a second feature vector by analyzing auditory data related to the lecturer's voice from the lecture image to be evaluated;
extracting a third feature vector by analyzing the lecture content of the lecturer from the lecture image to be evaluated; and
and calculating expected lecture evaluation information by inputting the lecture information and the first to third feature vectors into a neural network. In the neural network-based lecture image evaluation apparatus,
a neural network model storage unit for storing a neural network learned through a neural network learning method to evaluate a lecture image;
an input unit for receiving a lecture image to be evaluated and receiving lecture information on the lecture image; and
An AI processor for calculating expected lecture evaluation information by inputting the lecture information and the lecture image to the learned neural network; a neural network-based lecture image evaluation apparatus comprising a. A program recorded in a computer-readable recording medium in which a program code for executing a neural network learning method for evaluating a lecture image according to any one of claims 1 to 6 is recorded. A program recorded on a computer-readable recording medium in which a program code for executing a method for evaluating a neural network-based lecture image according to any one of claims 7 to 8 is recorded.

Images