KR20230092161A - Presentation video summary method and summary device for performing the same - Google Patents

Abstract

The present invention relates to a method for summarizing a presentation video performed by a presentation video summary device, comprising the steps of receiving a presentation video; Extracting frames from the presentation video; extracting an object within the extracted frame and masking a region of the extracted object within the extracted frame; extracting first text by reading characters in a frame in which a partial area is masked; clustering the extracted first text; and documenting the presentation video based on the clustered data. Accordingly, it is possible to create a document effective for video management by making the contents of the presentation video into a single document, and to improve the learner's learning speed.

Description

Presentation video summary method and summary device that performs it {PRESENTATION VIDEO SUMMARY METHOD AND SUMMARY DEVICE FOR PERFORMING THE SAME}

The present invention relates to a presentation video summary method and a summary device performing the same, and more particularly, to a presentation video summary method capable of summarizing and documenting a presentation video and a summary device performing the same.

Deep learning is a field of machine learning that attempts a high level of abstraction through a combination of several nonlinear transform methods. Network, RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), and Deep Q-Networks. It exists and is used in various application fields such as image recognition, natural language processing, and automatic voice recognition.

On the other hand, as non-face-to-face learning and video lectures are explosively increasing in accordance with the epidemic pandemic along with the development of technology, it is necessary to develop a methodology that enables learners to learn quickly as well as the need for efficient video management. Regarding data summary, many studies on extracting picture data from papers have been conducted, but there is a limitation that this is not suitable for dynamically changing video lectures.

Korean Patent Publication No. 10-2011-0121964

The present invention has been made to solve the above problems, and an object of the present invention is to create a document effective for video management by making the contents of a video presentation into one document, as well as to improve the learning speed of learners. It is to provide a presentation video summary method for summarizing the presentation video and a summary device that performs the same.

A presentation video summary method performed by a presentation video summary apparatus according to an embodiment of the present invention for achieving the above object includes receiving a presentation video; extracting frames from the presentation video; extracting an object within the extracted frame, and masking a region of the extracted object within the extracted frame; extracting first text by reading characters in a frame in which a partial area is masked; clustering the extracted first text; and documenting the presentation video based on the clustered data.

extracting second text from the audio included in the presentation video; generating a summary by summarizing the extracted second text at a predetermined ratio through a preset function; and extracting a keyword based on the extracted first text and the extracted second text.

In the masking step, when a plurality of objects are extracted from the frame and the areas of each object overlap each other, the areas of each object are merged into one image, and the merged image in the extracted frame In the step of masking and documenting, the presentation video may be documented based on the clustered data, the merged image, the summary, and the keyword.

In addition, in the step of extracting the first text, location information of the first text in the masked frame is extracted together with the first text, and in the step of clustering, based on the extracted location information, The extracted first text may be clustered.

On the other hand, the presentation video summary device according to an embodiment of the present invention for achieving the above object extracts a frame from the received presentation video, extracts an object within the extracted frame, and extracts the object extracted from the extracted frame. an image unit that masks an area, reads characters in a frame in which a partial area is masked, extracts first text, and clusters the extracted first text; Second text is extracted from the voice included in the received presentation video, and a summary is generated by summarizing the extracted second text at a predetermined ratio through a preset function, and the extracted first text and the second text are summarized. a voice unit for extracting keywords based on; and a combiner for documenting the presentation video based on the clustered data, the summary, and the keywords.

According to one aspect of the present invention described above, by providing a method for summarizing a presentation video and a summary device that performs the same, it is possible to create a document that is efficient in video management by making the contents of a presentation video into a single document, as well as to speed up learners' learning can improve

1 is a block diagram for explaining the configuration of a presentation video summary device according to an embodiment of the present invention;
2 is a diagram for explaining a framework of a presentation video summary device according to an embodiment of the present invention;
3 is a block diagram for explaining the configuration of an image unit according to an embodiment of the present invention;
4 is a view for explaining how an object is extracted from an image unit according to an embodiment of the present invention;
5 is a diagram for explaining pseudocode for extracting an object from an image unit according to an embodiment of the present invention;
6 is a block diagram for explaining the configuration of an audio unit according to an embodiment of the present invention;
7 is a diagram for explaining a final result in which a presentation video output from a presentation video summary device according to an embodiment of the present invention is documented;
8 is a flowchart for explaining a presentation video summary method performed in a presentation video summary device according to an embodiment of the present invention, and
9 to 12 are views for explaining experimental results of performance evaluation of a method for summarizing a presentation video according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in another embodiment without departing from the spirit and scope of the invention in connection with one embodiment. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Components according to the present invention are components defined not by physical division but by functional division, and may be defined by the functions each performs. Each of the components may be implemented as hardware or program codes and processing units that perform respective functions, and the functions of two or more components may be implemented by being included in one component. Therefore, the names given to the components in the following embodiments are not to physically distinguish each component, but to imply the representative function performed by each component, and the names of the components indicate the present invention. It should be noted that the technical idea of is not limited.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram for explaining the configuration of a presentation video summary apparatus 100 according to an embodiment of the present invention, and FIG. 2 is a block diagram for explaining a framework of a presentation video summary apparatus according to an embodiment of the present invention. it is a drawing The presentation video summary device 100 of the present invention is an input unit 110 to improve the efficiency of education by making the contents of a video including presentation materials for learning or presentation such as Power Point into one document. , It may include a communication unit 120, a storage unit 130, a control unit 140 and an output unit 150.

The input unit 110 is an input means for receiving a user command and may receive a presentation video requiring a summary, and the output unit 150 may include a display for displaying the process and results of documenting the presentation video. there is.

The communication unit 120 is provided to transmit and receive various types of information to and from the outside through a network, and transmits and receives transmitted and received information to the control unit 140 or can be stored in the storage unit 130 . In addition, the communication unit 120 is provided to transmit and receive necessary information from an external device or external network, and through this, learning data or a presentation video for documentation can be input.

The storage unit 130 stores a program for performing the presentation video summary method, provides a storage space necessary for the controller 140 to operate, temporarily or permanently stores data processed by the controller 140, and volatile It may include a storage medium or a non-volatile storage medium, but the scope of the present invention is not limited thereto. In addition, the storage unit 130 may store data accumulated while performing the presentation video summary method.

Meanwhile, the controller 140 extracts text and objects (tables, graphs, pictures, etc.) from frames and voices included in the presentation video, and merges the extracted information to generate a document summarizing the contents of the announcement video. can In particular, the control unit 140 according to the present invention extracts location information, which is a structure such as pictures and texts, in presentation materials included in a presentation video, and reflects the corresponding structure in the finally generated document to list summary contents. To this end, the control unit 140 may include an image unit 141, an audio unit 143, and a coupling unit 145.

As shown in FIG. 2, the image unit 141 extracts a frame from an input or received presentation video, extracts an object within the extracted frame, masks an area of the extracted object within the extracted frame, and partially Characters within the masked frame may be read to extract first text, and the extracted first text may be clustered.

The audio unit 143 extracts second text from the audio included in the input or received presentation video, summarizes the extracted second text at a predetermined ratio through a preset function to generate a summary, and generates a summary of the extracted first text. And keywords can be extracted based on the second text.

The combination unit 145 may combine the clustered data generated or extracted from the image unit 141 and the audio unit 143, the extracted object, summary, and keywords to document the presentation video as a single document.

On the other hand, Figure 3 is a block diagram for explaining the configuration of the image unit 141 according to an embodiment of the present invention, Figure 4 is a state of extracting an object from the image unit 141 according to an embodiment of the present invention A drawing for explanation, and FIG. 5 is a diagram for explaining pseudo code for extracting an object from the image unit 141 according to an embodiment of the present invention.

The image unit 141 is provided to extract first text from a frame included in a presentation video received through the communication unit 120 or input through the input unit 110 and to cluster the extracted first text, FIG. 3 As shown in , it may include a frame extraction unit 1411, an object extraction unit 1413, a text reading unit 1415, and a clustering unit 1417.

The frame extractor 1411 may be provided to extract each frame from the input presentation video. A process of extracting a frame by the frame extractor 1411 may be as follows. First, the frame extraction unit 1411 stores the first frame of a video in a PNG format, which is a picture file format, and sets the stored first frame as a comparison frame for selecting the next frame. After that, the frame extraction unit 1411 selects a frame that is 1 second apart from the comparison frame in the video as the current frame. Then, the frame extraction unit 1411 may calculate SSIM, which is a structural similarity index between the current frame and the comparison frame, that is, the previous frame, based on Equation 1 below.

[Equation 1]

는 비교 프레임과 현재 프레임인 이미지 x, y의 국소 평균,

,

는 표준편차,

는 교차 공분산, 그리고

는 국소 평균 또는 표준편차가 0에 가까운 이미지에 의한 불안정성 방지를 위한 정규화 상수이다. (0이 아닌 작은 값)here

is the local average of the comparison frame and the image x, y that is the current frame,

,

is the standard deviation,

is the cross-covariance, and

is a normalization constant for preventing instability caused by an image having a local average or standard deviation close to zero. (small non-zero value)

If the similarity index calculated through Equation 1 is lower than a specific threshold set in advance by the user, the current frame is determined as a new presentation material different from the comparison frame and saved in a picture file format (for example, PNG format) This may be selected as a comparison frame again.

The frame extraction unit 1411 can extract frames that are different presentation materials by repeatedly performing the above process until the video ends.

Meanwhile, the object extractor 1413 may receive the frames extracted from the frame extractor 1411, extract an object from each extracted frame, mask an area of the object extracted from the extracted frame, and A merger 14131 may be included.

The object extraction unit 1413 may use MASK-RCNN, one of object detection algorithms, as an image region recognition model to detect picture data and formulas in the extracted frames, that is, presentation materials.

In addition, the object extractor 1413 can learn with the MASK-RCNN model by labeling the object area in the extracted frame as 'Figure' and 'Formula', and more specifically, labels tables, graphs, and figures as 'Figure', , the formula can be labeled as 'Formula'.

In the case of the conventional MASK-RCNN, there is a limit to recognizing one picture data as multiple. More specifically, taking FIG. 5 as an example, the existing MASK-RCNN has a problem of recognizing a partial region of an object by dividing it into several parts due to a blank or a caption for the object among the entire object.

Accordingly, the object extraction unit 1413 according to the present embodiment is provided to include a region merging unit 14131 to calculate the intersection of each region of the detected object as described above, and if there is an intersection region, that is, an overlapping region, the corresponding region It is possible to solve the conventional problem by merging the images into one extended area including the images and integrating them into one image.

When a plurality of objects are extracted from the frame and the regions of each object overlap each other, the region merging unit 14131 merges the regions of each object and integrates them into a single image, and the object extraction unit 1413 merges the regions. In unit 14131, the integrated image may be masked within the extracted frame.

Specifically, as shown in FIG. 3, when a plurality of objects o1, o2, o3, o4, and o5 are extracted from the extracted frame, and the areas of each object overlap as shown in the drawing, the object extraction unit 1413 Through the region merging unit 14131, a plurality of objects o1, o2, o3, o4, and o5 having overlapping regions may be merged into one image o. Then, the object extraction unit 1413 may perform masking by extracting picture data from one integrated image o and replacing the extracted area with a white box.

Accordingly, the object extraction unit 1413 including the area merging unit 14131 may merge the overlapping object areas based on the pseudo code of the merging function as shown in FIG. 5 . The merge function according to the present embodiment first receives input values of 'Figure' and 'Formula' regions recognized through the MASK-RCNN model, and may determine whether the recognized regions intersect. At this time, the merging method draws a box based on the coordinates located at the top left and the bottom right among the coordinates of the regions. The above process may be repeated until the region where the intersection occurs no longer exists. And finally, one image (o) region integrated may be masked by replacing it with a white box in order to extract the first text.

Meanwhile, the character reading unit 14131 may extract first text by reading characters in a frame in which some areas, that is, a combined image o area are masked by the area merging unit 14131 . The character reading unit ) may extract the first text by applying OCR using pytesseract, a Python library based on the Google Tesseract engine. Through this, the text reading unit 1415 may extract text by storing text OCT data in the image in a data frame format when receiving a frame in which a partial area is masked. Then, the text reading unit 1415 may extract first text by reconstructing text data extracted in word units in line units using an aggregation function.

Positions where the texts included in the frame ( ) are arranged may be different depending on importance. Accordingly, the text reading unit 1415 may extract location information of the first text together with extraction of the first text. More specifically, taking presentation materials such as PowerPoint (PPT, PowerPoint) as an example, if you look at the layout of one slide, that is, the text arrangement, the title is placed at the top, and the details or details of the title are placed at the bottom of the title through indentation, etc. A table of contents will be placed. Accordingly, the text reading unit 1415 may also extract location information of the first text based on this information.

Meanwhile, the clustering unit 1417 is provided to perform Hierarchy clustering on the first text extracted from the text reading unit 1415, based on location information extracted from the text reading unit 1415. The extracted first text may be clustered.

The clustering unit 1417 can restore the hierarchy according to the table of contents of the document by applying Agglomerative Clustering of the Python library scikit-learn to the data frame extracted by the text reading unit 1415. Specifically, the clustering unit 1417 selects the number of clusters in the slide, that is, within 1 to 5, determines the clusters of the first text using Agglomerative Clustering, and then calculates a silhouette score for each cluster to obtain a score. A process of selecting the highest number of clusters may be performed, and through such a clustering unit 1417, the combination unit 145 may catalog texts in a finally created document and document them.

The data extracted, selected, and generated in the image unit 141 including the frame extractor 1411, the object extractor 1413, the text reader 1415, and the clustering unit 1417 are sent to the combiner 145. can be conveyed

Meanwhile, FIG. 6 is a block diagram for explaining the configuration of the audio unit 143 according to an embodiment of the present invention. The voice unit 143 recognizes a voice included in a presentation video received through the communication unit 120 or inputted through the input unit 110, extracts second text from the recognized voice, and based on the extracted second text It is provided for generating a summary or extracting keywords, and may include a text extraction unit 1431 and an analysis unit 1433.

The text extraction unit 1431 may be provided to recognize voice included in the presentation video and extract second text from the recognized voice. The text extractor 1431 in this embodiment is a STT (Speech To Text) capable of extracting text from a conventional video, that is, an audio file, and can utilize software such as Amazon Transcribe provided by Amazon. there is.

Meanwhile, the analysis unit 1433 may generate a summary by summarizing the second text extracted by the text extraction unit 1431 at a predetermined ratio through a preset function. The analysis unit 1433 may use the TextRank algorithm using TF-IDF for summarization, and may use the Python library gensim to apply the corresponding algorithm. In addition, the analysis unit 1433 may generate a summary by summarizing the entire article at a rate of 70% through a function provided by gensim, which is a preset function.

The analysis unit 1433 may extract keywords based on the first text extracted from the image unit 141 and the second text from the text extraction unit 1431 . Specifically, a predetermined number of keywords may be extracted from the first text extracted by the text reader 1415 of the image unit 141 and the second text extracted by the text extractor 1431 . The analysis unit 1433 according to this embodiment is set to extract a total of 50 keywords, but is not limited thereto and can be changed as much as possible. To extract keywords, the analyzer 1433 first merges the first text and the second text to create one text file, sets this as an input value, and extracts 50 keywords by applying a TextRank algorithm. The reason why about 50 keywords are extracted here is that a meaningful word is created by combining several words such as compound words.

In the present invention, the analyzer 1433 extracts keywords by merging the first text and the second text instead of extracting the keywords only from the second text as described above. This is because it has been confirmed that when keywords are extracted using only the second text, the frequency of extracting filler words, which fill in the middle words with keywords, is higher than when the first text and the second text are merged. Therefore, in order to prevent such a filler word from being extracted as a keyword, the analysis unit 1433 of the present invention merges the first text and the second text and extracts a keyword therefrom.

Meanwhile, FIG. 7 is a diagram for explaining the final result in which the presentation video output from the presentation video summary device 100 according to an embodiment of the present invention is documented. As described above, the combining unit 145 may document the presentation video as a single document by combining the clustered data generated or extracted from the image unit 141 and the audio unit 143, the extracted object, summary, and keywords. there is.

7 is a diagram showing an actual result finally generated by the combining unit 145 of the present invention. In the combining unit 145, the text for a frame, which is a presentation material in a document, is displayed on the left as slide contents. , and the text extracted from the voice in the presentation video can be placed on the right side as speech content. Of course, it is not necessarily limited to this arrangement, and the arrangement may be changed as much as possible by setting in consideration of user's readability.

When generating a document, the combiner 145 arranges the first text extracted by the text reader 1415 within the slide content, and lists the text as shown based on the clustering through the clustering unit 1417. can be placed.

In addition, the combiner 145 arranges the summary generated by the analyzer 1433 within the speech content, but the same text as the keyword extracted from the analyzer 1433 among the texts located in the slide content ( ) has a different thickness. can be highlighted.

In addition, the coupling unit 145 may insert and place the objects (graphs, pictures, formulas, etc.) extracted by the object extraction unit 1413 into the speech content as reference data. Also, the coupling unit 145 may generate and place a reference link under each slide title for the extracted object so that the user can move within the document.

Through this, the presentation video summary device 100 of the present invention can improve the efficiency of education by generating reference materials for video lectures, and can create a document effective for video management by making the contents of the video into a single document. . In particular, since non-face-to-face learning and video lectures, which are explosively increasing, can be documented as a single document, it will be possible to improve the learning speed of learners.

Meanwhile, FIG. 8 is a flowchart illustrating a method for summarizing a presentation video performed by the apparatus 100 for summarizing a presentation video according to an embodiment of the present invention.

The presentation video summary method according to the present embodiment may be performed in substantially the same configuration as the presentation video summary device 100 of FIGS. 1 and 2, and therefore, repeated description is omitted. In addition, the presentation video summary method according to this embodiment may be executed by software (application) for performing a presentation video summary.

First, the presentation video summary device 100 performs a step (S110) of receiving a presentation video. The presentation video may be received from an external device or an external network, or may be directly input by the user.

Thereafter, a step of extracting a frame from the presentation video (S120) may be performed. As described above, one frame is extracted at an interval of 1 second, and SSIM, a structural similarity index with the previous frame, is calculated. After determining that it is data, the current frame can be saved in a picture file format. After that, the current frame saved in a picture file format is selected as a comparison frame again, and the frame extracted after 1 second is selected as the current frame, and the above process is completed when the video is finished. It can be repeated until If the calculated similarity index is higher than the threshold value, the corresponding frame is judged as the same presentation material and is not saved in a picture file format.

Then, a step of extracting an object within the extracted frame and masking a region of the extracted object within the extracted frame (S130) may be performed. Here, the extracted frame may mean a frame stored in a picture file format.

In addition, in the masking step (S130), when a plurality of objects are extracted from the frame and the regions of each object overlap each other, the regions of each object are merged and integrated into one image, and the merged image is masked in the extracted frame can do.

Specifically, objects such as pictures, graphs, formulas, etc. are extracted from the extracted frame, but when each object area overlaps with each other, the overlapping object areas are merged into one area to integrate into one image, that is, one object. . This is to solve the problem of judging different objects by blank spaces or texts included in a picture even though they are one object.

Thereafter, a step of extracting first text by reading characters in a frame in which a partial region is masked (S140) is performed. In the step of extracting the first text ( S140 ), location information of the first text within the masked frame may be extracted together with the first text.

Then, a step (S1) of clustering the extracted first text may be performed. In the clustering step ( S150 ), the extracted first text may be clustered based on the extracted location information.

Then, a step of documenting the presentation video based on the clustered data (S160) may be performed.

In addition, the presentation video summary method is a step performed before the documenting step (S160), and extracts second text from the audio included in the announcement video, and summarizes the extracted second text at a certain ratio through a preset function. The method may further include generating a summary and extracting keywords based on the extracted first text and the second text.

Here, the step of extracting the second text from the voice included in the announcement video may be a step of recognizing the voice included in the announcement video and extracting the second text from the recognized voice.

Further, in the step of generating a summary by summarizing the extracted second text at a predetermined ratio through a preset function, the summary may be generated by summarizing the entire text of the second text at a ratio of 70%.

Meanwhile, in the step of extracting a keyword based on the extracted first text and the second text, a text file is created by merging the first text and the second text, and a TextRank algorithm is applied by setting this as an input value to obtain about 50 keywords. Several keywords can be extracted.

Therefore, in the documenting step (S160), the presentation video may be documented based on the above-described clustered data as well as merged images, summaries, and keywords.

Meanwhile, FIGS. 9 to 12 are views for explaining experimental results of performance evaluation of the presentation video summary method according to an embodiment of the present invention. Hereinafter, experimental results obtained by evaluating performance of extracting an image region from presentation data and text extraction performance from presentation data performed by the summary device 100 according to the present embodiment will be described. In addition, considering the limited number of presentation materials and labeling results used for the experiment, transfer learning was adopted, and "MASK-RCNN-coco.h5" provided by default in MASK-RCNN for transfer learning. was used as a pre-learning model. Batch 1, epoch 100, and learning rate 0.001 were specified as parameters for learning, and an AWS p2.xlarge instance environment with Ubuntu 18.04 operating system installed was adopted for the experiment. For implementation, Python 3.8.5, TensorFlow 2.2.0, Keras 2.3.1, and Cuda 10.1 were used.

Recall and Precision were used as evaluation indexes to evaluate image area extraction performance. In addition, the result of adding the configuration of the region merging unit 14131 of the present invention to the result of the conventional transfer learning model and the result of the transfer learning model were compared.

Meanwhile, in order to evaluate the text extraction performance, the text data extracted from the presentation video was saved as a TXT file, and each TXT file was vectorized based on TF-IDF, and then cosine similarity, Euclidean distance, jaro winkler similarity, and levenshtein distance were used as evaluation indicators. was used to measure the similarity with the actual text. In addition, to evaluate the performance of the present invention, 1) the result of simply applying OCR, 2) the result of applying OCR after masking the object area according to the transfer learning model, and 3) the object extracted and merged according to the transfer learning model After masking the region, the results of applying OCR were compared.

In addition, the STT performance measurement evaluation index for extracting text from speech is the same as the above-mentioned text performance measurement evaluation index, and the application result was compared with the actual script, and the summary creation, keyword extraction, and documentation were qualitatively evaluated.

As a data set for the experiment, the presentation data set was directly collected for object recognition in the presentation data, and thesis review PPT presentation videos released on YouTube were mainly extracted, and a total of 71 presentation videos were extracted. Most of the presentation materials consisted of text along with image areas such as figures, formulas, and tables, and a total of 1,123 presentation material images were extracted through this. Among them, 831 sheets of train data, 192 sheets of validation data, and 100 sheets of test data were used. For the object recognition model, each image was largely classified into two types, and pictorial materials were labeled as "Figure" and formulas as "Formula". The labeling object of Train data consists of 1,179 "Figure" and 537 "Formula" objects, and the Validation data consists of 414 "Figure" and 187 "Formula" objects. Also, in the test data, 114 Figure" and 18 "Formula" objects. It is a dog.

The test data for the evaluation of STT performance consisted of 5 images, and a total of 71 minutes and 6 seconds of speech was used.

9 shows a loss graph obtained by transfer learning according to the MASK-RCNN pre-learning model. According to the result, it can be confirmed that the loss converges around 0.7. In this experiment, epoch 95 with the least loss among the models before overfitting occurred was used (Training loss: 0.727083, Validation loss: 0.323328).

And [Table 1] below is the present invention in which the merge function is applied as an evaluation result before/after application of the merge function for merging object domains of Mask-RCNN transfer learning. Although there is no significant difference between recall and precision after merging is applied, it can be seen that the image area is identified with high accuracy in both cases. In addition, it can be seen that the recall mean increased (approximately 0.008 increase) and the precision mean decreased (approximately 0.01 decrease). Recall improvement leads to performance improvement of text extraction results, which will be described later.

[Table 1]

And the text extraction performance according to the application of MASK-RCNN and merger function is evaluated. 10 is an example of actual use of the framework of the present invention, and the entire process is as follows. First, the original image of FIG. 10 (a), that is, the extracted frame, is input to the MASK-RCNN model, and objects such as pictures, equations, and tables in the frame are extracted as shown in FIG. 10 (b). Then, if the extracted object is removed, that is, masked, a frame in which a partial area is masked is generated as shown in FIG. 10 (c). By applying OCR to a frame in which a part of the created region is masked, the resulting text as shown in FIG. 10 (d) can be extracted.

1) In the case of simply applying OCR, the extracted frame of FIG. 10 (a) was used as an input, 2) in the case of applying OCR after masking the object area according to the transfer learning model, and 3) in the case of applying OCR according to the transfer learning model In the case of applying OCR after extracting an object and masking the merged object region, a frame in which a partial region of FIG. 10 (c) is masked is used as an input.

Table 2 below shows the results of the four text similarity algorithms described above. All values are average values and rounded to four decimal places. For Euclidean Distance and Levenshtein Distance, the lower the index, the higher the similarity, and the higher the index, the higher the similarity for the Cosine Similarity and Jaro-Winkler Similarity. In the case of Cosine Similarity and Euclidean Distance, the extracted text was vectorized based on TF-IDF to measure the similarity, and the Jaro-Winkler Similarity and Levenshtein Distance measured the similarity using the extracted text itself.

As a result of performance evaluation, it was confirmed that the case where Mask-RCNN was applied in all four indicators significantly improved the text extraction performance compared to the case where simple OCR was applied, and the performance was further improved after applying the merge function as in the present invention. .

[Table 2]

The following describes the relationship between the change in precision and recall performance of object area extraction due to the application of the merge function in [Table 1] and the text extraction performance in [Table 2]. In [Table 1], a slight increase in recall and a slight decrease in precision are observed after applying the merge function. On the other hand, when the merge function is applied in the text extraction performance evaluation results, a clear performance improvement is observed in all indicators. This shows that the increase in recall for the image area due to merging had a positive effect on the improvement of text extraction performance, but the decrease in precision did not affect the decrease in text extraction performance.

Specifically, the reason why recall increases after merging is applied is that the missing image area before merging was extracted through merging. At this time, the text included in the missing image area is removed after applying the merger, leading to improvement in text extraction performance. And the reason why the precision decreases after merging is applied is that text or backgrounds that are not included in true positives may be included due to the nature of the merging function that identifies an image area as a single large rectangle covering two areas. According to the result of the text extraction accuracy greatly increased in the experimental results of [Table 2], the decrease in precision can be confirmed as the result of including the background rather than the removal of the text that needs to be extracted. This can be attempted to improve the accuracy of image area recognition by improving the merge function so that unnecessary backgrounds are not included.

Meanwhile, the text extraction performance of AWS Transcribe is evaluated through the STT text extraction performance evaluation. [Table 3] shows the results of the above-mentioned four text similarity algorithms.

[Table 3]

As above, the similarity of Levenshtein Distance was significantly lower than that of OCR. This is because filter words not included in the actual script were included through STT. This is connected to the use of TextRank Algorithm for summary creation and keyword extraction, which will be described later.

Hereinafter, the usefulness of the summary creation and keyword generation of the present invention will be described for evaluation. For evaluation of keyword extraction, the text extracted from the STT, that is, the result value of extracting the keyword using only the second text, and the first text (text extracted after region merging) and the second text are used together as in the present invention The results of extracting keywords were compared.

First, Figure 11 is the result of making a summary based on the actual TextRank algorithm. The summary shows sufficient usefulness, and it can be confirmed that this is due to the deletion of meaningless sentences such as filler words from the STT text extract.

Meanwhile, FIG. 12 is a diagram comparing text extraction through STT, that is, a keyword result using only the second text and a keyword result using both the first text and the second text as in the present invention. Both results were ranked in descending order based on important keywords. In all of the keyword results, the important keywords appear at the top, but as they go down, words that can be understood only by understanding the context, such as little and earlier, are extracted from keywords using only the second text, while as in the present invention, the first text And in the keywords using the second text together, it can be confirmed that comparatively words are extracted based on nouns even when going down. From this, it can be seen that setting the first text and the second text as input values is relatively meaningful in the task of extracting actual keywords.

As described above, it was shown that the presentation video summary method according to this embodiment recognizes the image area, that is, the object area, in the presentation material with high accuracy, and the accurate recognition of this object area greatly improves the performance of text extraction accuracy. Able to know. In particular, it can be seen that when the merge function for obtaining the intersection of object regions is applied to improve the performance of text extraction accuracy, the text extraction performance accuracy is greatly improved compared to the case where object regions overlap.

The presentation video summary method of the present invention can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although various embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is commonly used in the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: presentation video summary device 110: input unit
120: communication unit 130: storage unit
140: control unit 141: image unit
1411: frame extraction unit 1413: object extraction unit
14131: region merging unit 1415: text reading unit
1417: clustering unit 143: voice unit
1431: text extraction unit 1433: analysis unit
145: coupling unit 150: output unit

Claims (5)

In the presentation video summary method performed by the presentation video summary device,
Receiving an announcement video;
extracting frames from the presentation video;
extracting an object within the extracted frame, and masking a region of the extracted object within the extracted frame;
extracting first text by reading characters in a frame in which a partial region is masked;
clustering the extracted first text; and
A method for summarizing a presentation video comprising the step of documenting the presentation video based on the clustered data. According to claim 1,
extracting second text from audio included in the presentation video;
generating a summary by summarizing the extracted second text at a predetermined ratio through a preset function; and
The presentation video summary method further comprising extracting a keyword based on the extracted first text and the extracted second text. According to claim 2,
In the masking step,
When a plurality of objects are extracted from the frame and the regions of each object overlap each other, the regions of each object are merged and integrated into a single image, and the merged image is masked in the extracted frame;
In the documenting step,
and documenting the presentation video based on the clustered data, the merged image, the summary, and the keyword. According to claim 1,
In the step of extracting the first text,
Extracting location information of the first text in the masked frame together with the first text;
In the clustering step,
A method for summarizing a presentation video, characterized in that the extracted first text is clustered based on the extracted location information. A frame is extracted from the received presentation video, an object is extracted from the extracted frame, an area of the object extracted from the extracted frame is masked, and a character in the frame in which a partial area is masked is read to extract the first text and an image unit which clusters the extracted first text;
Second text is extracted from the voice included in the received presentation video, and a summary is generated by summarizing the extracted second text at a predetermined ratio through a preset function, and the extracted first text and the second text are summarized. a voice unit for extracting keywords based on; and
A presentation video summary device comprising a combiner for documenting the presentation video based on the clustered data, the summary, and the keywords.

Images