KR102512396B1 - Automatic extraction method of e-sports highlights based on multimodal learning - Google Patents

Abstract

The automatic highlight image generation method performed by the apparatus for automatically generating a highlight image of the present invention includes the steps of inputting an image; extracting video features, audio features, and chatting features as raw features from the input video; pre-processing the extracted raw features to a predetermined size and then connecting them; receiving connected features and selecting frames classified as highlight frames as video clips to be included in a highlight image; and generating a highlight image by connecting the selected image clips. As a result, it is possible to improve the production efficiency of video contents by automatically generating highlights in the relay video of E-sports.

Description

Method for automatically generating multimodal learning-based E-sports highlight video and device for performing the same

The present invention relates to a method for automatically generating a multimodal learning-based E-sports highlight video and an apparatus for performing the same, and more particularly, to a multimodal learning-based method capable of automatically generating a highlight video by extracting only important highlight scenes from an original video. It relates to a method for automatically generating an E-sports highlight video and an apparatus for performing the same.

In the case of E-Sports, which is increasingly popular as one of the sports, it was selected as an official sport in the 2018 Asian Games, and as of 2019, the number of viewers of E-Sports reached 443 million.

In addition, Twitch, a relay broadcasting platform that holds 65% of E-Sports viewers, provides E-Sports game relay and highlight videos. The event is being held and a highlight video is being produced.

In particular, with the remarkable growth of digital video content, the need to extract more important events than the rest of the content, i.e. highlights, has emerged. Therefore, many studies have been conducted to extract highlights. Also, according to a recent YouTube announcement, the viewing time of sports highlight videos increased by about 80% in 2017 compared to 2016.

As game-related video contents such as E-sports and real-time game video streaming spread, the amount of video in the game field increases as in other fields, and it is necessary to extract highlights from the video. -As the amount of sports game videos increases, the need for an efficient method of indexing and sharing through various channels is emerging.

In addition, general gamers want to see highlights in E-sports videos and real-time game video streaming videos, but long broadcasting time is an obstacle for viewers who judge viewing and watching broadcasts to enter new broadcasts, so extracting video highlights solves this problem. can be said to be one of the methods.

However, when producers provide highlight videos directly or through professional editors, there is a problem that it takes too much time and money, so the need for automatic highlight extraction to automatically extract only important scenes from the original video is increasing. there is.

Korean Patent Publication No. 2010-0114131

The present invention has been made to solve the above problems, and an object of the present invention is a multi-modal learning-based E-sports highlight video that can automatically extract highlights from an E-sports relay video using multi-modal deep learning. It is to provide an automatic generation method and a device for performing the same.

To achieve the above object, a method for automatically generating a highlight image performed by an apparatus for automatically generating a highlight image according to an embodiment of the present invention includes inputting an image; extracting a raw feature from an input video, wherein the raw feature includes a video feature, an audio feature, and a chat feature; pre-processing the extracted raw features to a predetermined size and interconnecting the pre-processed video features, audio features, and chatting features; classifying highlight frames from the concatenated video feature, audio feature and chat feature; selecting the classified highlight frames as video clips to be included in a highlight video; and generating a highlight video by connecting the selected video clips.

Here, the step of interconnecting the preprocessed video feature, audio feature, and chat feature includes the extracted video feature, audio feature, and chat feature included in the extracted raw feature through a Long Short-Term Memory (LSTM) model. They can be interconnected by resizing to the same size.

In the step of classifying the highlight frame from the connected video, audio, and chat features, a single Fully-connected (FC) layer is added to the LSTM model to obtain the connected video, audio, and chat features through the LSTM model. can be classified as a highlight frame or a non-highlight frame.

The method may further include applying a smoothing technique based on a moving average to at least one of the extracted original features before performing the connecting step.

Meanwhile, an apparatus for automatically generating a highlight image according to an embodiment of the present invention for achieving the above object includes an input unit for receiving an image; an extraction unit for extracting raw features from an input video, wherein the original features include video features, audio features, and chatting features; a connection unit for pre-processing the extracted raw features to a predetermined size and interconnecting the pre-processed video features, audio features, and chatting features; a clip creation unit that classifies highlight frames from the connected video, audio, and chatting features and selects the classified highlight frames as video clips to be included in a highlight video; and a highlight generator generating a highlight image by connecting the selected image clips.

According to one aspect of the present invention described above, by providing a multimodal learning-based method for automatically generating an E-sports highlight video, it is possible to improve video content production efficiency by automatically generating a highlight in an E-sports relay video.

In addition, according to the present invention, the overall performance of automatic highlight extraction can be improved by integrating data collected from multiple sources using multimodal deep learning and selecting effective features for each data.

1 is a block diagram for explaining the configuration of an apparatus for automatically generating a highlight image according to an embodiment of the present invention;
2 is a flowchart for explaining a method for automatically generating a highlight image according to an embodiment of the present invention;
3 is a diagram for explaining how extracted raw features are connected according to an embodiment of the present invention;
4 is a state of connecting raw features extracted according to an embodiment of the present invention after preprocessing, and,
5 is a diagram for explaining the multimodal learning-based model of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in another embodiment without departing from the spirit and scope of the invention in connection with one embodiment. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram for explaining the configuration of an apparatus 10 for automatically generating a highlight image according to an embodiment of the present invention. 15), an output unit 17 and a processor 19 are provided.

The input unit 11 is an input means for receiving a user command and can receive a video for highlight extraction, such as a game video or a relay video, and the output unit 17 is a function for displaying the process and result of automatic highlight extraction. may include a display.

In addition, the communication unit 13 is provided to transmit and receive necessary information from an external device or external network, and through this, learning data or a video for extracting highlights can be input.

The memory 15 stores a program for automatically generating a highlight image, provides a storage space necessary for the processor 19 to operate, temporarily or permanently stores data processed by the processor 19, and is volatile. It may include a storage medium or a non-volatile storage medium, but the scope of the present invention is not limited thereto. In addition, the memory 15 may store data accumulated while performing the automatic highlight image generation method.

Meanwhile, the processor 19 learns according to multimodal learning, which is a deep learning model for automatic highlight extraction, selects video clips to be included in the highlight video using the multimodal learning-based model, connects the selected video clips, and highlights With CPUs and GPUs for generating images, this processor 19 can control the entire process of providing a method for automatically generating a highlight image, and software (application) for this can be installed and executed.

The processor 19 may include an extraction unit 191, a connection unit 193, a clip generation unit 195, and a highlight generation unit 197 in order to perform the aforementioned method of automatically generating a highlight image.

The extraction unit 191 may extract video features, audio features, and chatting features as raw features from video data, audio data, and chatting data included in the video input through the input unit 11 or the communication unit 13, respectively.

The extraction unit 191 extracts all of the source data, which is information that affects the win/loss probability, such as the gold obtained, the number of destroyed towers, the number of opponent characters defeated, and the number of Barons and Dragons in each team, taking LoL during the game as an example. It can be integrated and used as video data to extract video features.

Also, the extraction unit 191 extracts audio features and chatting features from audio data including voices of broadcasters and shouts of on-site audiences.

More specifically, the extractor 191 may extract a feature vector of spectrum information using a speech recognition technique such as Mel Frequency Cepstral Coefficient (MFCC) for audio data. The Mel frequency cepstral coefficient is a technique for extracting features of sound, and is a technique for extracting features by dividing the input sound into certain sections and analyzing the spectrum of the corresponding sections rather than targeting the entire input sound.

The feature vector of the audio data may include information about the volume and tone of the commentator's voice, speaking speed, and the like. In addition, the extraction unit 191 performs undersampling on the initial audio data at a sample rate of 44,100 to 4,410 to extract audio features, obtains 100 MFCC results with 13 coefficients per second, and then obtains an average value of each coefficient value per second. . The obtained 13 coefficient values per second are used as audio features.

In addition, the extraction unit 191 may extract chat features through chat data, which are chat details directly input and transmitted by viewers of the video. In detail, the extractor 191 according to the present embodiment can extract features with performance similar to that of one-hot encoding while reducing a vector space by using character level label encoding while using the entire chatting data. Since such label encoding requires only a vector space of size 1 when representing a labeled value for each character, the extraction unit 191 according to the present embodiment uses up to 1,000 characters per second to extract 99% of chatting characters per second. 1,000 values per second can be extracted with the chat feature.

In addition, the extraction unit 191 may extract a feature vector from the chat data using a natural language processing (NLP) tool, and the feature vector of the chat data includes information on how many chats appeared within the same time period. In order to use it together, you can add a specific character to the end of every chat to indicate that the chat is over.

The chatting feature vector may include information on the number of chatting chats, a specific keyword, and the like, and chats appearing in a certain time interval may all be expressed as a single feature vector. Using a feature vector extracted from video, audio, and chat data rather than the data itself has the advantage of improving computation time and memory efficiency in the process of learning using an artificial neural network for this purpose.

Meanwhile, the connection unit 193 may connect the features by pre-processing the extracted video features, audio features, and chat features to a predetermined size through the LSTM model.

Also, the connection unit 193 may apply a smoothing technique to at least one feature among the features extracted by the extraction unit 191 before connecting the original features. Here, the smoothing technique may be a smoothing technique based on a moving average that calculates the average of the previous N pieces of data at each time point and uses it as a smoothing value.

In particular, the connection unit 193 according to the present embodiment may apply a smoothing technique to a video feature among extracted original features such as video features, audio features, and chatting features. At this time, the moving average can be calculated using a window array that stores the video features and the reciprocal of the window size, and the window size of the moving average is set to the sequence length used in the multimodal learning-based model. It is desirable to do

As in the present invention, in highlight detection, since events include causal relationships in continuous time frames, continuous video segments should be extracted based on the point in time at which a specific event occurs. In particular, when using individual features such as score change, a mechanism for effectively selecting continuous frames based on a specific event is required. To this end, a smoothing technique based on a moving average is applied in this embodiment.

In addition, the connection unit 193 must integrate and connect the characteristics of various data having different sizes in connecting the video characteristics to which the smoothing technique is applied, the audio characteristics, and the chatting characteristics. Therefore, the connection unit 193 according to the present embodiment does not directly use the raw features extracted by the extraction unit 191, but performs preprocessing of resizing the original features to a certain size through the LSTM model, and then connects each feature do. Specifically, as shown in FIG. 4 , preprocessing of resizing each raw feature to a size of 128 is performed through an LSTM model. And, as the sequence length, 7 frames can be used.

In addition, the connection unit 193 can maintain sequential information between adjacent frames for all data types (video, audio, and chat) using the LSTM model. In this way, by preprocessing through the LSTM model in the connection unit 193, the connected features having the same size are transferred to the multimodal learning-based model of the clip generation unit 195.

Meanwhile, the clip generator 195 may select a frame classified as a highlight frame through a multimodal learning-based model of features connected by the connection unit 193 as an image clip to be included in the highlight image.

To this end, as shown in FIG. 5, the multimodal learning-based model of the present invention is based on a 3-layer LSTM model, and may be composed of three hidden layers with each layer having a size of 128. In addition, the multimodal learning-based model may classify the output of the LSTM model as a highlight frame or a non-highlight frame by adding a single Fully-Connected (FC) layer to the LSTM model of the connection unit 193.

Here, highlight or non-highlight frames may be classified based on a result learned based on an answer image used in a learning process through multimodal deep learning. That is, highlight or non-highlight frames are classified based on the learning result of E-sports highlight images uploaded online or E-sports highlights images separately input by the manager in advance as the correct answer images. More specifically, for example, if a predetermined similarity is satisfied by comparing the vector value of the feature extracted through the above process with the vector value of the image classified as a highlight frame in the correct answer image input for learning, the corresponding frame is a highlight frame. , otherwise it will be classified as a non-highlight frame.

And, as shown in FIG. 5 through a multimodal learning-based model using multimodal learning, the clip generation unit 195 may use 23 frames as a sequence length obtained as an average of highlight frames for the data set, and individual Seven frames can be used when features are used as comparisons. Through this, the clip generator 195 may select a frame classified as a highlight frame among the classified frames as an image clip to be included in the highlight image.

Also, the highlight generator 197 may connect the video clips selected by the clip generator 195 to finally generate a highlight image.

2 is a flowchart for explaining a method for automatically generating a highlight image according to an embodiment of the present invention, FIG. 3 is a diagram for explaining how raw features extracted according to an embodiment of the present invention are connected, and FIG. A diagram for explaining how extracted primitive features are preprocessed to a predetermined size and then connected according to an embodiment of the present invention, and FIG. 5 is a diagram for explaining the multimodal learning-based model of the present invention.

A method for automatically generating a highlight image according to an embodiment of the present invention is provided to automatically extract a highlight from a video such as an E-sports game video or a relay video, and is performed by the automatic highlight extraction device 10. can

In the method for automatically generating a highlight image according to the present embodiment, first, the automatic highlight extraction device 10 receives a video such as a match video or a relay video (S110).

Then, video features are extracted from the input image (S120).

In extracting video features, the highlight automatic extraction apparatus 10 according to the present embodiment extracts video features by using data affecting a win rate change in a game. Taking LoL during the game as an example, we integrate all the source data, which is information that affects the win-lose probability, such as the gold obtained, the number of towers destroyed, the number of opponent characters defeated, and the number of Barons and Dragons in each team, all of which are integrated into the video data can be used to extract video features.

Thereafter, the automatic highlight extraction device 10 extracts audio features from the input image (S130).

In order to extract audio features, in this embodiment, instead of targeting the entire input sound, Mel Frequency Cepstral Coefficients (MFCC, Mel Frequency A feature vector for spectrum information may be extracted using a speech recognition technique such as Cepstral Coefficient. More specifically, in this embodiment, undersampling is performed on the initial audio data at a sample rate of 44,100 to 4,410, 100 MFCC (Mel-Frequiency Cepstral Coefficients) results are obtained with 13 coefficients per second, and then the average of each coefficient value per second is obtained. . The obtained 13 coefficient values per second are used as audio features.

And chatting features can be extracted from the input video (S140). While the entire chatting data is used to extract chatting features according to the present embodiment, character level label encoding is used instead of conventional one-hot vector encoding. In this embodiment, by using label encoding, a vector space can be greatly reduced and features can be extracted with performance similar to that of one-hot encoding. In particular, conventional one-hot encoding converts all characters of chatting data into ASCII codes to represent each character and requires a vector space for binary expression of each ASCII code. On the other hand, label encoding according to the present embodiment requires only a vector space of size 1 to indicate a value labeled for each character. Therefore, in order to maximize the range of chatting characters, up to 1,000 characters per second were used, which can include 99% of chatting characters per second, so 1,000 values per second are extracted as chat features.

And, the automatic highlight extraction method according to the present embodiment may further include applying a smoothing technique to at least one feature among the original features extracted in the feature extraction step (S150). there is.

Specifically, in highlight detection, continuous video segments must be extracted based on when a specific event occurs because events contain causal relationships in continuous time frames. Especially when using discrete features such as score changes, we need a mechanism to effectively select consecutive frames based on specific events.

Therefore, in this embodiment, a smoothing technique based on a moving average is applied. Here, the smoothing technique is a smoothing technique based on a moving average that calculates the average of the previous N data at each point and uses it as a smoothing value. In addition, moving average can calculate convolution using a window array that stores video features and a reciprocal value of window size, and it is recommended to set the window size of moving average to the length of the sequence used in the multimodal learning-based model described later. desirable.

In this embodiment, the smoothing technique was applied to the video features among the extracted raw features, and as a result of comparing before and after applying the smoothing technique to the video features, it was confirmed that the precision of the video features increased from 0.707 to 0.726.

From this, it can be seen that there is an obvious effect when the smoothing technique is applied to the video feature, so it is assumed that the smoothing technique is applied to the video feature among the raw features, but it is not limited thereto, and the audio feature among the raw features extracted as needed Alternatively, the smoothing technique may also be applied to chat features.

Thereafter, the extracted raw features may be preprocessed to a predetermined size through the LSTM model and connected (S160).

The extraction unit 191 according to the present embodiment captures 1 frame per second, each feature is extracted for each frame from 1 to t, and each frame is labeled as 0 (not highlighted) or 1 (highlighted) can be specified. In the multimodal learning model according to this embodiment, the connection of each feature is simply represented by a "+" sign. For example, when video feature V, audio feature A, and chat feature C are connected, it will be expressed as V+A+C. can

Here, highlight or non-highlight frames may be classified based on a result learned based on an answer image used in a learning process through multimodal deep learning. That is, highlight or non-highlight frames are classified based on the learning result of E-sports highlight images uploaded online or E-sports highlights images separately input by the manager in advance as the correct answer images. More specifically, for example, if a predetermined similarity is satisfied by comparing the vector value of the feature extracted through the above process with the vector value of the image classified as a highlight frame in the correct answer image input for learning, the corresponding frame is a highlight frame. , otherwise it will be classified as a non-highlight frame.

As a criterion for connecting each extracted feature, as shown in FIG. 3, the extracted video feature, audio feature, and chat feature can be directly connected without separate preprocessing as raw features. This simply concatenates the raw features for the different data types (video, audio, and chat) extracted through the method described above for each frame, as shown. 3 shows the process of connecting raw features. As shown in the figure, among the extracted raw features, V _SCORE , which is a feature for video data, is one element per frame, and A _MFCC , which is a feature for audio data, is per frame. MFCC 13 elements, C _LABEL , which is a characteristic for chatting data, can be composed of 1,000 elements per frame.

As such, raw feature linking has the advantage of directly using raw features in the learning process, but has limitations in multimodal learning, which requires integrating features of multiple data with different features. In order to solve this limitation, in the present invention, pre-processing of resizing the extracted raw features to a predetermined size through a long short-term memory (LSTM) model is performed, and then they are connected.

4 is for explaining the feature connection process through the LSTM model of the present invention. As shown, the feature connection through the LSTM model according to the present embodiment is 128 as shown in FIG. As a result of the preprocessing process of resizing to the same size, each feature can have a uniform effect in multimodal learning. Here, seven frames can be used as the sequence length, and among the features applied to the LSTM model, the video features are video features to which the smoothing technique is applied as described above.

And, in this embodiment, the LSTM model is used to maintain sequential information between adjacent frames for all data types (video, audio, and chat). Equation 1 below defines a feature extracted from the LSTM model for each raw feature, and D denotes a raw feature.

[Equation 1]

After being preprocessed to the same size through the LSTM model in step S160, the connected features are processed through the multimodal learning-based model.

Then, the apparatus 10 for automatically extracting highlights selects a frame classified as a highlight frame through a multimodal learning-based model as an image clip to be included in a highlight image (S170). Specifically, FIG. 5 is based on a 3-layer LSTM model as shown in a diagram of a multimodal learning-based model according to this embodiment, and is composed of three hidden layers each layer having a size of 128. This multimodal learning-based model classifies the output of the LSTM model into a highlight or non-highlight frame by adding a single fully-connected (FC) layer to the LSTM model for connecting the above-described extracted features. The multimodal learning-based model using multimodal learning with three features in this embodiment can use 23 frames for the sequence length obtained as the average of highlight frames for the data set, and use individual features as comparisons. In this case, 7 frames can be used.

Among the classified frames, frames classified as highlight frames are selected as video clips to be included in the highlight image (S170).

Thereafter, the automatic highlight extraction device 10 may generate a highlight image by connecting the selected image clips (S180).

Through this, the method of automatically generating an E-sports highlight video according to the present embodiment can extract highlights by utilizing all three data of video, audio, and chatting in E-sports relay broadcasting.

The method for automatically generating a highlight image of the present invention may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, as well as those known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

[Experimental example]

In order to objectively evaluate the performance of the results of automatic highlight extraction according to this embodiment, 99 LoL World Cup matches held in 2019 were used, and the total length of the match video was 200,759 seconds (ie, about 55 hours). It is an average of 33 minutes per video.

Among them, 70 randomly selected game images were used as a training dataset, 14 as a validation dataset, and 15 as a test dataset.

To train the win-loss probability model for comparison, in this experiment, 10,810 more game videos from 2016 to 2019 were collected using an open API provided by Riot Games, the developer of LoL.

As the correct video for specifying the highlight label, the highlight of the LoL video provided by the YouTube channel Onivia was used. To extract the time information of the video, the area where the time is displayed on the screen is pre-processed, and then the Tesseract API, an open source OCR engine. was applied. A highlight frame takes a total of 36,416 seconds, and each game contains an average of 6 minutes of highlights. To solve the data imbalance, 22,000 highlight frames and 22,000 non-highlight frames were randomly selected for training. And the model was trained with a batch size of 32, 100 epochs, a Stochastic Gradient Descent (SGD) optimizer and a learning rate of 10 ^-2 , and cross-entropy was used for the loss function. And, as defined in Equations 2 to 4 below, the F1 score, precision, and recall were used as evaluation indices, the evaluation indices were used for each video, and the average was calculated for each video.

[Equation 2]

=

=

[Equation 3]

=

=

[Equation 4]

-score =

-score=

Here, H _gt is a measured value, and H _pred means a result predicted by each model.

Table 1 below shows the performance of the model proposed in the previous study. Although it cannot be fairly compared with the results reported in previous studies because the data set is different, it can be confirmed that the overall trend is similar. The purpose of this experiment is to show the effect of multimodal learning based on features extracted from individual reference models.

Features Precision Recall F1-score V _WINLOSS 0.610 0.768 0.673 A _EMD 0.210 0.436 0.278 C _1HOT 0.318 0.325 0.321

V _WINLOSS is a method of detecting the point where the odds calculated through a conventional odds ratio prediction model changes rapidly and extracting highlights by extracting them as video features, and A _EMD is an EMD (EMD) to decompose the audio data included in the match video Empirical Mode Decomposition) is a conventional method of extracting highlights by extracting audio features by removing unimportant data, and C _1HOT sets highlights and non-highlights to 1 and 0 for each frame for chatting data. This is a conventional method of extracting highlights by extracting chatting features by labeling each character and obtaining one-hot encoding for the input text.

On the other hand, Table 2 below shows the results of models using individual features of the highlight extraction method of the present invention. For performance evaluation, the LSTM-based multimodal learning-based models are all connected by connecting the features extracted through the LSTM model. We note the performance of the proposed feature extraction using features consistently. Models using A _MFCC and C _LABEL improve by 90.68% and 7.79%, respectively, in F1 scores using the existing features in Table 1. In the case of V _SCORE , the performance decreased by 17.04% in the F1 score compared to V _WINLOSS , but it can be seen in Table 3 that the multimodal model using V _SCORE improves the performance of all previous studies. Among other features, especially V Compared to _WINLOSS , the accuracy of V _SCORE is very high, so it can be seen that the performance improves when combined with other features in multimodal learning.

Features Precision Recall F1-score V _SCORE 0.726 0.481 0.575 A _MFCC 0.536 0.554 0.530 C _LABEL 0.237 0.630 0.346

Table 3 below shows the results of the multimodal learning model of the present invention, and the model of the present invention in which the features extracted for each type (video, audio, and chat) are connected and used through LSTM is compared to the conventional Research and multimodal learning based on it were compared.

First, the performance of raw feature concatenation is 5.04% lower than the model using V _SCORE , which shows the best performance among individual models using the features of the present invention with an F1 score of 0.546. This means that simple concatenation of raw features is not effective for multimodal learning due to the non-uniform nature of different features.

Second, feature linking through LSTM greatly improves the performance of raw feature linking. As a result, feature linking through LSTM improves the F1 score of raw feature linking by 32.23% and improves it by 7.28% compared to the conventional odds prediction model V _WINLOSS . At this point, it can be seen that the multimodal learning-based model of the present invention in the E-sports highlight improves both the precision and recall of the win rate prediction model V _WINLOSS .

In addition, in order to verify the effect of the features extracted through the present invention, a multimodal learning model combining the features used in the conventional V _WINLOSS, A _EMD , and C _1HOT was constructed. To this end, V _WINLOSS, A _EMD , and C _1HOT were connected as one feature, and as described above, feature connection through LSTM was used for multimodal learning. As a result, it is shown that the model of the present invention improves the precision and reproducibility of the multimodal learning model by using existing features with significant performance differences.

Features Precision Recall F1-score V _SCORE + A _MFCC + C _LABEL 0.525 0.611 0.546 LSTM(V _SCORE )+LSTM(A _MFCC )+
LSTM(C _LABEL ) 0.637 0.838 0.722 V _WINLOSS 0.610 0.768 0.673 V _WINLOSS + A _EMD + C _1HOT 0.608 0.818 0.691

In addition, a demo video (https://url.kr/ey8bvz) showing the entire highlight detection process of LoL Twitch broadcasting was produced according to the highlight extraction method according to the embodiment of the present invention. To this end, we built a website that receives information about input images from users and creates video highlights. Highlight detection is performed automatically only when the user enters a simple Twitch video link, metadata (e.g. game name, video start time), CPU 3.50GHz, memory 64G, GPU RTX 2080 for a video about 30 minutes long It took less than 4 minutes in the Ti equipped environment.

Although various embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is commonly used in the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

10: automatic highlight video generation device 11: input unit
13: communication unit 15: memory unit
17: output unit 19: processor
191: extraction unit 193: connection unit
195: clip generation unit 197: highlight generation unit

Claims (5)

A method of automatically generating a highlight image performed by an apparatus for automatically generating a highlight image,
Step of inputting an E-sports relay image;
extracting a raw feature from an input video, wherein the raw feature includes a video feature, an audio feature, and a chat feature;
pre-processing the extracted raw features to a predetermined size and interconnecting the pre-processed video features, audio features, and chatting features;
classifying highlight frames from the concatenated video feature, audio feature, and chat feature;
selecting the classified highlight frames as video clips to be included in a highlight video;
Creating a highlight video by connecting the selected video clips;
In the step of extracting the raw features,
Extracting the video feature from video data included in the input video, wherein the video data is a combination of source data, which is information that affects win-loss probabilities,
Further comprising applying a smoothing technique to at least one of the extracted raw features before performing the concatenation step,
In the step of applying the smoothing technique,
A method for automatically generating a highlight image, wherein a smoothing technique based on a moving average is applied to the video features to extract continuous video segments based on a point in time when a specific event including a score change occurs. According to claim 1,
The step of interconnecting the preprocessed video feature, audio feature and chat feature,
The extracted video features, audio features, and chat features included in the extracted raw features are resized to the same size through a Long Short-Term Memory (LSTM) model and interconnected. Automatically generating a highlight image, characterized in that method. According to claim 2,
Classifying highlight frames from the connected video features, audio features, and chat features,
Automatically generating a highlight image, characterized in that by adding a single fully-connected (FC) layer to the LSTM model and classifying the connected video features, audio features, and chat features into highlight frames or non-highlight frames through the LSTM model. . delete an input unit that receives an E-sports relay video;
an extraction unit for extracting raw features from an input video, wherein the original features include video features, audio features, and chatting features;
a connection unit for pre-processing the extracted raw features to a predetermined size and interconnecting the pre-processed video features, audio features, and chatting features;
a clip creation unit that classifies highlight frames from the connected video, audio, and chatting features and selects the classified highlight frames as video clips to be included in a highlight video; and
A highlight generator for generating a highlight video by connecting selected video clips;
The extraction part,
The video feature is extracted from video data included in the input video, wherein the video data is a combination of source data, which is information that affects win-loss probabilities,
The connection part,
Applying a smoothing technique to at least one of the extracted original features before performing the interconnection;
The connection part,
An apparatus for automatically generating a highlight image that applies a smoothing technique based on a moving average to video features to extract continuous video segments based on a point in time at which a specific event including a score change occurs.

Images