KR20200092502A - Aparatus and method for generating a highlight video using chat data and audio data - Google Patents

Abstract

An objective of the present invention is to provide an apparatus for automatically generating a highlight video. The apparatus for generating a highlight video using chat data and audio data created while providing a video comprises: a separation unit to separate data extracted from a video into a first time section and a second time section longer than the first time section to generate first segment data and second segment data; an extraction unit to extract a first feature vector from the first segment data of the separation unit and extract a second feature vector from the second segment data of the separation unit; a learning unit to learn about the first feature vector and the second feature vector of the extraction unit through a recurrent neural network (RNN) and use learned results to generate a first result vector and a second result vector; a prediction unit to generate a score vector for a probability which can be predicted as a highlight from the first result vector and the second result vector of the learning unit; and a determination unit to use the score vector generated by the prediction unit to generate a final highlight.

Description

Apparatus and method for highlight video generation using chat data and audio data{APARATUS AND METHOD FOR GENERATING A HIGHLIGHT VIDEO USING CHAT DATA AND AUDIO DATA}

The present invention relates to an apparatus and method for generating a highlight image using chat data and audio data, and more specifically, chat data and audio for identifying and generating a final highlight image by predicting a probability of being a highlight for an image of a certain time period. The present invention relates to an apparatus and method for generating a highlight image using data.

Machine learning is a field of artificial intelligence that develops algorithms and technologies that enable computers to learn. A layer is required for the implementation of machine learning, and various layers such as perceptron, convolution neural network (CNN) and recurrent neural network (RNN) are used.

Perceptron (Perceptron) is a kind of artificial neural network, a classification algorithm that predicts based on a linear prediction function that combines a set of weights with a feature vector. It is an algorithm for supervised learning of a binary classifier, a function that can determine whether an input represented by a numeric vector belongs to a specific class.

Multi-layer perceptron (MLP) is a kind of feed-forward artificial consisting of at least three layers including input layer, output layer and hidden layer. It is a neural network. For training, a supervised learning technique called back-propagation is used. Unlike Perceptron, it is possible to distinguish data that cannot be separated linearly.

Recurrent neural network (RNN) is a type of artificial neural network, and it has a characteristic that the connection between units is a cyclic structure. This structure allows the state to be stored inside the neural network to model the time-varying dynamic features. Unlike forward delivery neural networks, cyclic neural networks (RNNs) can process sequence-like inputs using internal memory. Therefore, the circulating neural network (RNN) can process data having time-varying characteristics, such as handwriting recognition or speech recognition. One type of cyclic neural network (RNN) includes Long Short Term Memory (LSTM).

Long Short Term Memory (LSTM) is composed of a cell, an input gate, an output gate, and a forget gate. At this time, a cyclic neural network (RNN) composed of long- and short-term memory (LSTM) units is called a long- and short-term memory (LSTM) network. Long- and short-term memory (LSTM) networks are suitable for classifying, processing, and predicting predictions based on time-series data, as unknown durations can be delayed between time-series and important events.

Chat data and audio data generated when providing a streaming format image can be processed through a cyclic neural network (RNN) because they are time-series data having time-varying characteristics. At this time, in order to properly process data using a cyclic neural network (RNN), the characteristics of the input data must be clear.

Mel Frequency Cepstral Coefficient (MFCC) is commonly used to extract feature values from audio data, and natural language processing (NLP, is commonly used to extract feature values from text data such as chat history). Natural Language Processing) algorithm.

Mel Frequency Cepstral Coefficient (MFCC) is a technique that extracts the characteristics of sound, and does not target the entire input sound, but divides it into constant 　 intervals and analyzes 　 spectrum for that section to analyze the 　 characteristics. It is an extraction technique.

The Natural Language Processing (NLP) algorithm is a computational technique for automating human language analysis and expression. fastText is a natural language processing (NLP) algorithm that is a library for learning word insertion and text classification.

Meanwhile, the transmission and relay of sports game videos through the broadcasting platform are increasing. The person who transmits or relays the corresponding video collects the part of the viewer's interest and produces and provides a highlight video, and it is common for the viewer to view and select the highlight video to find the desired video.

Accordingly, it is a general trend to provide a highlight image, and the demand for production is increasing, but in the case of small-scale broadcasting such as general personal broadcasting, it is difficult to produce the highlight image due to time, cost, and technical constraints.

In order to overcome this, a method of extracting a highlight using the intensity of an audio signal included in an image (Japanese Patent Publication JP.H08079674.A) and a method of extracting a highlight using a signal energy level of an audio track (US registration) Patents US 6,973,256 B1) and the like have been disclosed and used. When broadcasting a sports game or an e-sports game, it is highly likely that the part of the broadcaster's voice or the audience's shout is growing. However, depending on the relaying method, the sound may not enter or may enter small, and you can watch quietly when you are nervous. Therefore, it is inaccurate to determine whether it is a highlight by only the intensity or energy density of the audio signal at the moment. Accordingly, there is a need to use more information to predict highlights.

1 is an implementation screen of an Internet personal broadcast of the present invention.

Referring to FIG. 1, in the case of personal broadcasting, it is provided through a display of a computer device. The screen includes a broadcast area such as an area 110 in which a relay video is provided, a chat area 120 in which viewers' opinions are disclosed, and a creator, personal streamer, streamer, and broadcasting Jockey, etc. Broadcaster) is divided into an area 130 in which an image is photographed, and a viewer watching a broadcast video can communicate with a broadcaster through the chat area 120.

Accordingly, audio data such as sound in a relay video and voice of a broadcaster may be generated according to the flow of video, and chat data may be generated according to the flow of video.

Korean Patent Publication No. 10-2005-0003071 (2005.01.10) Korean Patent Publication No. 10-2014-0056618 (2014.05.12) Korean Registered Patent No. 10-1867082 (2018.6.5) Japanese Patent Publication JP.H08079674.A (1996.03.22) U.S. registered patent US 6,973,256 B1 (2005.12.06)

The technical problem of the present invention has been devised in this regard, and the object of the present invention is to divide the chat data and audio data generated when transmitting a personal broadcast into regular time units and learn them in a recurrent neural network (RNN) before and after. It is to provide a device for automatically generating a highlight image in consideration of the context of.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In the highlight image generation apparatus using chat data and audio data generated during the provision of an image according to an embodiment of the present invention for achieving the above object, the data generated during the provision of the image in the first time interval and the first A separating unit separating the second time interval longer than the time interval to generate first segment data and second segment data; An extraction unit that extracts a first feature vector from the first segment data of the separation unit and extracts a second feature vector from the second segment data of the separation unit; A learning unit learning the first feature vector and the second feature vector of the extraction unit through a recurrent neural network (RNN), and generating a first result vector and a second result vector using the learned result; A prediction unit generating a score vector for a probability that can be predicted as a highlight from the first result vector and the second result vector of the learning unit; And a determination unit that generates a final highlight using the score vector generated by the prediction unit.

In one embodiment of the present invention, data generated during the provision of the video may be any one of chat data and audio data, and may include both chat data and audio data.

In one embodiment of the present invention, the extraction unit may use a natural language processing tool (NLP) to extract a feature vector of chat data.

In one embodiment of the present invention, the extraction unit may use Mel Frequency cepstral coefficients (MFCC) to extract a feature vector of audio data.

In an embodiment of the present invention, the learning unit includes a forward learning unit that generates a result vector using information of a previous time interval for the first feature vector and the second feature vector of the extraction unit; And a short-term backward learning unit generating a result vector by using information of a time interval after the first feature vector and the second feature vector of the extraction unit.

In an embodiment of the present invention, the Recurrent Neural Network (RNN) may use a Long Short Term Memory (LSTM).

In one embodiment of the present invention, the prediction unit may use a multi-layer perceptron (MLP) to generate a score vector.

In one embodiment of the present invention, the determining unit determines a highlight section in a predetermined order in a high probability order from the score vector generated by the prediction unit, and determines a final highlight with a predetermined length to generate a highlight image. can do.

In the method for generating a highlight image using chat data and audio data generated during the provision of an image according to an embodiment of the present invention for achieving the above object, the data extracted from the image is the first time period and the first time period Separating into a longer second time interval to generate first segment data and second segment data; Extracting a first feature vector from the first segment data and extracting a second feature vector from the second segment data; Learning the first feature vector and the second feature vector through a recurrent neural network (RNN), and generating a first result vector and a second result vector using the learned results; Generating a score vector for probability that can be predicted as a highlight from the first result vector and the second result vector; And generating a final highlight using the score vector.

According to one aspect of the present invention described above, the effect provided by the highlight image generating apparatus and method using the chat data and the audio data of the present invention, a broadcaster (Broadcaster) that performs a personal broadcast, and chat data through an artificial neural network A highlight image can be automatically generated by a device that has learned the characteristics of audio data. Therefore, a broadcaster can save a separate time and cost, and the viewer has an advantageous effect of being able to receive various highlight images.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art from the following description. .

1 is an implementation screen of an Internet personal broadcast of the present invention.
2 is a schematic block diagram of a highlight image generating apparatus using chat data and audio data according to an embodiment of the present invention.
FIG. 3 is a block diagram showing data flow in a cell in a general long-term memory (LSTM).
4 is a detailed block diagram showing the flow of data in a single data highlight prediction model (S-BiLSTM).
5 is a simplified block diagram showing the flow of data in a single data highlight prediction model (S-BiLSTM).
6 is a block diagram showing the flow of data in a multiple data highlight prediction model (M-BiLSTM).
7 is a block diagram showing the flow of data in a single data highlight prediction model for multiple time intervals.
8 is a block diagram illustrating the flow of data in a multiple data highlight prediction model for multiple time intervals.
9 is a block diagram illustrating a method of generating highlights in a multiple data highlight prediction model for multiple time intervals.

For a detailed description of the present invention, which will be described later, reference is made to the accompanying drawings that illustrate specific embodiments in which the present invention may be practiced. These embodiments are described in detail enough to enable those skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different, but need not be mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in connection with one embodiment. In addition, it should be understood that the location or placement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the invention. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of the present invention, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed. In the drawings, similar reference numerals refer to the same or similar functions across various aspects.

Hereinafter, a highlight image generating apparatus according to a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

2 is a schematic block diagram of a highlight image generating apparatus using chat data and audio data according to an embodiment of the present invention.

Referring to FIG. 2, the highlight image generation apparatus 200 using chat data and audio data according to an embodiment of the present invention includes a separation unit 210, an extraction unit 220, a learning unit 230, and a prediction unit ( 240) and the crystal unit 250.

The highlight image generation method 200 using chat data and audio data of the present invention may be installed and executed with a software (application) for performing a highlight image generation operation, a separation unit 210, an extraction unit 220, and learning The configuration of the unit 230, the prediction unit 240, and the determination unit 250 may be controlled by software executed in the device 200.

The chat data of the present invention is a conversation history that is directly input and transmitted by viewers of a video, and the audio data means a voice of a broadcaster and a shout of the audience.

The separating unit 210 generates first segment data and second segment data by separating data generated during the provision of an image into a first time interval and a second time interval for use in generating highlights.

At this time, data generated during the provision of the video means chat data and audio data, and segment data may be generated for any one of chat data and audio data, and segment data may be generated for both.

When only one of the chat data and audio data is used, a total of two pieces of segment data (ie, first segment data and second segment data) are generated. When both chat data and audio data are used, two segment data are generated for each, so a total of four segment data (i.e., first segment data for chat data and second segment data and first segment for audio data) Data and second segment data) are generated.

Meanwhile, the first time period means a time with a slight addition or subtraction in 1 second or second units, and the second time period is a time period longer than the first time period, and has a slight difference in 1 minute or minute units. Means Depending on the implementation method, it may be longer and shorter, but the second time interval is not longer than the first time interval.

The extraction unit 220 may extract a feature vector from the segmentation data generated by the separation unit 210.

The extracting unit 220 uses a first language vector (x _t ) and a second feature vector using natural language processing (NLP) tools such as FastText for the first segment data and the second segment data for the chat data. (x _t ) can be extracted. This is expressed as Equation (1).

[Equation 1]

In this case, in order to use information on how many chats appeared in the same time in the feature vector for chat data, a specific character may be added to the end of all chats to indicate that the chat is over. The chat feature vector may include information on the number of chats, specific keywords, and the like. All chats appearing in a certain time period can be expressed as a single feature vector.

The extracting unit 220 uses the speech recognition technology such as Mel Frequency Cepstral Coefficient (MFCC) for the first segment data and the second segment data for audio data to provide first features for spectrum information. The vector (x _t ) and the second feature vector (x _t ) can be extracted. This is expressed as Equation (2).

[Equation 2]

The feature vector for audio data may include information about the voice size, tone, and speech speed of the commentator.

On the other hand, Recurrent Neural Network (RNN, Recurrent Neural Network) the chat in the case of learning by using the data (chat _t) and the audio data (audio _t) computation time and memory efficiency when using the feature vector (x _t) derived from it, not itself This has the advantage of being improved.

The learning unit 230 learns the first feature vector and the second feature vector for the chat data (chat _t ) through a recurrent neural network (RNN), and uses the learned result and the first result vector. A second result vector can be generated. The first result vector and the second result vector can be generated for the audio data (audio _t ) through the same process.

At this time, the learning unit 230 is a forward learning unit that learns in the order of past viewpoint-present viewpoint-future viewpoint in order to utilize the features of a recurrent neural network (RNN) capable of extracting information from sequential data. It may include a reverse learning unit 233 for learning in the order of (231) and the future viewpoint-present-past viewpoint.

The forward learning unit 231 learns the information of the previous viewpoint and uses it to generate the current result vector, and the reverse learning unit 233 learns the information of the later viewpoint and uses it to generate the current short-term result vector.

The forward learning unit 231 and the reverse learning unit 233 are composed of a recurrent neural network (RNN), each of which may represent one unit, and may include all units.

At this time, the recurrent neural network (RNN) is a long-term memory (LSTM) to overcome a problem of a vanishing gradient problem in which learning ability is significantly deteriorated when the interval between individual units is far apart. ) Can be used.

Long Short Term Memory (LSTM) may be composed of a forget gate, an input gate, and an output gate. Each gate performs calculations in a given way to generate various intermediate vectors. The intermediate vector may include an oblivion vector 332 of a forget gate, an input vector 334 for an input gate, and an output vector 338 for an output gate. At this time, the long and short-term memory (LSTM) may be designed in a structure modified in various ways.

FIG. 3 is a block diagram showing data flow inside a cell of a Long Short Term Memory (LSTM).

Referring to FIG. 3, an operation process using the feature vector 310, the result vector 330 of the previous step, and the cell state vectors 350 of the previous step can be seen.

, 351,353, 359), 덧셈 연산(

, 355) 및 하이퍼볼릭 탄젠트(tanh, 335, 357) 연산으로 구성됨을 알 수 있다. Each gate is for selecting the flow of information, sigmoid layer (σ, 331, 333, 337), multiplication operation (

, 351,353, 359), addition operation (

, 355) and hyperbolic tangents (tanh, 335, 357).

At this time, the sigmoid layer (σ, 331, 333, 337) outputs a value between 0 and 1, it is possible to determine how much each component will affect.

The cell state vector 360 is a state at the current point in time of a cell, and is a refined result by adding or removing information to a cell through a gate while traversing the entire chain.

The long- and short-term memory 300 may protect and adjust the cell state vector 360 using the oblivion gate 331, the input gate 333, and the output gate 337.

The forgetting gate 331 may determine whether to keep or remove the feature vector 310 and the result vector 330 of the previous time interval.

The oblivion vector 332 is a value obtained by adding a bias vector (b _f ) as an offset to a value calculated in consideration of the weight (w _f ) of the corresponding layer for the feature vector 310 and the result vector 330 of the previous time interval. It can be obtained by passing through the sigmoid layer 331. This is expressed as Equation (3).

[Equation 3]

The input gate 333 looks at the feature vector 310 and the result vector 330 of the previous time interval, and determines what new information is stored in the cell state vector 360.

The input vector 334 adds the bias vector (b _i ) which is an offset to the calculated value in consideration of the weight (w _i ) of the input layer for the feature vector 310 and the result vector 330 of the previous time interval. It can be obtained by passing through the sigmoid layer 333. This is expressed as Equation (4).

[Equation 4]

The candidate vector 336 may include new candidate values to be added to the cell state vector 360. The candidate vector 336 is a hyperbolic tangent layer (tanh) obtained by adding a bias vector (b _c ) as an offset to a value calculated in consideration of the weight (w _c ) of the input layer with respect to the result vector 330 of the previous time interval. layer, 335). This is expressed by Equation (5).

[Equation 5]

The cell state vector 360 is a result value obtained by performing a vector product 351 of the previous cell state vector 350 and the oblivion vector 332 of the oblivion gate 331 and the input vector 334 and candidate vector of the input gate. The cell state vector 360 can be obtained by summing (355) the result values obtained by performing the vector product 353 on (336). This is expressed as Equation (6).

[Equation 6]

The output gate 337 looks at the feature vector 310 and the result vector 330 of the previous time interval, and determines which portion of the cell state to output.

The output vector 338 adds a bias vector (b _o ) which is an offset to a value calculated in consideration of the weight (w _o ) of the output layer for the feature vector 310 and the result vector 330 of the previous time interval. It can be obtained by passing through the sigmoid layer 337. This is expressed as Equation (7).

[Equation 7]

Finally, the result vectors 340 and 380 determine what to produce the final result as the result vector of the long and short-term memory 300. The result vectors 340 and 380 may be obtained by performing a vector product 359 on the output vector 338 and the result of passing the cell state vector 360 through the hyperbolic tangent layer 357. This can be expressed as Equation (8).

[Equation 8]

The process described above is based on the forward direction using the result vector 330 of the previous time interval and the cell state vector 350 of the previous time interval to generate the result vectors 340 and 380 of the current time. In the case of the reverse direction, there is a difference in using the result vector (h _t+1 ) of the subsequent time interval and the cell state vector (c _t+1 ) of the subsequent time interval, and the same process as described above can be performed. . In addition, the above process may be performed in the same manner in the forward learning unit 231 and the backward learning unit 233 of the learning unit 230.

The prediction unit 240 determines whether a time interval obtained from the result vector can be predicted as a highlight by using information obtained from the result vector obtained from the learning unit 230 to obtain a score vector S _{t as} a probability vector Can. At this time, if only one of the chat data or audio data is utilized for a single time interval (FIGS. 4 and 5), both of the chat data and audio data are used (FIG. 6), and the chat is performed for multiple time intervals. Since there is a difference between using only one of data or audio data (FIG. 7) and using both chat data and audio data (FIG. 8), the following description will be given separately.

4 is a detailed block diagram showing the flow of data in a single data highlight prediction model (S-BiLSTM).

Referring to FIG. 4, it can be seen that a process of generating a score vector S _t for a prediction probability to be a highlight for feature vectors X110, X130, and X150 of each constant time interval is shown. The probability that the image of the first time interval is predicted as a highlight is calculated by the feature vectors (X110, X ₁ ) of the first time interval through the first forward and short-term memory (LSTM, F110) and reverse long-term and short-term memory (LSTM, B110). It can be obtained from the result vector (h ₁ ). The result obtained through forward and short-term memory (F110, F130, and F150) and reverse long- and short-term memory (B110, B130, B150) for all input sections (X110, X ₁ , X130, X ₂ , X150, X _n ) Score vectors (S _t , S110) can be obtained from vectors (h ₁ , h ₂ , h _n ).

5 is a simplified block diagram showing the flow of data in a single data highlight prediction model (S-BiLSTM) for a single time interval.

)와 역방향 장단기 메모리(B210, LSTM_backward)를 거친 결과 벡터(

)를 고려하여 최종적으로 스코어 벡터(S210, s_t)를 생성하는 과정을 알 수 있다. 이 때, 스코어 벡터(s_t)는 순방향에 대한 결과 벡터(

)와 역방향에 대한 결과 벡터(

)에 대해 가중치(w_s)를 감안하여 시그모이드 레이어(σ)를 통과하여 얻을 수 있다. 모든 일정 시간 구간(즉, t=1, 2, ... , n)에 대해 동일한 과정을 수행할 수 있다. 이를 수식으로 표현하면 수학식 9와 같다.Referring to FIG. 5, a model showing a process briefly described in FIG. 4 is a result vector (for a feature vector (X _t , X210) that has undergone forward and short-term memory (F210, LSTM _forward ))

) And backward and short-term memory (B210, LSTM _backward ).

), the process of finally generating the score vectors S210 and s _t can be seen. At this time, the score vector (s _t ) is the result vector for the forward direction (

) And the result vector for the inverse (

) Can be obtained by passing the sigmoid layer (σ) in view of the weight (w _s ). The same process can be performed for all predetermined time periods (ie, t=1, 2, ..., n). This is expressed as Equation (9).

[Equation 9]

6 is a block diagram showing the flow of data in a multiple data highlight prediction model (M-BiLSTM) for a single time interval.

Referring to FIG. 6, unlike the single data highlight prediction model (S-BiLSTM) using either chat data or audio data, the multi-data highlight prediction model (M-BiLSTM) differs in that it utilizes both data. You can see that there is. Due to this difference, the multi-data highlight prediction model (M-BiLSTM) may further include a multi-layer perceptron (M310, MLP, Multi-Layer Perceptron).

)에 대해 순방향 장단기 메모리(F310, LSTM_forward)의 결과 벡터(

) 및 역방향 장단기 메모리(B310, LSTM_backward)의 결과 벡터(

)와 오디오 데이터의 특징 벡터(X330,

)에 대해 순방향 장단기 메모리(F330, LSTM_forward)의 결과 벡터(

) 및 역방향 장단기 메모리(B330, LSTM_backward) 의 결과 벡터(

)들을 모두 이어 붙여 다중 데이터 특징 벡터(

)를 생성할 수 있다. 생성된 다중 데이터 특징 벡터(

)를 다층 퍼셉트론(M310, MLP)에 입력하여 스코어 벡터(S310)를 생성할 수 있다. 이를 수식으로 표현하면 수학식 10과 같다.Chat data feature vector (X310,

Result vector of forward and short-term memory (F310, LSTM _forward )

) And the result vector () of reverse long and short-term memory (B310, LSTM _backward )

) And audio data feature vector (X330,

Result vector of forward and short-term memory (F330, LSTM _forward )

) And the result vector () of the reverse long- and short-term memory (B330, LSTM _backward )

Multiple data feature vectors (

). Generated multiple data feature vectors (

) Into a multi-layer perceptron (M310, MLP) to generate a score vector (S310). This is expressed as Equation (10).

[Equation 10]

7 is a block diagram showing the flow of data in a single data highlight prediction model for multiple time intervals.

Referring to FIG. 7, the flow of data can be seen in a state in which a single data highlight prediction model (S-BiLSTM) is applied to the first time period and the second time period.

A feature vector (X410) for any one of chat data and audio data in the first time period (M410) and a feature vector (X430) for any one of chat data and audio data in the second time period (M430) are generated. Can be.

The first result vector may be generated from information learned through the forward and short-term memory F410 and the backward and short-term memory B410 for the feature vector X410 for the first time period M410.

A second result vector may be generated from information learned through the forward and short-term memory F430 and the backward and short-term memory B430 for the feature vector X430 for the second time period M410.

Score vectors S410 and s _t may be generated by joining the first result vector and the second result vector and inputting them to the multilayer perceptron (MLP, M450).

8 is a block diagram illustrating the flow of data in a multiple data highlight prediction model for multiple time intervals.

Referring to FIG. 8, it is possible to know the flow of data in a state in which multiple data highlight prediction models (M-BiLSTM) are applied to the first time period and the second time period, respectively.

) 및 오디오 데이터에 대해 특징 벡터(X530,

)를 생성할 수 있다. 상기 채팅 데이터에 대한 특징 벡터(X510,

)에 대해 순방향 장단기 메모리(F510, LSTM_forward) 및 역방향 장단기 메모리(B510, LSTM_0backward)를 통해 학습한 정보로부터 채팅 데이터에 대한 제 1 결과 벡터(

,

)를 생성할 수 있다. 오디오 데이터에 대한 특징 벡터(X530,

)에 대해서도 동일한 과정을 통해 오디오 데이터에 대한 제 1 결과 벡터(

,

)를 생성할 수 있다.Feature vector (X510, for chat data) for the first time interval M510

) And feature data for audio data (X530,

). Feature vector for the chat data (X510,

), the first result vector for chat data from information learned through the forward and short-term memory (F510, LSTM _forward ) and the reverse long- and short-term memory (B510, LSTM _0backward )

,

). Feature vector for audio data (X530,

) For the first result vector for the audio data through the same process (

,

).

) 및 상기 오디오 데이터에 대한 특징 벡터(X570,

)로부터 채팅 데이터에 대한 제 2 결과 벡터(

,

) 및 오디오 데이터에 대한 제 2 결과 벡터(

,

) 를 생성할 수 있다.Through the same method, the feature vector (X550, for chat data) for the second time interval (M530)

) And the feature vector for the audio data (X570,

), the second result vector for chat data (

,

) And second result vector for audio data (

,

).

,

,

,

)와 제 2 시간 구간에 대한 제 2 결과 벡터(

,

,

,

)를 이어 붙여 다층 퍼셉트론(MLP, M550)에 입력하여 스코어 벡터(S510, s_t)을 생성할 수 있다.First result vector for the first time interval (

,

,

,

) And the second result vector for the second time interval (

,

,

,

) To input to the multi-layer perceptron (MLP, M550) to generate score vectors (S510, s _t ).

The determining unit 250 may select the highlight sections from the score vectors S110, S210, S310, S410, and S510 of the prediction unit 240 in the order of highest highlight prediction probability.

The ratio selected as the highlight can be selected by presetting the ratio of the entire image, or an image exceeding a specific score is selected as the highlight. At this time, the ratio selected as the highlight can be set to a specific ratio such as 10% or 15% based on the entire video, and can be set to a specific time period such as 10 minutes or 15 minutes, and when the probability score exceeds a certain value It can be set to always be selected as a highlight.

When it is selected as the highlight, it can be determined as the final highlight by the length of a predetermined image. At this time, the predetermined length of the video can be added in a predetermined time, such as 2 seconds or 10 seconds, respectively, before and after the section that is identified as a highlight, and the time added before and after can be individually specified. have.

9 is a block diagram illustrating a method of generating highlights in a multiple data highlight prediction model for multiple time intervals.

The method for estimating a data classification rule according to the present invention may be performed by the highlight image generation device 200 according to the present invention described above. To this end, the highlight image generation apparatus 200 according to the present invention may be pre-installed with an application (software) for performing each step constituting the data classification rule estimation method described later. For example, a platform for a method for estimating data classification rules according to the present invention may be pre-installed in the form of software on a user's computer, and a user may execute a software installed on a computer to execute the data classification rule estimation method according to the present invention Various services provided can be provided.

The highlight image generating apparatus 200 may separate chat data and audio data into a first time period and a second time period (1110, 1130), and extract feature vectors for each time period (1210, 1230). ). The highlight image generating apparatus 200 may generate short-term result vectors and long-term result vectors using the cyclic neural network for the extracted first time intervals (1310, 1330). The highlight image generating apparatus 200 may input a short-term result vector and a long-term result vector into a multi-layer perceptron (MLP) to generate a score vector for probability that can be predicted as a highlight as a result value (1510). The highlight image generation apparatus 200 may determine the highlight sections in the order of the highest probability in the score vector and classify them as final highlights in a predetermined length (1710).

Since the detailed description of each of the above steps has been described with reference to FIGS. 4 to 8, repeated description will be omitted.

Such a technique for providing a method for generating a highlight image may be implemented as an application or implemented in the form of program instructions that can be executed through various computer components to be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, or the like alone or in combination.

The program instructions recorded on the computer-readable recording medium are specially designed and configured for the present invention, and may be known and available to those skilled in the computer software field.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like.

Examples of program instructions include not only machine language codes produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although described above with reference to embodiments, those skilled in the art understand that various modifications and changes can be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. Will be able to.

200: Highlight video generation device using chat data and audio data
210: separation
220: extraction unit
230: learning department
240: prediction unit
250: decision section

Claims (9)

In the highlight video generation device using the chat data and audio data generated during the provision of the video,
A separating unit separating data generated during the provision of the image into a first time period and a second time period longer than the first time period to generate first segment data and second segment data;
An extraction unit that extracts a first feature vector from the first segment data of the separation unit and extracts a second feature vector from the second segment data of the separation unit;
A learning unit learning the first feature vector and the second feature vector of the extraction unit through a recurrent neural network (RNN), and generating a first result vector and a second result vector using the learned result;
A prediction unit generating a score vector for a probability that can be predicted as a highlight from the first result vector and the second result vector of the learning unit; And
A determination unit for generating a final highlight using the score vector generated by the prediction unit; Highlight image generating apparatus using chat data and audio data.
According to claim 1, Data generated during the provision of the image,
A highlight image generating apparatus using chat data and audio data, which may be any one of chat data and audio data, and may include both chat data and audio data.
According to claim 1, wherein the extraction unit,
Highlight image generation device using chat data and audio data, which uses natural language processing (NLP) to extract feature vectors of chat data.
According to claim 1, wherein the extraction unit,
Highlight image generation device using chat data and audio data, using Mel Frequency cepstral coefficients (MFCC) to extract feature vectors of audio data.
According to claim 1, The learning unit,
A forward learning unit generating a result vector using information of a previous time interval for the first feature vector and the second feature vector of the extraction unit; And
And a short-term backward learning unit generating a result vector by using information of a time interval after the first feature vector and the second feature vector of the extraction unit, wherein the highlight image generation apparatus includes chat data and audio data.
According to claim 1, wherein the cyclic neural network (RNN, Recurrent Neural Network),
Highlight video generation device using chat data and audio data, using Long Short Term Memory (LSTM).
The method of claim 1, wherein the prediction unit,
A highlight image generating apparatus using chat data and audio data, which uses a multi-layer perceptron (MLP) to generate a score vector.
The method of claim 1, wherein the determining unit,
Highlight images using chat data and audio data to generate highlight images by determining the highlight sections in the order of high probability from the score vector generated by the prediction unit in a predetermined order and determining the final highlights with a predetermined length. Generating device.
In the method of generating a highlight image using chat data and audio data generated during the provision of the video,
Separating the data extracted from the image into a first time period and a second time period longer than the first time period to generate first segment data and second segment data;
Extracting a first feature vector from the first segment data and extracting a second feature vector from the second segment data;
Learning the first feature vector and the second feature vector through a recurrent neural network (RNN), and generating a first result vector and a second result vector using the learned results;
Generating a score vector for probability that can be predicted as a highlight from the first result vector and the second result vector; And
Generating a final highlight using the score vector; Method for generating a highlight image using chat data and audio data.

Images