KR102198480B1 - Video summarization apparatus and method via recursive graph modeling - Google Patents

Abstract

The present invention provides a video summary generation device and a method thereof. The video summary generation device regards a plurality of frames of a video image as nodes of a relationship graph, and infers semantic similarity between multiple frames using a graph convolutional neural network having a recursive reasoning structure, so as to extract an accurate key frame considering the global and long-term interrelationships between multiple frames, thereby generating a video summary in various video platforms with a relatively small number of parameters and high accuracy.

Description

Video summary generation device and method through recursive graph modeling {VIDEO SUMMARIZATION APPARATUS AND METHOD VIA RECURSIVE GRAPH MODELING}

The present invention relates to an apparatus and method for generating a video summary, and to an apparatus and method for generating a video summary through recursive graph modeling.

Due to recent changes in video platforms such as online streaming services, it has become difficult for users to access desired video data. In addition, as the length of the video is gradually increasing, it becomes more difficult for users to obtain useful information from the video. Accordingly, research is being conducted to enable users to efficiently search for video data. As part of this research, video summarization, which selects useful scenes from videos and allows concise description of the video contents, is drawing attention as a tool that can greatly improve user convenience.

However, the video summary has a difficulty in selecting useful frames in consideration of interrelationships between all frames of a video. Accordingly, various researches on video summarization methods for extracting key frames using an artificial neural network are being conducted.

Previously, video summarization algorithms using artificial neural networks were mainly implemented based on convolutional neural networks (CNN) or recurrent neural networks (RNN). However, artificial neural networks such as CNN and RNN have a problem of locality in which they cannot express globally with a small receptive field, so they are not suitable for reflecting the long-term dependency of video. There is a limit to not.

Korean Patent Publication No. 10-2019-0099027 (published on August 23, 2019)

An object of the present invention is to provide a video summary generation apparatus and method capable of accurately extracting a key frame for video summary by considering a plurality of frames of a video as nodes of a relationship graph and applying them to a graph convolutional neural network.

Another object of the present invention is to provide an apparatus and method for generating a video summary capable of extracting a key frame in consideration of global and long-term interrelationships between multiple frames by recursively inferring semantic similarity between multiple frames.

A video summary generation apparatus according to an embodiment of the present invention for achieving the above object receives an input video composed of a plurality of frames, and generates a plurality of feature maps each extracted from a plurality of frames according to a previously learned pattern estimation method. An initial graph generator configured to generate an initial feature graph based on a node vector, and to generate an initial adjacency matrix by calculating a similarity between the node vectors; The initial feature graph or the node of the feature graph from a pattern between the initial feature graph and the initial adjacency matrix, or a feature graph that is repeatedly recursively applied and an adjacency matrix using a plurality of graph convolution modules in which a pattern estimation method is previously learned A graph correlation unit for extracting a correction graph by correcting a pattern therebetween; Obtaining an adjacency matrix and a feature graph for extracting the next correction graph based on the correction similarity and the previous adjacency matrix indicating the similarity between the two weighted correction graphs by applying the correction graph and weighting with different weighting functions previously specified. A recursive graph acquisition unit; And using a separate graph convolution network in which the pattern estimation method is pre-learned, the final correction graph obtained by iterative recursion at a predetermined number of times and the final adjacency matrix, according to the semantic similarity between each node of the final correction graph. And a key frame extractor for selecting a plurality of key frames by estimating a probability that each frame becomes a key frame.

The initial graph generator comprises: a feature map acquisition unit for generating the plurality of feature maps by extracting features for each of the plurality of frames of the input video according to a previously learned pattern estimation method; An initial graph acquisition unit for obtaining the initial feature graph by considering each of the plurality of feature maps as a node of a graph consisting of a plurality of nodes and a plurality of edges connecting the plurality of nodes to each other; And an initial adjacency matrix obtaining unit for obtaining an initial adjacency matrix by calculating a degree of similarity between a plurality of nodes of the initial feature graph.

The initial graph acquisition unit may obtain the initial feature graph by converting each of the plurality of feature maps into a one-dimensional node vector.

The initial adjacency matrix acquisition unit calculates the degree of similarity between each node vector (x ₁ , …, x _T ) of the initial feature graph (X ⁰ ).

(Where a _ij denotes the element of the initial adjacency matrix (A ⁰ ), and x _i and x _j are the i-th node vectors of the multiple node vectors (x ₁ , …, x _T ) of the initial feature graph (X ⁰ ), respectively And j-th node vectors, T denotes a transpose matrix, and ∥ ∥ ₂ denotes an L2 norm function.), the initial adjacency matrix A ⁰ may be obtained.

The initial graph generation unit may further include a frame selection unit that extracts frames from a plurality of frames of the input video in units of a predetermined time interval and transmits the extracted frames to the feature map acquisition unit.

In the graph correlation unit, the plurality of graph convolution modules are sequentially connected in series, and the graph correlation module at the initial stage among the plurality of graph correlation modules includes an initial feature graph or a recursive feature graph and an initial adjacency matrix or a recursive adjacency matrix. It is applied to estimate the pattern to extract the first intermediate feature graph, and the remaining graph correlation modules receive the intermediate feature graph extracted from the previous stage and the adjacent matrix applied to the graph correlation module of the initial stage, respectively, and estimate the pattern by estimating the intermediate feature. The graph or the correction graph can be extracted.

The recursive graph acquisition unit receives the previously acquired correction graphs, respectively, and weights different predetermined weight functions to the correction graph so that the correction graphs are projected on different linear embedding spaces to obtain two projection correction graphs. part; An adjacent correction value acquisition unit for obtaining the correction similarity by calculating a similarity between the two projection correction graphs; And an adjacency matrix obtaining unit that obtains the adjacency matrix by adding the previous adjacency matrix and the obtained correction similarity.

The adjacent correction value obtaining unit calculates the correction similarity dA ^k

(Where W _θ Z ^k , W _φ Z ^k represents the projection correction graph weighted with the weight function (W _θ , W _φ ) on the correction graph (Z ^k ), T represents the transpose matrix, and ∥ ∥ ₂ is the L2 norm It can be calculated according to (L2 norm) function.)

The key frame extractor applies the final correction graph and the final adjacency matrix to a separate graph convolution network in which a pattern estimation method is learned in advance, and the input video is applied according to a semantic similarity pattern between nodes of the final correction graph. A correlation estimating unit for extracting a key frame probability map indicating a probability that each of the plurality of frames is a key frame; And a key frame selection unit for selecting a plurality of key frames by analyzing a probability that each of the plurality of frames of the input video will be key framed from the key frame probability map.

A video summary generation method according to another embodiment of the present invention for achieving the above object is to receive an input video composed of a plurality of frames, and a plurality of feature maps each extracted from a plurality of frames according to a previously learned pattern estimation method. Generating an initial feature graph by regard to the node vector, and generating an initial adjacency matrix by calculating a similarity between the node vectors; The initial feature graph or the node of the feature graph from a pattern between the initial feature graph and the initial adjacency matrix, or a feature graph that is repeatedly recursively applied and an adjacency matrix using a plurality of graph convolution modules in which a pattern estimation method is previously learned Extracting a correction graph by correcting the pattern therebetween; Obtaining an adjacency matrix and a feature graph for extracting the next correction graph based on the correction similarity and the previous adjacency matrix indicating the similarity between the two weighted correction graphs by applying the correction graph and weighting with different weighting functions previously specified. The step of doing; And using a separate graph convolution network in which the pattern estimation method is pre-learned, the final correction graph obtained by iterative recursion at a predetermined number of times and the final adjacency matrix, according to the semantic similarity between each node of the final correction graph. And selecting a plurality of key frames by estimating a probability that each frame will be a key frame.

Accordingly, the apparatus and method for generating a video summary according to an embodiment of the present invention considers multiple frames of a video image as nodes of a relationship graph, and uses a graph convolutional neural network having a recursive inference structure to achieve semantic similarity between multiple frames. By inferring, it is possible to extract an accurate key frame in consideration of global and long-term interrelationships between multiple frames. Therefore, it is possible to easily provide video summaries to users in various video platforms with relatively small number of parameters and high accuracy.

1 shows a schematic structure of an apparatus for generating a video summary according to an embodiment of the present invention.
FIG. 2 shows a detailed configuration of an initial graph generator of the video summary generation apparatus of FIG. 1.
3 shows detailed configurations of a graph correlation unit and a recursive graph acquisition unit of the video summary generation apparatus of FIG. 1.
4 shows an example of a key frame extracted according to the number of recursion repetitions.
5 shows a video summary generation method according to an embodiment of the present invention.
6 shows a result of comparing a video summary generation method according to the present embodiment with a video summary generated according to a conventional video summary generation method.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the implementation of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless specifically stated to the contrary. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean units that process at least one function or operation, which is hardware, software, or hardware. And software.

1 shows a schematic structure of an apparatus for generating a video summary according to an embodiment of the present invention.

Referring to FIG. 1, the video summary generation apparatus according to the present embodiment may include an initial graph generation unit 100, a graph correlation unit 200, a recursive graph acquisition unit 300, and a key frame extraction unit 400. have.

The initial graph generator 100 receives the input video I composed of a plurality of frames and extracts features for each frame to generate a plurality of feature maps (X = {x ₁ , …, x _T }), Regarding each of the generated feature maps (x ₁ , …, x _T ) as node vectors, an initial feature graph (X ⁰ ) is generated, and the affinity between each node vector of the initial feature graph (X ⁰ ) ) To generate an initial adjacency matrix (A ⁰ ). Here, the initial adjacency matrix A ⁰ may be obtained with a size of T × T.

The graph correlation unit 200 includes an initial feature graph (X ⁰ ) and an initial adjacency matrix (A ⁰ ) generated by the initial graph generator 100 or a previously acquired feature graph (X ^k-1 ) and a recursive graph acquisition unit ( 300) adjacent to the matrix (a ^k-1) is received, the initial characteristic graph (X ⁰⁾ and the initial adjacency matrix in accordance with the pattern estimation method using a weighting matrix (W) obtained in advance by learning a obtained in (a ⁰⁾ Alternatively, a correction graph (Z) by correcting the pattern between the nodes of the initial feature graph (X ⁰ ) or the feature graph (X ^k-1 ) from the pattern between the feature graph (X ^k-1 ) and the adjacency matrix (A ^k-1 ). ^k ) is extracted.

The recursive graph acquisition unit 300 receives the correction graph Z ^k extracted from the graph correlation unit 200, weights it with a predetermined weight function (W _θ , W _φ ), and projects it to different linear embedding spaces, using the corrected similarity (dA ^k) from the previous adjacent matrix (a ^k-1) between the two calibration graph projected on other linear embedding space (W _θ Z ^k, W _φ Z ^k) the adjacency matrix (a ^k) Acquire. Then, the obtained correction graph (Z ^k ) is transferred to the graph correlation unit 200 along with the adjacency matrix (A ^k ) as a feature graph (X ^k = Z ^k ), so that the graph correlation unit 200 sends the next correction graph ( Make it possible to extract Z ^k+1 ).

The key frame extraction unit 400 applies the final correction graph Z ^K and the final adjacency matrix A ^K obtained by recursively recursively at a predetermined number of times by the graph correlation unit 200 and the recursive graph acquisition unit 300 receiving, the likelihood of the key frame (F _key) a number of frames, each of the patterned final calibration graph, the applied according to the estimation method (Z ^K) semantic estimate the degree of similarity patterns input video (I) between each node in the learning in advance The indicated key frame probability map (Y) is extracted, and a key frame key frame (F _key ) is selected according to the extracted key frame probability map (Y).

That is, the video summary generation apparatus according to the present embodiment regards the feature map extracted from each of the plurality of frames of the input video I as a node, and indicates the similarity between the initial feature graph X ⁰ composed of a plurality of nodes and the nodes. generating an initial adjacency matrix (a ^0), and a compensation based on the similarity of adjacent features of each node in the matrix from the initial characteristic graph (X ⁰⁾ and the initial adjacency matrix (a ⁰⁾ correction characteristic graph (Z ^k) After obtaining the final corrected feature graph (Z ^K ) and the final adjacency matrix (A ^K ) obtained by recursively recursively extracting the and adjacency matrix (A ^k ) at a known number of times, the final corrected feature graph (Z ^K ) The probability that each of the plurality of frames of the input video I is a key frame F _key is estimated from the final adjacency matrix A ^K and the semantic similarity pattern.

Accordingly, a key frame having a high semantic similarity can be accurately selected based on the final correction feature graph Z ^K and the final adjacency matrix A ^{K in} which the correlation between the plurality of frames of the input video I is corrected.

Hereinafter, each configuration of the video summary generating apparatus of FIG. 1 will be described in detail.

FIG. 2 shows a detailed configuration of an initial graph generator of the video summary generation apparatus of FIG. 1.

Referring to FIG. 2, the initial graph generation unit 100 may include a frame selection unit 110, a feature map acquisition unit 120, an initial graph acquisition unit 130, and an initial adjacency matrix acquisition unit 140. .

The frame selection unit 110 receives the input video I including a plurality of frames, selects the frames in a predetermined time interval unit (for example, 1 second), and transmits them to the feature map acquisition unit 120. In general, a video image is composed of 30 to 60 frames per second, and most of the frames are very similar to each other within a short time period. Therefore, it is inefficient to select a key frame for generating a video summary by analyzing the semantic similarity between all the frames of the input video I.

Accordingly, the frame selection unit 110 selects frames from the applied input video I in units of predetermined time intervals to improve the efficiency of the video summary generation apparatus and transmits them to the feature map acquisition unit 120.

However, this is to increase efficiency in consideration of the fact that in general, video summary does not require precise selection of key frames in frame units from video images. When key frames must be extracted very accurately, frames may not be selected. have. That is, the frame selection unit 110 may be omitted.

The feature map acquisition unit 120 is implemented as an artificial neural network in which the pattern estimation method is learned in advance, receives the input video (I) or the input video from which the frame is selected by the frame selection unit 110, and uses the previously learned pattern estimation method. Accordingly, features of each frame are extracted and a plurality of feature maps (X = {x ₁ , …, x _T }) are obtained.

The feature map acquisition unit 120 may be implemented as a variety of artificial neural networks, and since various artificial neural networks that extract feature maps (x ₁ , …, x _T ) from each frame image are already publicly disclosed, among these disclosed artificial neural networks It can also be implemented using one. The feature map acquisition unit 120 may be implemented with, for example, a pre-learned convolutional neural network (CNN).

The initial graph acquisition unit 130 obtains an initial feature graph X ⁰ by considering each of the plurality of feature maps X as node vectors. The initial graph acquisition unit 130 converts a plurality of feature maps (X) acquired in a two-dimensional or three-dimensional size determined by the feature map acquisition unit 120 into one-dimensional node vectors, for example, (X ⁰ ) can be obtained.

The initial graph acquisition unit 130 regards each of the plurality of feature maps (X) as a node of a graph composed of a plurality of nodes and a plurality of edges connecting the plurality of nodes to each other, so that a graph convolutional network : GCN) is obtained as an initial characteristic graph (X ⁰ ) suitable for the graph correlation unit 200. In this case, the initial graph acquisition unit 130 may be configured to obtain an initial feature graph X ⁰ by converting each of the plurality of feature maps X into a node vector having a predetermined format. As an example, when each of the plurality of feature maps X is acquired as a two-dimensional vector having a size of T × D, the initial graph acquisition unit 130 may convert it into a form of a one-dimensional vector having a length of T × D.

The initial adjacency matrix acquisition unit 140 includes each node vector (x ₁ , …, x _T ) of the initial feature graph X ⁰ acquired by the initial graph acquisition unit 130, that is, between a plurality of feature maps X. The similarity is calculated according to Equation 1 to obtain an initial adjacency matrix A ⁰ .

Where a _ij denotes an element of the initial adjacency matrix (A ⁰ ), and x _i and x _j are the i-th node vectors of the multiple node vectors (x ₁ , …, x _T ) of the initial feature graph (X ⁰ ), respectively. It represents the j-th node vector, T represents the transpose matrix, and ∥ ∥ ₂ represents the L2 norm function.

That is, the initial adjacency matrix A ⁰ can be viewed as a weight matrix of edges connecting node vectors (x ₁ , …, x _T ) of the initial feature graph X ⁰ according to similarity.

The initial graph generator 100 converts the initial feature graph X ⁰ and the initial adjacency matrix A ⁰ generated by the initial adjacency matrix acquisition unit 130 and the initial adjacency matrix acquisition unit 140 into a graph correlation unit 200. To pass.

3 shows detailed configurations of a graph correlation unit and a recursive graph acquisition unit of the video summary generation apparatus of FIG. 1.

Referring to FIG. 3, the graph correlation unit 200 includes a plurality of graph correlation modules 210 to 230 sequentially connected in series. Each of the plurality of graph correlation modules 210 to 230 is implemented as a graph convolution network (GCN) in which the pattern estimation method is learned in advance, and is initially adjacent to the initial feature graph X ⁰ applied from the initial graph generator 100. The correction graph Z ¹ is extracted by correcting the pattern estimated from the matrix A ⁰ . In addition, the graph correlation unit 200 transfers the extracted correction graph Z ¹ to the recursive graph acquisition unit 300, and the adjacent matrix A ¹ and the feature graph X ¹ transmitted from the recursive graph acquisition unit 300 ) Is applied and the process of extracting the next correction graph (Z ² ) is repeated.

That is, the graph correlation unit 200 is a pattern estimation method of a graph convolution network (GCN) that can be expressed by Equation 2 by receiving the already acquired feature graph (X ^k-1 ) and the adjacency matrix (A ^k-1 ). According to the correction graph (Z ^k ) is extracted.

At this time, the graph correlation module 210 of the initial stage among the plurality of graph correlation modules 210 to 230 receives the feature graph X ^k-1 and the adjacency matrix A ^k-1 to estimate the pattern and After the feature graph (X ^' ) is extracted, the remaining graph correlation modules 220 and 230 receive the intermediate feature graphs (X', X") and adjacency matrix (A ^k-1 ) extracted from the previous stage, respectively, and By estimating the first intermediate feature graph (X") or the correction graph (Z ^k ) is extracted.

Here, W represents the weight of the learned graph convolution network (GCN), and σ represents a nonlinear activation function such as ReLU (Rectified Linear Unit).

In the present embodiment, for example, the graph correlation unit 200 is shown to include three graph correlation modules 210 to 230 sequentially connected in series, but the graph correlation modules 210 to 230 included in the graph correlation unit 200 The number of) can be variously adjusted by experimentally analyzing the performance of extracting the correction graph Z ^k . That is, the performance of extracting the correction graph (Z ^k ) can be adjusted by the number of graph correlation modules (210 to 230), and here, as an experimental result, a case including three graph correlation modules (210 to 230) is illustrated. .

Meanwhile, the recursive graph acquisition unit 300 may include first and second projection units 310 and 320, an adjacent correction value acquisition unit 330, and an adjacent matrix acquisition unit 340.

Each of the first and second projection units 310 and 320 receives the correction graph Z ^k obtained from the graph correlation unit 200, so that the correction graph Z ^k is projected in different linear embedding spaces. A projection correction graph (W _θ Z ^k , W _φ Z ^k ) is obtained by weighting a predetermined weighting function (W _θ , W _φ ) to the applied correction graph Z ^k .

The adjacent correction value acquisition unit 330 receives the projection correction graphs (W _θ Z ^k , W _φ Z ^k ) obtained from the first and second projection units 310 and 320 and calculates the correction similarity (dA ^k ). Acquired according to 3.

Here, too, T represents the transpose matrix, and ∥ ∥ ₂ represents the L2 norm function.

The adjacency matrix acquisition unit 340 adds the previous adjacency matrix (A ^k-1 ) and the correction similarity (dA ^k ) obtained from the adjacent correction value acquisition unit 330, so that the graph correlation unit 200 performs the next correction graph (Z). ^An adjacency matrix (A ^k ) for estimating ^k+1 ) is obtained.

That is, the adjacency matrix acquisition unit 340 accumulates and adds the correction similarity (dA ^k ) obtained later from the initial adjacency matrix (A ⁰ ) to obtain an adjacency matrix (A ^k ) for estimating the next correction graph (Z ^k+1 ). do. Therefore, if the recursive graph acquisition unit 300 recursively recurses a predetermined number of K times to obtain an adjacency matrix (A ^k ), the final adjacency matrix (A ^K ) that is finally obtained can be calculated as in Equation 4 have.

At this time, the adjacency matrix acquisition unit 340 may transfer the already acquired correction graph Z ^k to the graph correlation unit 200 together with the adjacency matrix A ^k as the next feature graph X ^k .

Referring back to FIG. 1, the key frame extracting unit 400 may include a correlation estimating unit 410 and a key frame selecting unit 420.

The correlation estimating unit 410 is implemented as a graph convolutional network (GCN) in which the pattern estimation method is learned in advance, and is applied from the final correction graph Z ^K and the final correction graph Z from the pattern of the final adjacency matrix A ^K. A key frame probability map consisting of a probability indicating the probability that each of the plurality of frames of the input video I represented by each node vector is a key frame (F _key ) by estimating the correlation between the plurality of node vectors of ^K ) ( Y) is extracted according to the semantic similarity.

The correlation estimating unit 410 may extract the key frame probability map Y according to the pattern estimation method of the graph convolution network GCN represented by Equation (5).

Here, W _c represents the weight of the learned graph convolution network (GCN), and σ represents the activation function.

The key frame probability map (Y) may be obtained with a size of T × 2 to represent the number of frames (T) and the probability that the frame corresponding to each node vector is a key frame and a probability that it is not a key frame, respectively.

In addition, the key frame selection unit 420 selects a frame with a high probability of being a key frame and a low probability of not being a key frame among a plurality of frames of the input video I from the key frame probability map (Y), Select with (F _key ).

The key frame selection unit 420 selects, for example, a predetermined number of key frames (F _key ), or a second threshold value in which the probability of a key frame is equal to or greater than a predetermined first threshold value, or a probability of not being a key frame The following frames can be selected as key frames (F _key ).

Meanwhile, the graph correlation unit 200 and the key frame extraction unit 400 implemented including a graph convolution network (GCN) need to be learned in advance. Accordingly, the video summary generation apparatus according to the present embodiment may further include a learning unit (not shown) for learning the graph correlation unit 200 and the key frame extraction unit 400.

The learning unit may train the graph correlation unit 200 and the key frame extraction unit 400 to minimize losses calculated according to a plurality of predetermined loss functions.

In this embodiment, the learning unit may perform learning in one of a supervised learning method and an unsupervised learning method.

First of all, when learning is performed in a supervised learning method, the learning unit acquires a video in which a truth key frame probability map (Y ^* ) is obtained in advance based on the importance score normalized and averaged by a plurality of users as training data, and input video ( I), by comparing the truth key frame probability map (Y ^* ) of the training data with the key frame probability map (Y) obtained from the video summary generator to calculate the supervised learning loss (L _sup ) and _{backpropagating it} to learn. Perform.

In the supervised learning method, the learning unit calculates the classification loss (L _c ), the sparsity loss (L _s ), the restoration loss (L _r ), and the diversity loss (L _d ), respectively.

Classification loss (L _c ) is a binary cross-entropy loss between the key frame probability map (Y) and the truth key frame probability map (Y ^* ) generated by the video summary generation device according to Equation 6 Can be calculated.

Here, y ^* _t is the truth label of the t-th frame, w _t is the weight of the t-th frame, and can be calculated as w _t = median_freq/freq(s). freq(s) is a value obtained by dividing the number of key frames by the total number of frames, and median_freq represents the median value of the occurrence frequency of key frames.

The sparsity loss (L _s ) is a loss derived from the assumption that the number of key frames should be sparse in multiple frames of the input image (I), and L1 is applied to each element of the adjacency matrix (a _ij ∈ A ^K ) as shown in Equation 7 It can be obtained by applying a norm function.

)를 생성하고, 생성된 추가 보정 특징 그래프(

)와 초기 특징 그래프(X⁰) 사이의 평균 제곱 오차(MSE)를 수학식 8과 같이 계산하여 획득할 수 있다.Restoration loss (L _r ) is a loss added according to the assumption that the key frame must exist in various ways, and the key frame probability map (Y) is separately included in one graph convolution module and is added to the previously learned graph convolution network. Applied to the initial feature graph (X ⁰ ) and the additional correction feature graph (

), and the generated additional correction feature graph (

) And the initial feature graph X ⁰ may be obtained by calculating the mean squared error (MSE) as shown in Equation 8.

)에서 키 프레임으로 선택되는 노드들 사이의 반발 규격화(repelling regularizer)를 적용하여 수학식 9에 따라 획득할 수 있다.Finally, the diversity loss (L _d ) is the additional correction feature graph (

) Can be obtained according to Equation 9 by applying a repelling regularizer between nodes selected as key frames.

And the learning unit is the total loss (L _sup ) in supervised learning from the classification loss (L _c ), the sparse loss (L _s ), the restoration loss (L _r ), and the diversity loss (L _d ) calculated according to Equations 6 to 9 Is calculated as in Equation 10 and backpropagated.

Here, λ, α, and β are weights for controlling the importance of each loss.

On the other hand, when the learning unit performs unsupervised learning, since the training data does not exist, excluding the classification loss (L _c ) in Equation 10, the sparsity loss (L _s ), the restoration loss (L _r ), and the diversity loss (L _{From d} ), the total loss (L _unsup ) during unsupervised learning is calculated as in Equation 11 and _{backpropagated} .

4 shows an example of a key frame extracted according to the number of recursion repetitions.

In FIG. 4, (a) shows a case of performing one recursive iteration (K = 1), (b) shows a case of performing three recursive iterations (K = 3), and (c) shows a case of performing 5 times recursion It shows the case where iteration (K=5) was performed. And in each of (a) to (c), the upper left frame represents the key frame (F _key ) selected from the input video (I), and the rest are selected in the order of high key frame probability excluding the key frame (F _key ). Represents a frame. And s displayed with each frame represents a priority score that has been pre-annotated, normalized and averaged by a number of users.

As shown in FIG. 4, it can be seen that the connection between semantically similar frames, that is, key frames having high semantic similarity, is gradually strengthened as the number of recursion repetitions increases. However, as the number of recursive iterations increases, the amount of computation and the computation time increase, but the semantic similarity between key frames is not greatly enhanced. Therefore, for efficiency, the number of recursive iterations may be predetermined through experimentation.

5 shows a video summary generation method according to an embodiment of the present invention.

When the video summary generation method of FIG. 5 is described with reference to FIGS. 1 to 3, first, a feature map (X) extracted from each frame is applied to the input video (I) of a plurality of frames for which a video summary is to be generated is a node vector. An initial feature graph (X ⁰ ) is generated as regarded as, and an initial adjacency matrix (A ⁰ ) is generated by calculating the similarity between node vectors (S10).

In the step (S10) of generating the initial feature graph (X ⁰ ) and the initial adjacency matrix (A ⁰ ), an input video I composed of a plurality of frames is first obtained (S11). In addition, a plurality of feature maps X is generated by extracting features for each of a plurality of frames of the input video I according to a previously learned pattern estimation method (S12). When a number of feature maps (X) are created, each of the plurality of feature maps (X) is regarded as a node vector representing a node of a graph consisting of a number of nodes and a number of edges connecting a number of nodes to each other, and an initial feature graph (X ⁰ ) is obtained (S13).

And when an initial feature graph (X ⁰ ) including a plurality of nodes corresponding to each of the plurality of feature maps (X) is obtained, the similarity between the plurality of nodes is calculated according to Equation 1, and the initial adjacency matrix (A ⁰ ) Is obtained (S14).

When the initial feature graph (X ⁰ ) and the initial adjacency matrix (A ⁰ ) are acquired, the initial feature graph (X ⁰ ) and the initial adjacency matrix (A ⁰ ) are applied to a plurality of graph convolution networks in which the pattern estimation method is learned in advance. Thus, the correction graph Z ^k is extracted by correcting the pattern between the plurality of nodes of the initial feature graph X ⁰ (S20 ).

Then, it is determined whether the number of recursion repetitions (k) is equal to or greater than the predetermined number (K) (k ≥ K) (S30).

If the number of recursion repetitions (k) is less than the predetermined number (K), the correction graph (Z ^k ) is projected into different linear embedding spaces, and the projected two correction graphs (W _θ Z ^k , W _φ Z ^k ) An adjacency matrix (A ^k ) for obtaining the next correction graph (Z ^k+1 ) is obtained by calculating the correction similarity (dA ^k ) between (S40).

Specifically, the recursive repetition number (k) a group weighting is less than a specified number of times (K), different groups specified weighting function (W _θ, W _φ) to the calibration graph (Z ^k), the correction graph (Z ^k) Projected onto different linear embedding spaces (S41). Then, the similarity between the two projection correction graphs W _θ Z ^k and W _φ Z ^k is calculated to obtain a corrected similarity dA ^k (S42). When the correction similarity (dA ^k ) is obtained, the next adjacency matrix (A ^k ) is obtained by adding the previous adjacency matrix (A ^k-1 ) and the correction similarity (dA ^k ), and the correction graph (Z ^k ) is ^converted to the next feature graph. It is applied as (X ^k ) (S43). The obtained next adjacency matrix (A ^k ) and the next feature graph (X ^k ) are recursively applied to a plurality of graph convolution networks in which the pattern estimation method is learned in advance, so that the next adjacency matrix (A ^k ) and the next feature graph (X ^k ) ^The next correction graph (Z ^k+1 ) is extracted from ^k ) (S20).

However, if the number of recursion repetitions (k) is more than the predetermined number (K), the final correction graph (Z) according to the pattern estimation method learned in advance from the finally obtained final adjacency matrix (A ^K ) and the final correction graph (Z ^k ) ^A key frame (F _key ) is selected by estimating the semantic similarity between each node of ^K ) (S50).

To select a key frame (F _key ), first, a final adjacency matrix (A ^K ) and a final correction graph (Z ^k ) that are finally obtained are first applied to a separate graph convolution network in which the pattern estimation method is learned in advance. A key frame probability map Y representing the probability that each of the plurality of frames of the input video I is a key frame F _key is extracted according to the semantic similarity pattern between each node of the correction graph Z ^K (S51). . Then, from the extracted key frame probability map (Y), a plurality of key frames (F _key ) are selected by analyzing the probability that each of the plurality of frames of the input video (I) is key framed (S52). The plurality of key frames F _key selected here are frames having high semantic similarity among the plurality of frames of the input video I, and can be viewed as a video summary.

6 shows a result of comparing a video summary generation method according to the present embodiment with a video summary generated according to a conventional video summary generation method.

In FIG. 6, (a) and (c) show the result of generating a video summary according to the SUM-FCN technique previously disclosed, and (b) and (d) are based on the SumGraph technique, which is a video summary technique according to the present embodiment. It shows the result of generating the video summary accordingly. In addition, the bar graphs displayed above the frames selected in FIGS. 6A to 6D represent the importance values labeled on the training data. The red area in the bar graph represents the key frame selected according to each selected technique.

As shown in FIG. 6, it can be seen that the video summary generation method according to the present embodiment is more visually diverse than the conventional technique, and selects a frame corresponding to the peak of the labeled importance value as a key frame. This indicates that the video summary generation method according to the present embodiment can accurately estimate the semantic relationship between a plurality of frames in order to generate an optimal meaningful summary in the input video.

The method according to the present invention can be implemented as a computer program stored in a medium for execution on a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (Read Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those of ordinary skill in the art will appreciate that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: initial graph generation unit 110: frame selection unit
120: feature map acquisition unit 130: initial graph acquisition unit
140: initial adjacency matrix acquisition unit 200: graph correlation unit
210 ~ 230: graph correlation module 300: recursive graph acquisition unit
310: first projection unit 320: second projection unit
330: adjacent correction value acquisition unit 340: adjacent matrix acquisition unit
400: key frame extraction unit 410: correlation estimation unit
420: key frame selection unit

Claims (18)

An input video consisting of a plurality of frames is received, and an initial feature graph is generated by considering a plurality of feature maps each extracted from a plurality of frames as a node vector according to a previously learned pattern estimation method, and the similarity between the node vectors is calculated. An initial graph generator that calculates and generates an initial adjacency matrix;
The initial feature graph or the node of the feature graph from a pattern between the initial feature graph and the initial adjacency matrix, or a feature graph that is repeatedly recursively applied and an adjacency matrix using a plurality of graph convolution modules in which a pattern estimation method is previously learned A graph correlation unit for extracting a correction graph by correcting a pattern therebetween;
Obtaining an adjacency matrix and a feature graph for extracting the next correction graph based on the correction similarity and the previous adjacency matrix indicating the similarity between the two weighted correction graphs by applying the correction graph and weighting with different weighting functions previously specified. A recursive graph acquisition unit; And
Multiple frames according to the semantic similarity between each node of the final correction graph from the final correction graph obtained by recursion repeatedly at a predetermined number of times using a separate graph convolution network in which the pattern estimation method is learned in advance A video summary generation apparatus including a key frame extracting unit for selecting a plurality of key frames by estimating a probability that each will be a key frame. The method of claim 1, wherein the initial graph generator
A feature map acquisition unit for generating the plurality of feature maps by extracting features for each of the plurality of frames of the input video according to a previously learned pattern estimation method;
An initial graph acquisition unit for acquiring the initial feature graph by considering each of the plurality of feature maps as a node of a graph composed of a plurality of nodes and a plurality of edges connecting the plurality of nodes to each other; And
And an initial adjacency matrix acquisition unit for obtaining an initial adjacency matrix by calculating a similarity between a plurality of nodes of the initial feature graph. The method of claim 2, wherein the initial graph acquisition unit
A video summary generation apparatus for obtaining the initial feature graph by converting each of the plurality of feature maps into a one-dimensional node vector. The method of claim 3, wherein the initial adjacency matrix obtaining unit
The degree of similarity between each node vector (x ₁ , …, x _T ) of the initial feature graph (X ⁰ ) is equation

(Where a _ij denotes the element of the initial adjacency matrix (A ⁰ ), and x _i and x _j are the i-th node vectors of the multiple node vectors (x ₁ , …, x _T ) of the initial feature graph (X ⁰ ), respectively And the j-th node vector, T represents the transpose matrix, and ∥ ∥ ₂ represents the L2 norm function.)
A video summary generation apparatus that calculates according to and obtains the initial adjacency matrix A ⁰ . The method of claim 2, wherein the initial graph generator
A video summary generation apparatus further comprising a frame selection unit for extracting frames from the plurality of frames of the input video in units of a predetermined time interval and transmitting the extracted frames to the feature map acquisition unit. The method of claim 2, wherein the graph correlation unit
The plurality of graph convolution modules are sequentially connected in series, and the graph correlation module at the initial stage among the plurality of graph correlation modules receives an initial feature graph or a recursive feature graph and an initial adjacency matrix or a recursive adjacency matrix to generate a pattern. The first intermediate feature graph is extracted by estimating, and the remaining graph correlation modules receive the intermediate feature graph extracted from the previous stage and the adjacency matrix applied to the graph correlation module of the initial stage, respectively, and estimate the pattern, and the intermediate feature graph or the correction. Video summary generation device to extract graphs. The method of claim 6, wherein the recursive graph acquisition unit
A projection unit that receives the previously obtained correction graphs, and obtains two projection correction graphs by weighting different predetermined weight functions to the correction graph so that the correction graph is projected on different linear embedding spaces;
An adjacent correction value acquisition unit for obtaining the correction similarity by calculating a similarity between the two projection correction graphs; And
An apparatus for generating video summarization, comprising: an adjacency matrix obtaining unit for obtaining the adjacency matrix by adding a previous adjacency matrix and the obtained correction similarity. The method of claim 7, wherein the adjacent correction value acquisition unit
The correction similarity (dA ^k ) is ^calculated by the equation

(Where W _θ Z ^k , W _φ Z ^k represents the projection correction graph weighted with the weight function (W _θ , W _φ ) on the correction graph (Z ^k ), T represents the transpose matrix, and ∥ ∥ ₂ is the L2 norm (L2 norm) function.)
Video summary generation device that calculates according to. The method of claim 7, wherein the key frame extraction unit
By applying the final correction graph and the final adjacency matrix to a separate graph convolution network in which the pattern estimation method is learned in advance, each of the plurality of frames of the input video is generated according to the semantic similarity pattern between each node of the final correction graph. A correlation estimation unit for extracting a key frame probability map indicating a probability of a key frame; And
And a key frame selection unit for selecting a plurality of key frames by analyzing a probability that each of the plurality of frames of the input video will be key framed from the key frame probability map. An input video consisting of a plurality of frames is received, and an initial feature graph is generated by considering a plurality of feature maps each extracted from a plurality of frames as a node vector according to a previously learned pattern estimation method, and the similarity between the node vectors is calculated. Calculating and generating an initial adjacency matrix;
The initial feature graph or the node of the feature graph from a pattern between the initial feature graph and the initial adjacency matrix, or a feature graph that is repeatedly recursively applied and an adjacency matrix using a plurality of graph convolution modules in which a pattern estimation method is previously learned Extracting a correction graph by correcting the pattern therebetween;
Obtaining an adjacency matrix and a feature graph for extracting the next correction graph based on the correction similarity and the previous adjacency matrix indicating the similarity between the two weighted correction graphs by applying the correction graph and weighting with different weighting functions previously specified. Step to do; And
Multiple frames according to the semantic similarity between each node of the final correction graph from the final correction graph obtained by recursion repeatedly at a predetermined number of times using a separate graph convolution network in which the pattern estimation method is learned in advance And selecting a plurality of key frames by estimating a probability that each will be a key frame. The method of claim 10, wherein generating the initial adjacency matrix
Generating the plurality of feature maps by extracting features for each of the plurality of frames of the input video according to a previously learned pattern estimation method;
Obtaining the initial feature graph by considering each of the plurality of feature maps as nodes of a graph consisting of a plurality of nodes and a plurality of edges connecting the plurality of nodes to each other; And
And obtaining an initial adjacency matrix by calculating a similarity between a plurality of nodes of the initial feature graph. The method of claim 11, wherein obtaining the initial feature graph comprises:
A video summary generation method for obtaining the initial feature graph by converting each of the plurality of feature maps into a one-dimensional node vector. The method of claim 11, wherein obtaining the initial adjacency matrix
The degree of similarity between each node vector (x ₁ , …, x _T ) of the initial feature graph (X ⁰ ) is equation

(Where a _ij denotes the element of the initial adjacency matrix (A ⁰ ), and x _i and x _j are the i-th node vectors of the multiple node vectors (x ₁ , …, x _T ) of the initial feature graph (X ⁰ ), respectively And the j-th node vector, T represents the transpose matrix, and ∥ ∥ ₂ represents the L2 norm function.)
A video summary generation method for obtaining the initial adjacency matrix (A ⁰ ) by calculating according to. The method of claim 11, wherein generating the initial adjacency matrix
Before generating the plurality of feature maps, the method further comprising extracting frames from the plurality of frames of the input video in units of a predetermined time period. The method of claim 11, wherein extracting the correction graph
The plurality of graph convolution modules are sequentially connected in series, and the graph correlation module at the initial stage among the plurality of graph correlation modules receives an initial feature graph or a recursive feature graph and an initial adjacency matrix or a recursive adjacency matrix to generate a pattern. The first intermediate feature graph is extracted by estimating, and the remaining graph correlation modules receive the intermediate feature graph extracted from the previous stage and the adjacency matrix applied to the graph correlation module of the initial stage, respectively, and estimate the pattern, and the intermediate feature graph or the correction. How to generate video summaries to extract graphs. The method of claim 15, wherein obtaining the adjacency matrix and the feature graph comprises:
Obtaining two projection correction graphs by respectively receiving the previously obtained correction graphs and weighting different predetermined weight functions to the correction graph so that the correction graphs are projected onto different linear embedding spaces;
Obtaining the correction similarity by calculating a similarity between the two projection correction graphs; And
And obtaining the adjacency matrix by adding a previous adjacency matrix and the acquired correction similarity. The method of claim 16, wherein obtaining the correction similarity
The correction similarity (dA ^k ) is ^calculated by the equation

(Where W _θ Z ^k , W _φ Z ^k represents the projection correction graph weighted with the weight function (W _θ , W _φ ) on the correction graph (Z ^k ), T represents the transpose matrix, and ∥ ∥ ₂ is the L2 norm (L2 norm) function.)
Video summary generation method calculated according to. The method of claim 16, wherein selecting the plurality of key frames comprises:
By applying the final correction graph and the final adjacency matrix to a separate graph convolution network in which the pattern estimation method is learned in advance, each of the plurality of frames of the input video is generated according to the semantic similarity pattern between each node of the final correction graph. Extracting a key frame probability map indicating a probability of a key frame; And
And selecting a plurality of key frames by analyzing a probability that each of the plurality of frames of the input video will be key framed from the key frame probability map.

Images