KR102505592B1 - Video captioning method based on multi-representation switching, recording medium and system for performing the same - Google Patents

Abstract

A multi-expression switching-based video captioning method performed by the multi-expression switching-based video captioning system of the present invention includes the steps of extracting object features in units of frames from an input moving picture; extracting motion features based on the extracted object features; extracting grammatical features based on given word sequences; calculating a weighted sum by calculating weights for the object features and motion features; and generating a description of the video based on the grammar feature, the weighted object feature, and the weighted motion feature. Accordingly, it is possible to automatically generate a plurality of captions including various expressions for the video.

Description

Video captioning method based on multi-representation switching, recording medium and system for performing the same

The present invention relates to a video captioning method based on multi-expression switching, a recording medium and a system for performing the same, and more particularly, to a video captioning method based on multi-expression switching capable of automatically generating natural language sentences for moving pictures composed of continuous images. It relates to a captioning method, a recording medium and a system for performing the captioning method.

Recently, as deep learning research developed from neural networks is being actively conducted, there are many deep learning researches that outperform existing machine learning methods in the areas of computer vision and natural language processing.

Video description is one of the research fields in which image processing and natural language processing research are integrated into one, which is summarizing moving images. In the past, computer vision research such as image classification and natural language processing research such as automatic summarization have been studied separately.

Recently, however, as the need for multi-modal data analysis such as automatic video caption generation and video surveillance increases, technologies in which computer vision and natural language processing technologies are integrated are being actively researched.

However, prior art cases focus on methods for extracting expressions including object and motion characteristics of a video, and these methods have a problem in that it is difficult to model whether a given word is a word expressing video information or a grammatically necessary word. there is.

In addition, in the case of the prior art, since it solves a problem using a method with high computational complexity, it is a bigger problem in the case where a lot of resources are required for learning and evaluation and the reaction time in the application step must be fast.

Republic of Korea Patent Publication No. 10-2015-0057591

The present invention has been made to solve the above problems, and an object of the present invention is a multi-expression switching base that can explain various characteristics of frames in a video and automatically generate a plurality of captions including various expressions. It is to provide a video captioning method, a recording medium and a system for performing the same.

To achieve the above object, a multi-expression switching-based video captioning method performed by a multi-expression switching-based video captioning system according to an embodiment of the present invention includes extracting object features from an input video in frame units; extracting motion features based on the extracted object features; extracting grammatical features based on given word sequences; calculating a weighted sum by calculating weights for the object features and motion features; and generating a description of the video based on the grammar feature, the weighted object feature, and the weighted motion feature.

Here, extracting the object features may include mean pooling the extracted object features.

In the step of extracting the grammatical feature, the grammatical feature may be extracted based on a combination of an object feature vector extracted along with the given word sequence and an average vector for all frames of motion feature vectors.

In addition, the step of calculating the weighted sum calculates the weight for each frame using an attention mechanism, and the extracted object features and motion features required when generating words for the description of the video use the attention mechanism Through this, the word can be selected whenever it is generated.

And the step of generating the description comprises: generating a word generation probability expression when only grammar features are used, when only object features are used, and when only operation features are used; calculating a final word probability distribution based on the word generation probability expression; and generating a word having the highest probability value in the calculated word probability distribution as a final word.

In addition, the generating of the explanation may include generating the word generation probability expression, calculating the final word probability distribution, and generating the word having the highest probability value as the final word to repeat the steps of generating the video description. You can create a sequence of words for

Also, in the step of generating the description, the generation of the word sequence may be generated through a beam search algorithm.

Meanwhile, it may be a computer-readable recording medium on which a computer program for performing a multi-expression switching-based video captioning method according to an embodiment of the present invention for achieving the above object is recorded.

Meanwhile, a multi-expression switching-based video captioning system according to an embodiment of the present invention for achieving the above object includes an input unit for receiving a video input; and a processor that generates video captions using a deep learning model when the video is input, receives a video and description pair as training data, and trains the deep learning model.

Here, the deep learning model includes a 2D convolutional neural network (CNN) module that extracts object features in frame units from an input video and a first recurrent neural network (RNN) module that extracts motion features based on the extracted object features. Encoder to do; and a second recurrent neural network module for extracting grammatical features using a given word sequence, an attention module for calculating a weighted sum by calculating weights per frame for the object features and motion features, and the extracted grammatical features and weighted object features. and a decoder including a switch module for generating a description of the moving image based on the weighted summed operating characteristics.

The 2D convolutional neural network (CNN) module may perform a mean pooling operation on the extracted object features.

In addition, the second recurrent neural network module may extract a grammar feature based on a combination of an object feature vector extracted with the given word sequence and an average vector for all frames of motion feature vectors.

In addition, the attention module calculates a weight for each frame using an attention mechanism, and the extracted object features and motion features required when generating words for the description of the video are obtained through the attention mechanism. can be selected whenever is created.

In addition, the switch module generates word generation probability expressions when only grammar features are used, when only object features are used, and when only operation features are used, and a final word probability distribution is calculated based on the word generation probability expressions, A word sequence for video description may be generated by repeating a process of generating a word having the highest probability value as a final word in the calculated word probability distribution.

And, the switch module may generate the word sequence through a beam search algorithm.

According to one aspect of the present invention described above, by providing a multi-expression switching-based video captioning method, a recording medium and a system for performing the same, various characteristics of image frames constituting a moving picture are described, and various expressions are included. Not only can multiple captions be automatically generated, but also captions can be automatically generated without preprocessing or a separate loss function based on cup computer vision and natural language processing techniques.

1 is a diagram for explaining data used for video captioning;
2 is a diagram for explaining the structure of a deep learning model for multi-expression switching-based video captioning according to an embodiment of the present invention;
3 is a diagram for explaining feature extraction of an object in a deep learning model according to this embodiment;
4 is a flowchart for explaining a video captioning method based on multi-expression switching according to this embodiment;
5 is a diagram for explaining a video captioning result according to the present embodiment;
6 is a graph for explaining a video captioning result according to the present embodiment;
7 is a graph for explaining a video captioning result according to the present embodiment;
8 to 10 are diagrams for explaining video captioning results according to the present embodiment, and
11 is a block diagram illustrating a video captioning system based on multi-expression switching according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in another embodiment without departing from the spirit and scope of the invention in connection with one embodiment. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a diagram for explaining data used for video captioning, and FIG. 2 is a diagram for explaining the structure of a deep learning model for video captioning based on multi-expression switching according to an embodiment of the present invention. FIG. is a diagram for explaining feature extraction of an object in a deep learning model according to this embodiment, and FIG. 4 is a flowchart for explaining a video captioning method based on multi-expression switching according to this embodiment.

A video given as data used in a deep learning model for video captioning according to an embodiment of the present invention. This video is composed of a sequence of images, and a given description is composed of a sequence of words. The relationship between these images and subtitles is as shown in FIG. 1 .

1 is a video of a baby dancing on a chair, and 'baby', 'dancing', and 'chair' in the related captions are words representing objects and their motions in the video, and 'a' and 'is'. is a grammatically necessary word and 'on' is a word expressing the relationship between objects. That is, although information about objects and motions in a video has a direct relationship with words, it can be seen that grammatically necessary words or words representing relationships between objects are words that can be expressed only after learning a natural language.

Therefore, in order to learn the relationship between the video and the description without distinguishing between articles, nouns, and verbs using a separate natural language processing method, a deep learning model with a structure that can separately learn two words inside as in this embodiment To this end, the video captioning method according to an embodiment of the present invention is based on multi-expression switching.

The deep learning model according to the present embodiment preferably has an encoder-decoder framework to generate explanations by learning video and description pairs, as shown in the conceptual diagram shown in FIG. 2, and the encoder 100 It includes a 2D convolutional neural network (CNN) module 110 and a first recurrent neural network (RNN) module 130 to extract features of a video including objects and motions, and a decoder ( 200) extracts word features and selects the extracted object features, motion features, and word features to generate a description of the image; ) module 230.

Regarding the arrows shown in FIG. 2 , a thin arrow means that one vector is transmitted, and a thick arrow means that several vectors are transmitted. Specifically, for example, a thick arrow between the 2D convolutional neural network module 110 and the first recurrent neural network module 120 indicates that the vector information of all frames output from the 2D convolutional neural network module 110 is the first recurrent neural network module. (120) It means that it is passed to the module.

의 각 프레임들

의 벡터를 입력받을 수 있다. In the multi-expression switching-based video captioning method according to the present embodiment, object features for each frame in a video are first extracted in the 2D convolutional neural network module 110 (S110). The step of extracting object features is a step of extracting object information in units of frames from video. Prior to the step of extracting object features (S110), an image in units of frames from a video is input to the 2D convolutional neural network module 110 of the encoder 100. Specifically, a 2D convolutional neural network module provided to extract object features 110 videos

each frame of

A vector of can be input.

은 224x224x3의 RGB 프레임으로 입력되며, 각 프레임은 색상 정보인 RGB 정보만 가지고 있다. 각 프레임 사이의 동작 정보 등은 따로 사용하지 않는 것이 바람직하며, 각 프레임을 입력받기 위해서 별도의 프레임 생성모듈(미도시)을 통해 프레임이 각각 독립적으로 처리될 수도 있을 것이다. each frame

is input as a 224x224x3 RGB frame, and each frame has only color information, RGB information. It is preferable not to separately use operation information between each frame, and each frame may be independently processed through a separate frame generation module (not shown) to receive input of each frame.

The 2D convolutional neural network module 110 according to this embodiment uses ResNet50V2, which shows good performance with small parameters, and outputs each layer of stacked ResNet50V2 to extract object features of various granularity from a video. is used by selecting and using a block whose output size changes from a stack of convolution blocks. Specifically, one frame is represented by 6 vectors including the final layer, and vector information extracted for each block is as shown in FIG. 3 .

로 표현하고, 이것은 하기의 수학식1과 같이 정의된다. And the REsNet50V2 network function

, which is defined as Equation 1 below.

[Equation 1]

Also, since the learnable parameters of the 2D convolutional neural network module 110 can be sufficiently inferred from the structure of the existing ResNet50V2, they are not expressed for readability, and the characteristics of each object are shown in Table 1 below.

layer output
(Layer Output) layer name
(Layer Name) Dimension
(Dimension)

Pool1_pool 56x56x64

Conv2_block3_out 28x28x256

Conv3_block4_out 14x14x512

Conv4_block5_out 7x7x1024

Conv5_block6_out 7x7x2048

predictions 1000

Layer names in [Table 1] may differ depending on the deep learning library.

으로 임베딩한다. 이를 통해

번째 프레임의 최종 객체 특징인

는 하기의 수학식 2와 같이 정의된다. And in order to learn motion features using this object representation, mean pooling is performed on each object feature, and then the same dimension

embed into because of this

the last object feature of the first frame

Is defined as in Equation 2 below.

[Equation 2]

는 각 객체에 대한 임베딩 행렬이며, 학습 가능한 파라미터이고,

는

번째 표현(representation)의 출력 벡터의 마지막 차원 번호를 나타내는 것으로 표 1을 예로 들면,

는 64이다. 그리고

는 평균 풀링 작업을 의미하고,

번째 프레임의 객체 특징인

는

차원의 벡터이다. here

is an embedding matrix for each object, is a learnable parameter,

Is

Taking Table 1 as an example to indicate the last dimension number of the output vector of the th representation,

is 64. and

denotes an average pooling operation,

object features of the first frame

Is

is a vector of dimensions.

In addition, the average pooling of object characteristics for each frame extracted by the convolutional neural network module 110 may be transmitted to the second recurrent neural network module 210 .

Thereafter, the first recurrent neural network module 120 extracts motion characteristics using the extracted information for each frame (S120). Extracting the motion feature extracts motion information according to the difference between frames using information for each frame, and the first CNN module 120 calculates motion information for each frame sequence. The motion information is motion information extracted using first, second, and third frame information.

Based on the object features extracted by the convolutional neural network module 110 according to the present embodiment, the first recurrent neural network module 120 may learn the motion characteristics of the object. To this end, the first recurrent neural network module according to the present embodiment 120 preferably uses a long short-term memory network (LSTM).

로 표현되며, i번째 프레임에서 LSTM의 은닉 상태

는 하기 수학식 3과 같이 정의된다. The LSTM of the encoder 100 is a function

, and the hidden state of LSTM in the ith frame

Is defined as in Equation 3 below.

[Equation 3]

는 영벡터로 정의되고, 본 실시예에서는 이 은닉 상태를 동작 특징으로 고려한다. Since the learnable parameters of the first recurrent neural network module 120 can be sufficiently inferred from the existing LSTM structure, they are not separately expressed for readability. And the initial hidden state of the LSTM of the encoder 100

is defined as a zero vector, and this embodiment considers this hidden state as an operating characteristic.

Then, the second recurrent neural network module 210 extracts grammar features (S130).

The decoder 200 according to this embodiment is composed of two parts as shown in FIG. 2. The first part is a textural feature extraction part including the second recurrent neural network module 210, and the second part is It is a description generation part including the attention module 220 and the switch module 230.

Since the given description is a sequence of words, the second recurrent neural network module 210 of the decoder 200 may extract grammatical features using the given word sequence. It is possible to extract and learn what word information is required when generating the next word using the word in the previous step. To this end, it is preferable to use LSTM like the first recurrent neural network module 120 described above.

로 표현되며, t번째 단계의 LSTM의 은닉 상태

는 하기의 수학식 4와 같이 정의된다.Similar to the first recurrent neural network module 120 of the encoder 100, the LSTM of the second recurrent neural network module 210 of the decoder 200 is a function

, and the hidden state of the LSTM at the tth step

Is defined as in Equation 4 below.

[Equation 4]

Since the learnable parameters of the second recurrent neural network module 210 can also be sufficiently inferred from the existing LSTM structure, they are not separately expressed for readability.

는 단어를 표현하는 원핫(one-hot) 벡터를 나타내고,

는 임베딩 매트릭스를 나타내며,

는 단어

의

차원 임베딩 벡터를 나타낸다.

는 학습 가능한 파라미터이다. 디코더(200)의 LSTM의 초기 은닉 상태

와 셀 상태는 인코더(100)의 마지막 은닉 상태

와 셀 상태로 정의된다. here

represents a one-hot vector representing a word,

denotes an embedding matrix,

is the word

of

Represents a dimensional embedding vector.

is a learnable parameter. Initial Concealment State of LSTM of Decoder 200

and the cell state is the last hidden state of the encoder 100

and is defined as the cell state.

와 동작 특징 벡터

의 전체 프레임에 대한 평균 벡터의 결합 벡터인

는 하기의 수학식 5와 같이 정의된다.And the object feature vector

and motion feature vector

is the combined vector of the mean vectors for all frames of

Is defined as in Equation 5 below.

[Equation 5]

That is, since the video description, which is a given word description, contains video information, when extracting grammar features, grammar features are extracted based on the average of object features for each frame, which is general information of the video, and the average of motion features for each frame sequence. will be.

Thereafter, the attention module 220 calculates a weighted sum of object features and motion features, respectively (S140). In generating an actual word using the information extracted through the above process, information necessary for generating a word is selected through the attention module 220 at each stage. At this time, since the necessary information is information related to the word to be generated in this step, weights for important information are calculated based on object characteristics and motion characteristics, and these are selected as a weighted sum.

In addition, the attention module 220 may select required object characteristics and motion characteristics whenever a word is generated through an attention mechanism.

와 이전 단어 정보만을 이용하여 학습하기 때문에 단어 생성 시 중요한 프레임 정보를 참조할 필요가 있으며, 이를 위해 본 실시예에 따른 주의 모듈(220)은 주의 메커니즘(attention mechanism)을 사용하여 인코더(100)로부터 출력된 정보를 참조하며, 그 정보는 문맥 및 맥락(context)으로써 정의될 수 있다. The decoder 200 including the attention module 220 is the entire information of the image

Since learning is performed using only word information and previous words, it is necessary to refer to important frame information when generating words. It refers to the output information, and the information can be defined as context and context.

는 객체 표현(representation)

의 가중 합이며 이는 하기 수학식 6과 같다.First, the object context of the tth step,

is an object representation

is the weighted sum of , which is shown in Equation 6 below.

[Equation 6]

는 t번째 단어와 k 번째 프레임 간의 객체 정보에 대한 주의 (attention) 점수를 의미하며, 이는 하기 수학식 7과 같다. here

Means an attention score for object information between the t-th word and the k-th frame, which is expressed in Equation 7 below.

[Equation 7]

는 표현 소프트맥스 함수(represents a softmax function)이고,

,

는 학습 가능한 파라미터이다. 그리고,

는 입력 i번째 영상 벡터

와 출력 t번째 은닉 상태 벡터

와의 주의집중 점수를 의미하며, 이상의 수학식 6과 수학식 7에서

는 모든 입력

={1, 2, …, I}에 대한 합을 계산하기 위하여 k로 변수를 선언한 것으로 동일한 정의를 따르게 된다.At this time

is the representation softmax function,

,

is a learnable parameter. and,

is the input ith image vector

and the output tth hidden state vector

Means the attention score of and, in Equations 6 and 7 above

is all input

={1, 2, … , I} follows the same definition as declaring a variable as k to calculate the sum for I}.

번째 단계의 동작 문맥(context)인

은 동작 표현

의 가중 합으로 하기 수학식 8과 같다. Meanwhile

The operating context of the first step,

the action expression

As the weighted sum of Equation 8 below.

[Equation 8]

는 k번째 동작 표현

과

번째 문법 표현

사이의 주의(attention) 점수를 나타내며 이는 하기 수학식 9와 같이 정의될 수 있다.

is the kth action expression

class

first grammatical expression

represents an attention score between .

[Equation 9]

는 입력 i번째 은닉 상태 벡터

와 출력 t번째 은닉 상태 벡터

와의 주의집중 점수를 의미하며, 이상의 수학식 8과 수학식 9에서

는 모든 입력 i={1, 2, …, I}에 대한 합을 계산하기 위하여 k로 변수를 선언한 것으로 동일한 정의를 따르게 된다.and,

is the input ith hidden state vector

and the output tth hidden state vector

Means the attention score of and, in Equation 8 and Equation 9 above

is for all inputs i={1, 2, … , I} follows the same definition as declaring a variable as k to calculate the sum for I}.

Through the above process, in this embodiment, object and motion information of a video and previous word information can be obtained.

Afterwards, the switch module 230 generates word sequences for explanation (S150).

,

및

을 기반으로 하는 다음 단어 확률 로짓(logits)

,

은 하기의 수학식 10에서와 같이 정의된다.This is to generate a word generation probability logit based on the grammatical features calculated in the second recurrent neural network module 210, the weighted object features calculated in the attention module 220, and the weighted operational features, in the tth step Each fixed-size expression

,

and

Next word probability logits based on

,

Is defined as in Equation 10 below.

[Equation 10]

는 활성 함수(activation function)를 나타내고,

,

,

,

,

,

는 학습 가능한 파라미터이다.here

denotes an activation function,

,

,

,

,

,

is a learnable parameter.

In addition, in order to appropriately select necessary information from these expressions when generating the next word, the present embodiment includes the switch module 230. This means next word information when used and next word information when only operation characteristics are used.

를 하기 수학식 11을 통해 구할 수 있다. Final word probability distribution using multi-representation switching in this switch module 230

Can be obtained through Equation 11 below.

[Equation 11]

와

는 학습 가능한 파라미터이고, 각

는 t번째 단어를 디코딩할 때 각 표현의 중요성을 나타내며 이는 하기 수학식 12와 같이 정의된다.

and

is a learnable parameter, and each

Represents the importance of each expression when decoding the tth word, which is defined as in Equation 12 below.

[Equation 12]

는 학습 가능한 파라미터이다. here

is a learnable parameter.

The switch module 230 generates a word having the highest probability value in the word probability distribution, and repeats this process to generate a word sequence for video description. In this case, the word sequence may be generated through a beam search algorithm, which is one of optimal first search methods.

In this embodiment, thanks to the multi-expression switching through the switch module 230, the decoder 200 can separately consider the object and action information of the video and the text information in the description, and through this, one object or action can be converted into several words. can be modeled with That is, it is possible to select an expression suitable for generating the next word from expressions generated through each grammatical feature, object feature, and operation feature through the switch module 230 .

는 정답 단어

에 대한 음의 로그 우도 함수(negative log likelihood function)이며, 하기 수학식 13과 같이 정의된다. Loss function for training in deep learning models

is the correct answer word

is a negative log likelihood function for , and is defined as in Equation 13 below.

[Equation 13]

5 is a diagram for explaining a video captioning result according to this embodiment, FIG. 6 is a graph for explaining a video captioning result according to this embodiment, and FIG. 7 describes a video captioning result according to this embodiment. A graph for this and FIGS. 8 to 10 are diagrams for explaining video captioning results according to the present embodiment.

To objectively evaluate the performance of video caption generation through the above-described captioning method, the Microsoft Research Video Description (MSVD) data set, which is widely used in video captioning problems, was used. The MSVD data set is an open domain data set composed of images from various sources, such as cooking and movies. The total number of images is 1,970, and each image has about 41 descriptions on average.

Here, the explanations were tokenized using the wordpunkt tokenizer of the Natural Language Toolkit (NLTP), and all were changed to lowercase. Except for this space tokenization, preprocessing using any natural language processing technique was not performed. The number of all words in the data set is about 570,000, and the average number of tokens in each description is about 7. The total number of vocabularies in this data set is 9,745. As with previous studies, 1200 images were used for learning, 100 images for verification, and the remaining 670 images for evaluation. Then, while sampling a total of 20 frames at 2 frames per second, each frame was cut into squares and resized to 224x224 pixels. To center the RGB values, these values are divided by 127.5 and 1 is subtracted. The sample data is as shown in FIG. 5, and only 5 descriptions are shown.

Looking at FIG. 5 in detail, first, in FIG. 5 (a), it can be confirmed that the same entity 'squirrel' is referred to as 'Small animal', 'chipmunk', or 'hamster' with different words. You can also see 'peanut' or simply 'food' for the 'nut' you are eating. A similar pattern exists in the case of FIG. 5 (b), and there are typos such as 'ingrediants'. In the description of the video in Fig. 5(c), there is an onion, but you can confuse it with an orange if you don't pay enough attention. The description of this MSVD data is mainly composed of progressive form.

And for transfer learning, Microsoft Common Objects in Context (MSCOCO) and Flickr30k image data set were used in this embodiment. These images and descriptions were preprocessed identically to the MSVD dataset.

, 복수의 순환신경망(RNN)인

,

, 어텐션 메커니즘의 내부 프로젝션(projection)

, 단어 로짓

을 위한 차원수들은 모두 512로 설정되었다. 최대 입력

, 출력

길이는 각각 20, 10으로 설정되었다. Leaky rectified linear unit을 활성함수로 사용하였고, Adam optimizer를 학습률 5e-5,

of 0.9,

of 0.999,

of 1e-7로 사용하였다. 또한 MSVD의 training 셋과 이미지 데이터 셋이 등장한 단어들만 사용하였다. 이 설정에서 사전은 21,992 단어로 구성된다. 제안 방법의 초 매개변수를 정리하면 표 2와 같다.For the video captioning method according to this embodiment, ResNet50V2 pre-trained with ImageNet is used and frame embedding

, which is a plurality of recurrent neural networks (RNNs)

,

, the internal projection of the attention mechanism

, the word logit

The dimensions for are all set to 512. max input

, Print

The lengths were set to 20 and 10, respectively. A leaky rectified linear unit was used as an activation function, and an Adam optimizer with a learning rate of 5e-5,

of 0.9,

of 0.999,

of 1e-7 was used. In addition, only words that appeared in MSVD's training set and image data set were used. In this setup, the dictionary consists of 21,992 words. Table 2 summarizes the hyperparameters of the proposed method.

Hyper parameter Value Hyper parameter Value

20

512

10

512

512

512

[56, 28, 14, 7, 7, 1]

21,992

512

The beam search algorithm is one of greedy tree search algorithms, and is an algorithm in which the number of candidate groups for finding an optimal node is limited by a beam size. To generate word sequences based on the learned natural language generation model, many researchers have used this beam search algorithm. In this case, the goal of the search is to find the combination of words with the highest probability. In the present invention, a candidate for which an end token is generated among several generated candidate groups is separately stored. Then, the best word sequence was selected by combining the searched candidate groups up to the maximum length and the candidate groups ending with the previously stored end token. Since the length of the given description is short, such as machine translation or automatic summary problems, no length penalty was applied. In addition, a beam search algorithm with a beam size of 5 was used to explain the explanation.

As evaluation scales, like previous studies, we used the Bilingual Evaluation Understudy (BLEU), Metric for Evaluation of Translation with Explicit Ordering (METEOR), and Consensus-based Image Description Evaluation (CIDEr). Evaluation) was used as an evaluation scale. BLEU is a widely used scale for evaluating machine translation methods, and is a scale for evaluating the n-gram precision of automatically generated descriptions based on ground truth. We used BLEU-4 for evaluation. Similarly, METEOR is also a widely used scale in the evaluation of machine translation methods, and is a scale that evaluates word matching rates in semantic ways such as morphological analysis and synonym matching between automatically generated explanations and ground truth. CIDEr is a widely used metric for evaluating image captioning models, and is a metric for evaluating n-gram similarity weighted by the term Term Frequency - Inversed Term Frequency (TF-IDF). These scales have the meaning that higher values indicate better performance.

In addition, a workstation equipped with an RTX 3090 graphics card was used for fast learning, and the present invention was implemented with Tensorflow 2.3.

로 표현한다. 이 연구에서, 15개의 단일 파라미터가 저장되었고

로 표현된다.For the experiment, the present invention was pre-trained for up to 10 cycles of the image data set with a batch size of 32. After that, the MSVD data set was trained for 30,000 more steps with a batch size of 8, and parameters were saved at every 2,000 steps during learning. Here, this stored parameter is used as a single parameter

express it as In this study, 15 single parameters were stored and

is expressed as

하거나 단일 매개 변수의 METEOR 점수에 따라 가중 평균

하는 방법이다. 먼저 앙상블을 위하여 가장 성능이 좋은 다섯개의 단일 매개변수를 선택하였고, 또한 가장 좋은 두 개, 세 개 등의 조합도 고려하였다. 이 매개변수들은 가장 좋은 두 개의 조합부터 가장 좋은 다섯 개의 조합까지 각각

and

로 표현된다. 이 23개의 선택된 매개변수 중 가장 검증 성능이 좋은 매개변수는 검증 데이터를 이용하여 최종 미세조정(finetuning)된다.To further enhance the performance of the present invention, these single parameters were ensemble. The ensemble method used here is the arithmetic average of the values of single parameters.

or a weighted average based on the METEOR score of a single parameter

way to do it First, five single parameters with the best performance were selected for the ensemble, and the best two, three, etc. combinations were also considered. These parameters range from the best two combinations to the best five combinations, respectively.

and

is expressed as Among these 23 selected parameters, the one with the best verification performance is finally finetuned using verification data.

를 최적 매개변수

로 선택하였다. 앙상블을 위하여 가장 좋은 다섯 개의 매개변수

,

,

,

,

를 METEOR 점수에 따라 선택하였으며, 앙상블 매개변수의 BLEU-4와 METEOR 점수는 도 7에 도시된 바와 같으며 도 7 (a)는 산술 평균의 앙상블 매개변수, 도 7 (b)는 METEOR 점수에 의한 가중 평균의 앙상블 매개변수 점수이다. First, a single parameter was verified in the same way as in the evaluation using a beam search algorithm and an evaluation scale, and the BLEU-4 and METEOR scores of the single parameter are shown in FIG. 6 . In FIG. 6, the highest score is indicated by a circle, the corresponding value is indicated by a number, and the index of the optimal parameter is indicated by a vertical dotted line. Like the highest BLEU-4 and METEOR scores shown in Figure 6, a single parameter of 6 thousand (3 * 2,000) steps based on the verification data of MSVD

to the optimal parameter

was selected as The five best parameters for an ensemble

,

,

,

,

was selected according to the METEOR score, and the BLEU-4 and METEOR scores of the ensemble parameters are as shown in FIG. 7, FIG. 7 (a) is the ensemble parameter of the arithmetic mean, and FIG. is the weighted average ensemble parameter score.

와 33.51의

, 33.54의

를 테스트를 위하여 선택하였다. 최고의 METEOR 점수를 달성한 매개변수는

이다. 본 발명에서는 이 매개변수

를 MSVD의 검증 데이터 셋을 학습률 1e-5로 단 1 주기만 학습시켰고, 이 파인튜닝된 매개변수는

로 표현된다. 최적 단일 매개변수 기반 본 발명은 "Multi-Representation Switching with single parameter (MRS-s)"로 표현되었고, 최적 산술 평균 기반 앙상블 매개변수

와 가중 평균 기반 앙상블 매개변수

는 각각 "MRS-ea", "MRS-ew"로 표현되었다. 마지막으로 파인튜닝된 매개변수

기반 본 방법은 "MRS-ew+"로 표현되었다. Through verification, the optimal parameter METEOR score of 32.96

with 33.51

, of 33.54

was selected for testing. The parameter that achieved the highest METEOR score was

am. In the present invention, this parameter

was trained for only one cycle with a learning rate of 1e-5 on the verification data set of MSVD, and this fine-tuned parameter is

is expressed as Based on the optimal single parameter, the present invention was expressed as "Multi-Representation Switching with single parameter (MRS-s)", and the optimal arithmetic mean based ensemble parameter

and a weighted average based ensemble parameter

were expressed as "MRS-ea" and "MRS-ew", respectively. Last fine-tuned parameter

Basis This method was expressed as "MRS-ew+".

The experimental results are shown in Table 3 below. Methods for comparison are sorted in ascending order based on METEOR score.

Models BLEU-4 METEOR CIDEr Meanpool [16] 30.7 27.7 - SA-LSTM [4] 41.9 29.6 51.7 S2VT [2] - 29.8 - LSTM-GANs [8] 42.9 30.4 - GRU-RCN [5] 43.3 31.6 68.0 BAE [6] 42.5 32.4 63.5 h-RNNs [7] 49.9 32.6 65.8 PickNet [24] 46.1 33.1 76.0 STAT_V [25] 52.0 33.3 73.8 TSA-ED [26] 51.7 34.0 74.9 RecNetlocal [27] 52.3 34.1 80.3 MRS-s 51.8 32.0 64.9 MRS-ea 52.4 32.9 74.8 MRS-ew 53.3 32.9 73.1 MRS-ew+ 54.3 34.0 80.3

As can be seen from Table 3, the video captioning method according to the present invention consistently shows good performance for three criteria. In the ensemble model MRS-ea and MRS-ew, compared to the single model MRS-s, the BLEU-4 and METEOR scores increased slightly, but the CIDEr scores increased significantly. In particular, the MRS-ew+ of the present invention, which is a fine-tuned model, recorded scores exceeding the performance of most video captioning methods. This score can be regarded as a score achieved in 6,000 steps for MRS-s and 30,000 steps for MRS-e. Since the batch size is 8, 1 epoch consists of about 6,000 steps, and each model can be considered to have been trained only up to 1 epoch and 5 epochs.

With respect to learning time and method, PickNet, STAT_V, TSA-ED, and RecNet methods with similar performance to the video captioning method according to this embodiment were compared. PickNet was trained in three steps, and each step was trained up to 100 epochs. In the first step, we train encoder-decoder networks with negative log-likelihood loss for the target word. In the second step, the picking network is learned based on reinforcement learning, and the last step is to jointly learn the two networks.

TSA-ED learned up to 30 epochs, or stopped learning when verification performance did not improve for 20 epochs. RecNet, similar to TSA-ED, was trained until there was no change in validation loss for 20 epochs. In contrast, the present invention was only trained up to 5 epochs. RecNet additionally used reconstruction loss for learning. Compared to PickNet, the video captioning method according to the present invention showed good performance in all scales without reinforcement learning. Compared to TSA-ED, the proposed method recorded high scores in BLEU-4 and CIDEr, and the METEOR scores were the same. Compared to RecNet, the present invention recorded a BLEU-4 score 2 points higher, a METEOR score 0.1 point lower, and the same CIDEr score.

STAT_V used pretrained 2D CNN, C3d, and R-CNN for video feature extraction. According to the characteristics of the MSVD dataset without separate tagging of image information, the R-CNN used in STAT_V cannot be finetuned in an end-to-end manner. This structure has a limitation in that the feature extraction step and description generation step are inevitably separated during learning or evaluation. In contrast, the video captioning method according to the present invention has the advantage of smoothly fine-tuning each network without separating the feature extraction and description generation steps, and showed good performance in all scales compared to STAT_V.

From this result, it can be seen that the structure proposed in the present invention is effective in extracting information from a given video and description pair. This is because the video captioning method according to the present embodiment can be learned in a fully end-to-end manner, and we do not apply preprocessing based on any computer vision or natural language processing technique or an additional loss function.

Meanwhile, in order to confirm the qualitative difference of the video captioning method according to the present embodiment, descriptions generated with test data were compared. An example generated with appropriate subtitles is shown in FIG. 8 . As shown, the video captioning method according to the present invention generally generates an appropriate description for a given video, and it seems that models with better performance use richer expressions.

Also, an example of matching a wrong object and a correct object with a word is as shown in FIG. 9 . In several frames of the violin playing scene on the left, the violin appears to be mistaken for a flute because of the angle of the face and bow. Meanwhile, it is a frame of a scene where a man on the right is playing the flute, and here you can see that the shape of the face and instrument is similar to the one on the left. Looking at another example, FIG. 10 , the picture on the left is a tomato, but it is not properly recognized, which is expected to be due to the color of the tomato. In addition, the right frame is an image of a gray rabbit playing with a pink rabbit, and it can be seen that all models recognized the rabbit as a dog, but the model according to the present embodiment recognized a rabbit doll.

As described above, in this embodiment, a multi-expression switching-based video captioning method for modeling the characteristics of a video and subtitle pair is disclosed, and important information can be efficiently extracted from a given video and subtitle pair through such multi-expression switching. It can be known that there will be Through experiments using the MSVD dataset, the present invention was learned according to the proposed intention and achieved BLEU-4 of 54.3, METEOR of 34.0, and CIDEr score of 80.3. It can be seen that this record is a score that surpasses existing methods with very few learning steps, without a separate cup computer vision and natural language processing-based preprocessing and loss function.

Meanwhile, FIG. 11 is a block diagram for explaining a multi-expression switching-based video captioning system 300 according to an embodiment of the present invention, and includes a communication unit 310, an input unit 320, a storage unit 330, and an output unit 340. ) and a processor 350.

The communication unit 310 is provided to transmit/receive necessary information from an external device and an external network, and through this, learning data or video for generating captions can be input.

The input unit 320 is an input means for receiving a user command and can receive video for generating captions, and the output unit 340 is for displaying the process and result of multi-expression switch-based video captioning and includes a display. can do.

In addition, the storage unit 330 stores the data processed by the processor 350 temporarily or permanently by reading a program for performing the video captioning method and providing a necessary storage space for the processor 350 to operate. A volatile storage medium or a non-volatile storage medium may be included, but the scope of the present invention is not limited thereto. Also, the storage unit 330 may store data accumulated while performing the video captioning method.

Meanwhile, the processor 350 is a CPU and GPUs for learning the deep learning model for video captioning based on the multi-expression switch and generating video captions based on the multi-expression switch using the learned deep learning model. The processor 350 may control the entire process of providing a video captioning method.

The multi-expression switching-based video captioning method of the present invention can be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, as well as those known and available to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although various embodiments of the present invention have been shown and described above, the present invention is not limited to the specific embodiments described above, and is commonly used in the technical field to which the present invention pertains without departing from the gist of the present invention claimed in the claims. Of course, various modifications are possible by those with knowledge of, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100: encoder 110: 2D convolutional neural network module
120: first recurrent neural network module 200: decoder
210: second recurrent neural network module 220: attention module
230: switch module 300: system
310: communication unit 320: input unit
330: storage unit 340: output unit
350: processor

Claims (15)

The multi-expression switching-based video captioning method performed by the multi-expression switching-based video captioning system includes:
extracting object features on a frame-by-frame basis from an input video;
extracting motion features based on the extracted object features;
extracting grammatical features based on given word sequences;
calculating a weighted sum by calculating weights for the object features and motion features; and
A multi-expression switching-based video captioning method comprising generating a description of the video based on extracted grammar features, weighted object features, and weighted motion features. According to claim 1,
The step of extracting the object features,
A video captioning method based on multi-expression switching, comprising calculating mean pooling on extracted object features. According to claim 2,
In the step of extracting the grammatical features,
A video captioning method based on multi-expression switching, characterized in that a grammar feature is extracted based on a combination of an object feature vector extracted with the given word sequence and an average vector for all frames of the motion feature vector. According to claim 3,
In the step of calculating the weighted sum,
Calculate the weight per frame using an attention mechanism;
The multi-expression switching-based video captioning method of claim 1 , wherein the extracted object features and motion features required when generating words for the video description are selected through the attention mechanism whenever the words are generated. According to claim 4,
The step of generating the description is,
generating word generation probability expressions when only grammatical features are used, when only object features are used, and when only motion features are used;
calculating a final word probability distribution based on the word generation probability expression; and
A multi-expression switching-based video captioning method comprising generating a word having the highest probability value as a final word in the calculated word probability distribution. According to claim 5,
The step of generating the description is,
Characterized in that a word sequence for video description is generated by repeating the steps of generating each word generation probability expression, calculating the final word probability distribution, and generating a word with the highest probability value as the final word. A multi-representation switching-based video captioning method. According to claim 6,
In the step of generating the description,
The multi-expression switching-based video captioning method, characterized in that the generation of the word sequence is generated through a beam search algorithm. A computer-readable recording medium having a computer program recorded thereon for performing the multi-expression switching-based video captioning method according to claim 1. delete an input unit for receiving video input; and
A processor that generates video captions using a deep learning model when the video is input and receives a video and description pair as training data to train the deep learning model;
The deep learning model,
An encoder including a 2D convolutional neural network (CNN) module that extracts object features in frame units from an input video and a first recurrent neural network (RNN) module that extracts motion features based on the extracted object features; and
A second recurrent neural network module extracting grammatical features using a given word sequence, an attention module calculating a weighted sum by calculating weights per frame for the object features and motion features, and the extracted grammatical features, weighted object features, and A multiple representation switching-based video captioning system comprising a decoder comprising a switch module for generating a description of the video based on weighted summed motion characteristics. According to claim 10,
The 2D convolutional neural network (CNN) module,
A video captioning system based on multi-expression switching, characterized in that a mean pooling operation is performed on extracted object features. According to claim 11,
The second recurrent neural network module,
A video captioning system based on multi-expression switching, characterized in that a grammar feature is extracted based on a combination of an object feature vector extracted with the given word sequence and an average vector for all frames of the motion feature vector. According to claim 12,
The attention module,
Calculate the weight per frame using an attention mechanism;
The multi-expression switching based video captioning system, characterized in that the extracted object features and motion features required when generating words for the video description are selected through the attention mechanism whenever the words are generated. According to claim 13,
The switch module,
In the case of using only grammar features, in the case of using only object features, and in the case of using only motion features, word generation probability expressions are generated, respectively, and a final word probability distribution is calculated based on the word generation probability expression, and the most calculated word probability distribution is calculated. A video captioning system based on multi-expression switching, characterized in that a word sequence for video description is generated by repeating a process of generating a word having a high probability value as a final word. According to claim 14,
The switch module,
A video captioning system based on multi-expression switching, characterized in that the word sequence is generated through a beam search algorithm.

Images