CN112488063B - A Video Sentence Localization Method Based on Multi-stage Aggregate Transformer Model - Google Patents

Abstract

The invention discloses a video sentence positioning method based on a multi-stage aggregation Transformer model. In the video sentence Transformer model, each video slice or word can be adaptively aggregated and aligned according to its own semantics. Information about other video slices or words. Through multi-layer stacking, the finally obtained joint representation of video sentences has rich visual-linguistic cue capture ability and can achieve more fine-grained matching. In the multi-stage aggregation module, the stage features of the initial stage, the stage features of the intermediate stage and the stage features of the end stage are concatenated to form the feature representation of the candidate segment. Since the obtained feature representation captures specific information at different stages, it is very suitable for accurately locating the start and end positions of video clips. These two modules are integrated together to form an effective and efficient network to improve the accuracy of video sentence localization.

Description

A Video Sentence Localization Method Based on Multi-stage Aggregate Transformer Model

technical field

The invention belongs to the technical field of video sentence positioning and retrieval, and more particularly, relates to a video sentence positioning method based on a multi-stage aggregation Transformer model.

Background technique

Video localization is a fundamental problem in computer vision systems with a wide range of applications. In the past decade, a lot of research and related application system development have been carried out on video action localization. In recent years, with the rise of multimedia data and the diversification of people's needs, the problem of locating sentences in videos (video sentence localization) has gradually become important. The purpose of video sentence localization is to locate a certain video segment corresponding to the sentence to be queried in a long video. Compared with video action localization, sentence localization has more challenges and broader application prospects, such as video retrieval, automatic generation of video subtitles, human-computer intelligent interaction, etc.

Video sentence localization is a challenging task. In addition to understanding the video content, it is also necessary to match the semantics between the video and the sentence.

Existing video sentence localization can generally be divided into two categories: one-stage method and two-stage method. The one-stage method takes the video and the query sentence as input, directly predicts the start point and end point of the queried video clip, and directly generates the video clip associated with the query sentence. One-stage methods can be trained end-to-end, but they can easily lose some correct video clips. However, the two-stage approach follows two processes of candidate segment generation and candidate segment ranking. They usually first generate a series of candidate segments from the video, and then rank the candidate segments according to how well they match the query. Many methods follow this route. Although the two-stage method can recall many possible correct candidate video clips, there are still several key problems that have not been well solved:

1) How to effectively perform fine-grained semantic matching between videos and sentences?

2) How to accurately locate the starting point and ending point of the video clip matching the sentence in the original long video?

For the first problem, most of the existing methods usually process the video and sentence sequences separately, and then match them. However, separately processing the video and sentence sequence, such as encoding the sentence into a vector first and then matching, will inevitably lose some detailed semantic content in the sentence, so that detailed matching cannot be achieved;

For the second problem, existing methods usually use full convolution, average pooling or RoI Pooling operations to obtain feature representations of candidate segments. However, the features obtained by these operations are not temporally discriminative enough. For example, a video clip usually contains several different phases, such as the beginning, middle, and end. The information of these stages is very important for the precise location of the start and end points of the time. However, the average pooling operation completely discards the stage information and cannot precisely match different stages to achieve precise localization. Although fully convolutional operations or RoI Pooling operations can characterize different stages to a certain extent, they do not rely on explicit stage-specific features, so they are also lacking in more precise localization.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art, and to provide a video sentence localization method based on a multi-stage aggregation Transformer model, so as to improve the accuracy of video sentence localization.

In order to realize the above-mentioned purpose of the invention, the present invention is based on the video sentence localization method of multi-stage aggregation Transformer model, it is characterized in that, comprises the following steps:

(1), video slice feature, word feature extraction

The video is evenly divided into N time points according to time. At each time point, a video slice (composed of consecutive multiple frames, such as 50 frames of images) is collected, and feature extraction is performed on each video slice to obtain slice features. (to obtain a total of N slice features), the N slice features are placed in the order of time to form a video feature sequence;

Perform word turn vector (Doc2Vec) on each word of the sentence to obtain word features, and then place them in the order in the sentence to form a sentence feature sequence;

语句特征序列

其中，

表示视频第i个切片的切片特征，

表示语句第j个单词的单词特征；Map the slice feature in the video feature sequence and the word feature of the sentence feature sequence to the same dimension to obtain the video feature sequence

sentence feature sequence

in,

represents the slice feature of the ith slice of the video,

Represents the word feature of the jth word of the sentence;

(2), build a video sentence Transformer model, and calculate the video feature sequence and sentence feature sequence

Build a D-layer video sentence Transformer model, where the output of the d-th layer, d=1,2,...,D is:

Among them, V and L represent videos and sentences, respectively, Q, K, and W are learnable parameters, where different subscripts represent different parameters, and Att( ) is the attention calculation function;

语句特征序列

作为视频语句Transformer模型的输入进行计算，得到第D层输出视频特征序列

语句特征序列

video feature sequence

sentence feature sequence

Calculated as the input of the video sentence Transformer model to obtain the output video feature sequence of the D-th layer

sentence feature sequence

(3), build a multi-stage aggregation module, calculate the stage feature sequence and prediction score sequence of the three stages

Calculate the sequence of stage features r ^sta , r ^mid , r ^end for the start, middle and end stages:

组成，中间阶段特征序列r^mid由N个切片的阶段特征

组成，结束阶段特征序列r^end由N个切片的阶段特征

组成，

分别为计算三个阶段的阶段特征序列的多层感知器(MLP，Multi-layer Perceptron)；Among them, the initial stage feature sequence r ^sta consists of stage features of N slices

The intermediate stage feature sequence r ^mid consists of N slices of stage features

composed, the end stage feature sequence r ^end consists of N slices of stage features

composition,

Multi-layer Perceptron (MLP, Multi-layer Perceptron) that calculates the three-stage stage feature sequence;

Compute the sequence of predicted scores p ^sta , p ^mid , p ^end for the start, middle, and end stages:

组成，中间阶段预测分数序列p^mid由N个切片的预测分数

组成，结束阶段预测分数序列p^end由N个切片的预测分数

组成，

分别为计算三个阶段的预测分数序列的多层感知器；Among them, the initial stage prediction score sequence p ^sta consists of the prediction scores of N slices

composed, the intermediate stage prediction score sequence p ^mid consists of the prediction scores of N slices

composed, the end stage prediction score sequence p ^end consists of the prediction scores of N slices

composition,

are the multi-layer perceptrons that calculate the sequence of prediction scores in the three stages;

(4), train a multi-stage aggregate Transformer model

The video sentence Transformer model and the multi-stage aggregation module constitute a multi-stage aggregation Transformer model;

结束位置

Build a video sentence training dataset, where each piece of data includes a video, a sentence, and the start position of the video slice of the video clip where the sentence is located

end position

Propose a piece of data from the video sentence training data set, randomly mask a word in the sentence, and replace it with the mark "MASK", then process the video and sentence according to steps (1) to (3), and then calculate the start stage of each video slice , the real score of the middle stage, the end stage

in,

σ _sta , σ _mid σ _end are the standard deviation of the unnormalized two-dimensional Gaussian distribution, σ _sta , α _mid , α _end are positive scalars used to control the value of the standard deviation;

4.1), calculate the weighted cross entropy loss L _stage on the prediction layer:

以及匹配分数预测值

4.2) Calculate the predicted value of the start position and end position of the video slice of the zth candidate segment

and match score predictions

4.3), calculate the boundary regression loss L _regress :

分别为第z个候选片段的视频切片开始位置、结束位置；where Z is the total number of candidate fragments,

are the start position and end position of the video slice of the zth candidate segment, respectively;

4.4), calculate the matching score weighted cross entropy loss L _match :

到结束位置

的视频）的重合度；Among them, y _z is the video segment where the z-th candidate segment and the sentence are located (starting position

to the end position

video) coincidence;

4.5), calculate the cross-entropy loss L _word of masked word prediction

L _word = -logp ^mask

预测为屏蔽的单词的概率；Among them, p ^mask is based on the sentence feature sequence

the probability of a word predicted to be masked;

4.6), calculate the loss L _total of the entire network for training the multi-stage aggregated Transformer model

L _total =L _stage +L _regress +L _match +L _word

4.7), update the parameters of the entire network

Take a piece of data from the video sentence training data set in turn, and update the parameters of the entire network according to the loss L _total until the data in the video sentence training data set is empty, thus obtaining a trained multi-stage aggregate Transformer model;

(5), video sentence positioning

Input the video and the complete query sentence without masked words, process according to steps (1) to (3), and then calculate the matching score prediction value of each candidate segment and the prediction of the start position and end position of the video slice according to step 4.2). value (forming a new candidate fragment), then sort the new candidate fragments according to the matching score from high to low, and then use NMS (Non-Maximum Suppression) to remove the new candidate fragments that overlap more than 70%, and return to the top 1 Or the first 5 new candidate segments are used as the final positioned video segments.

The object of the present invention is achieved in this way.

In view of the problems existing in the above-mentioned existing methods, the present invention constructs a multi-stage aggregated Transformer model for the video sentence localization network. The multi-stage aggregation Transformer model consists of two parts: the video sentence Transformer model and the multi-stage aggregation module on top of the video sentence Transformer model. In the video sentence Transformer model, a single BERT architecture is kept, but the BERT parameters are decoupled into different groupings to process video and sentence information separately. The Video Sentence Transformer model models both video and sentence modalities more efficiently while maintaining the compactness and efficiency of a single BERT structure. In the Video Sentence Transformer model, each video slice or word can adaptively aggregate and align information from all other video slices or words in both modalities according to its own semantics. Through multi-layer stacking, the finally obtained joint representation of video sentences has rich visual-linguistic cue capture ability and can achieve more fine-grained matching. also. In the multi-stage aggregation module, three stage features corresponding to different stages are calculated for each video slice, namely the stage features of the start stage, the middle stage and the end stage. Then, for a candidate segment, the phase features of the beginning stage, the stage features of the intermediate stage and the stage features of the end stage are concatenated to form the feature representation of the candidate segment. Since the obtained feature representation captures specific information at different stages, it is very suitable for accurately locating the start and end positions of video clips. These two modules are integrated together to form an effective and efficient network to improve the accuracy of video sentence localization.

Description of drawings

Fig. 1 is a kind of specific implementation flow chart of the video sentence location method based on multi-stage aggregation Transformer model of the present invention;

Fig. 2 is a video slice schematic diagram;

Fig. 3 is the principle schematic diagram of a specific embodiment of the video sentence localization method based on the multi-stage aggregation Transformer model of the present invention;

FIG. 4 is a flow chart of a specific implementation manner of video sentence location.

Detailed ways

The specific embodiments of the present invention are described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that, in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

FIG. 1 is a flow chart of a specific implementation of the video sentence localization method based on the multi-stage aggregation Transformer model of the present invention.

In the present embodiment, as shown in FIG. 1 , the video sentence localization method based on the multi-stage aggregate Transformer model includes the following steps:

Step S1: Video slice feature and word feature extraction

In this embodiment, as shown in FIG. 2, the video is evenly divided into N time points according to time, and at each time point, a video slice (composed of consecutive multiple frames, such as 50 frames of images) is collected, Feature extraction is performed on each video slice to obtain slice features (a total of N slice features are obtained), and the N slice features are placed in the order of time to form a video feature sequence.

The word turn vector (Doc2Vec) is performed on each word of the sentence to obtain the word features, and then placed in the order in the sentence to form a sentence feature sequence.

语句特征序列

其中，

表示视频第i个切片的切片特征，

表示语句第j个单词的单词特征。Map the slice feature in the video feature sequence and the word feature of the sentence feature sequence to the same dimension to obtain the video feature sequence

sentence feature sequence

in,

represents the slice feature of the ith slice of the video,

The word feature representing the jth word of the sentence.

Step S2: Build a video sentence Transformer model, and calculate the video feature sequence and sentence feature sequence

Most existing methods usually process video and sentence sequences separately and then match them. However, processing these two sequences separately, such as first encoding the sentence into a vector and then matching, will inevitably lose some of the detailed semantic content in the sentence, making it impossible to achieve fine-grained matching. In order to solve this problem, as shown in Figure 3, the present invention constructs a brand-new video sentence Transformer model as the backbone network, wherein the output of the d-th layer, d=1,2,...,D is:

Among them, V and L represent videos and sentences, respectively, Q, K, and W are learnable parameters, where different subscripts represent different parameters, and Att( ) is the attention calculation function;

语句特征序列

作为视频语句Transformer模型的输入进行计算，按照公式(1)逐层计算，得到第D层输出视频特征序列

语句特征序列

video feature sequence

sentence feature sequence

Calculated as the input of the video sentence Transformer model, calculated layer by layer according to formula (1), to obtain the output video feature sequence of the D layer

sentence feature sequence

Compared with the single BERT model widely used before, the architecture of the multi-stage aggregated Transformer model in the present invention has not changed, and no additional calculation has been introduced, but different parameters are used to process different modal contents, which not only maintains the model The compactness and efficiency of the model also improve the multi-modal modeling ability of the model. At the same time, the multi-stage aggregated Transformer model in the present invention is also different from other multi-modal BERT models, which use two BERT streams to realize the content of different modalities. This model based on two BERT streams introduces an additional cross-modal layer to realize multi-modal interaction, while the multi-stage aggregated Transformer model in the present invention maintains the same structure as the original BERT model, which is more compact and efficient.

The multi-stage aggregate Transformer model consists of multiple layers of computational processes such as Equation (1). After multi-layer stacking, the resulting joint representation of video sentences has the ability to aggregate and align rich visual-linguistic cues. Each slice in the video can interact with each word in the query, enabling more detailed and accurate video sentence matching. This is very important for precise positioning.

Step S3: Build a multi-stage aggregation module to calculate the stage feature sequence and prediction score sequence of the three stages

语句特征序列

具有更丰富的信息和更精细的匹配。然而，为了克服现有方法忽略了视频片段中所包含的不同阶段的问题，本发明在视频语句Transformer模型上提出了一个多阶段聚合模块，以其达到能精确地定位所查询的视频片段的起始位置和结束位置。在多阶段聚合模块中，本发明分别为视频序列中的每一视频切片计算对应于不同时间阶段的三个阶段的阶段特征，即开始阶段、中间阶段和结束阶段的阶段特征

为了提高对不同阶段特征的区分能力，本发明在这些阶段特征上添加了一个预测层，分别对开始阶段、中间阶段和结束阶段的分数进行预测。After passing through the video sentence Transformer model, the obtained joint representation of the video sentence is the video feature sequence

sentence feature sequence

With richer information and finer matching. However, in order to overcome the problem that the existing method ignores the different stages contained in the video clip, the present invention proposes a multi-stage aggregation module on the video sentence Transformer model, so as to achieve the goal of accurately locating the queried video clip. start and end positions. In the multi-stage aggregation module, the present invention calculates the stage features of the three stages corresponding to different time stages, namely the stage features of the start stage, the middle stage and the end stage, for each video slice in the video sequence.

In order to improve the ability to distinguish different stage features, the present invention adds a prediction layer to these stage features, and predicts the scores of the starting stage, the middle stage and the ending stage respectively.

In this embodiment, as shown in FIG. 3 , the multi-stage aggregation module is used to calculate the stage feature sequences r ^sta , r ^mid , and r ^end of the start stage, the middle stage and the end stage:

组成，中间阶段特征序列r^mid由N个切片的阶段特征

组成，结束阶段特征序列r^end由N个切片的阶段特征

组成，

分别为计算三个阶段的阶段特征序列的多层感知器(MLP，Multi-layer Perceptron)。Among them, the initial stage feature sequence r ^sta consists of stage features of N slices

The intermediate stage feature sequence r ^mid consists of N slices of stage features

composed, the end stage feature sequence r ^end consists of N slices of stage features

composition,

They are the Multi-layer Perceptron (MLP, Multi-layer Perceptron) that calculates the three-stage stage feature sequence, respectively.

Compute the sequence of predicted scores p ^sta , p ^mid , p ^end for the start, middle, and end stages:

组成，中间阶段预测分数序列p^mid由N个切片的预测分数

组成，结束阶段预测分数序列p^end由N个切片的预测分数

组成，

分别为计算三个阶段的预测分数序列的多层感知器。Among them, the initial stage prediction score sequence p ^sta consists of the prediction scores of N slices

composed, the intermediate stage prediction score sequence p ^mid consists of the prediction scores of N slices

composed, the end stage prediction score sequence p ^end consists of the prediction scores of N slices

composition,

are the multi-layer perceptrons that compute the sequence of prediction scores for the three stages, respectively.

Step S4: Train a multi-stage aggregated Transformer model

The video sentence Transformer model and the multi-stage aggregation module constitute a multi-stage aggregation Transformer model;

结束位置

Build a video sentence training dataset, where each piece of data includes a video, a sentence, and the start position of the video slice of the video clip where the sentence is located

end position

Propose a piece of data from the video sentence training data set, randomly mask a word in the sentence, and replace it with the mark "MASK", then process the video and sentence according to steps (1) to (3), and then calculate the start stage of each video slice , the real score of the middle stage, the end stage

σ_sta、σ_mid、σ_end为未归一化的二维高斯分布的标准差，α_sta、α_mid、α_end为正值的标量，用于控制标准差的值，α_sta、α_mid、α_end的值越大，在视频片段的起始/中间/结束位置附近视频切片的起始/中间/结束位置的得分越高。in,

σ _sta , σ _mid , σ _end are the standard deviations of the unnormalized two-dimensional Gaussian distribution, α _sta , α _mid , α _end are positive scalars, used to control the value of the standard deviation, α _sta , α _mid , The larger the value of α _end , the higher the score for the start/middle/end position of the video slice near the start/middle/end position of the video segment.

Step S4.1: Calculate the weighted cross-entropy loss L _stage on the prediction layer:

以及匹配分数预测值

Step S4.2: Calculate the predicted value of the start position and end position of the video slice of the zth candidate segment

and match score predictions

Because all three stages are specific to the onset, middle, and end stages, this concatenated feature is very discriminative for precise video segment localization.

Step S4.3: Calculate the boundary regression loss L _regress :

分别为第z个候选片段的视频切片开始位置、结束位置。where Z is the total number of candidate fragments,

are the start position and end position of the video slice of the zth candidate segment, respectively.

Step S4.4: Calculate the matching score weighted cross-entropy loss L _match :

到结束位置

的视频)的重合度。Among them, y _z is the video segment where the z-th candidate segment and the sentence are located (starting position

to the end position

video) coincidence.

Unlike the previous IoU prediction without calculating the regression, the IoU prediction in the present invention is to predict the IoU between the candidate segment after regression and the real one, which enables the present invention to measure the quality of boundary regression.

To generate candidate segments, any candidate segment generation method can be applied within the framework of the present invention. For simplicity, the present invention first enumerates all possible video segments consisting of consecutive video slices. Then, for shorter videos, they can be densely selected as candidate segments. For longer videos, the sampling interval can be gradually increased and sparsely selected as candidate segments. The main idea of this method is to remove redundant candidate segments with large overlaps.

Step S4.5: Calculate the cross-entropy loss L _word for masked word prediction

L _word = -logp ^mask

预测为屏蔽的单词的概率。Among them, p ^mask is based on the sentence feature sequence

Probability of words predicted to be masked.

During the training process, the present invention takes sentence pairs as the input of the network. Similar to the original Transformer model, a word in the sentence sequence is randomly masked. For masked words, it is replaced with a special token "[MASK]". Then, let the model predict masked words based on the unmasked words and information from the video sequence. It is worth noting that predicting some important words, such as some nouns corresponding to objects and some verbs corresponding to actions, requires information from video sequences. Therefore, masked word prediction not only allows the Transformer model to learn language, but also better aligns video and sentence modalities. The loss function for matching word predictions is the standard cross-entropy loss. The present invention does not pre-train the video sentence Transformer model on any other data set, and all parameters are initialized randomly.

Step S4.6: Calculate the loss L _total of the entire network for training the multi-stage aggregated Transformer model

L _total =L _stage +L _regress +L _match +L _word

Step S4.7: Update the parameters of the entire network

The present invention does not pre-train the video sentence Transformer model on any other data set, and all parameters are initialized randomly.

Take a piece of data from the video sentence training data set in turn, and update the parameters of the entire network according to the loss L _total until the data in the video sentence training data set is empty, thus obtaining a trained multi-stage aggregate Transformer model;

Step S5: Video Sentence Location

Input the video and the complete query sentence without masked words, as shown in Figure 4, process according to steps (1) to (3), and then calculate the matching score prediction value of each candidate segment and the start of video slice according to step 4.2). The predicted value of the position and end position (forming a new candidate segment), then sort the new candidate segments according to the matching score from high to low, and then use NMS (Non-Maximum Suppression) to remove new candidates that overlap more than 70% segment, and return the top 1 or top 5 new candidate segments as the final positioned video segment.

Performance evaluation

The present invention conducts experiments on two large public datasets, ActivityNet_Captions [14] and TACoS [24]. ActivityNet_Captions contains 20K videos and 100K query sentences. The average length of the video is 2 minutes. It contains 127 videos related to cooking activities with an average duration of 4.79 minutes. On average, there are 148 queries per video. The TACoS dataset contains 18,818 segment-statement pairs. TACoS is a very challenging data. Its query statement contains multiple levels of activities that contain different levels of detail.

The invention is evaluated using Rank n@IoU=m. It represents the percentage of correct localizations of all localizations, where correct localization is defined as having at least one segment in the output that matches the ground truth. A segment matches the ground truth if the IoU between it and the ground truth is greater than m.

The network in the present invention is optimized using Adam. The batch size is set to 16 and the learning rate is set to 0.0001. The number of Transformer layers is set to 6 layers. The feature dimension of all layers is set to 512. The standard deviation scalars are 0.25, α _s , and α _m 0.21, respectively. In ActivityNet_Captions and TACoS, the number of Transformer attention heads is set to 16 and 32, respectively. Extract video slice features using C3D network. The sampled video slice length is set to 32 for the ActivityNet_Captions dataset and 128 for the TACo S dataset.

The multi-stage aggregation Transformer network proposed by the present invention is compared with various current state-of-the-art methods, and the comparison results are shown in Table 1-2.

Table 1

Table 1 shows the comparison results with other methods on the ActivityNet_Captions dataset.

Table 2

Table 2 shows the comparison results with other methods on the TACoS dataset.

It can be seen from the experimental results that the present invention achieves a significant improvement compared with the previous method. Although the present invention is 1.09 points lower than the Rank1@IoU=0.5 value of CSMGAN [18] on the ActivityNet_captions dataset, it outperforms CSMGAN on all other metrics. Especially for Rank1@IoU=0.7 and Rank5@IoU=0.7 indicators, the present invention is 2.63 and 3.55 points higher than CSMGAN, respectively. Note that IoU=0.7 is a stricter criterion for judging whether a segment is correct, which indicates that the present invention can achieve higher quality localization. In addition, on the TACoS dataset, the present invention is more than 10 percentage points higher than CSMGAN in all evaluation indicators, which shows the superiority of the present invention compared with the CSMGAN method. In addition, compared with other methods, the present invention also achieves overwhelming advantages, and these results fully demonstrate the effectiveness of the present invention. In the video sentence Transformer model of the present invention, each video slice can interact with each word in the query sentence, so as to obtain more detailed and accurate video sentence alignment. Thanks to the multi-stage aggregation module of the present invention, the computed video segment representation can match different stages of activity. The two modules of the present invention are closely combined to form a very effective and efficient fragment localization network.

Although the illustrative specific embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, As long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the inventive concept are included in the protection list.

Claims (1)

1. a video sentence positioning method based on multi-stage aggregation Transformer model, is characterized in that, comprises the following steps: (1), video slice feature, word feature extraction The video is evenly divided into N time points according to time. At each time point, a video slice is collected, which is composed of consecutive multi-frame images, and feature extraction is performed on each video slice to obtain a total of N slice features, N The slice features are placed in the order of time to form a video feature sequence; Perform word turn on each word of the sentence to obtain word features, and then place them in the order in the sentence to form a sentence feature sequence; Map the slice feature in the video feature sequence and the word feature of the sentence feature sequence to the same dimension to obtain the video feature sequence

sentence feature sequence

in,

represents the slice feature of the ith slice of the video,

Represents the word feature of the jth word of the sentence; (2), build a video sentence Transformer model, and calculate the video feature sequence and sentence feature sequence Build a D-layer video sentence Transformer model, where the output of the d-th layer, d=1,2,...,D is:

Among them, V and L represent videos and sentences, respectively, Q, K, and W are learnable parameters, where different subscripts represent different parameters, and Att( ) is the attention calculation function; video feature sequence

sentence feature sequence

Calculated as the input of the video sentence Transformer model to obtain the output video feature sequence of the D-th layer

sentence feature sequence

(3), build a multi-stage aggregation module, calculate the stage feature sequence and prediction score sequence of the three stages Calculate the sequence of stage features r ^sta , r ^mid , r ^end for the start, middle and end stages:

Among them, the initial stage feature sequence r ^sta consists of the stage features r _i ^sta of N slices, i=1, 2,...N, and the intermediate stage feature sequence r ^mid consists of the stage features r _i ^mid of N slices, i= 1,2,...N, the ^end stage feature sequence rend consists of N slices of stage features ^riend , _i =1,2,...N, MLP ₁ ^sta , MLP ₁ ^mid , MLP ₁ ^end Multi-layer Perceptron (MLP, Multi-layer Perceptron) that calculates the three-stage stage feature sequence; Compute the sequence of predicted scores p ^sta , p ^mid , p ^end for the start, middle, and end stages:

Among them, the initial stage prediction score sequence p ^sta consists of the prediction scores of N slices

composed, the intermediate stage prediction score sequence p ^mid consists of the prediction scores of N slices

composed, the end stage prediction score sequence p ^end consists of the prediction scores of N slices

composition,

are the multi-layer perceptrons that calculate the sequence of prediction scores in the three stages; (4), train a multi-stage aggregate Transformer model The video sentence Transformer model and the multi-stage aggregation module constitute a multi-stage aggregation Transformer model; Build a video sentence training dataset, where each piece of data includes a video, a sentence, and the start position of the video slice of the video clip where the sentence is located

end position

Propose a piece of data from the video sentence training data set, randomly mask a word in the sentence, and replace it with the mark "MASK", then process the video and sentence according to steps (1) to (3), and then calculate the start stage of each video slice , the real score of the middle stage, the end stage

in,

σ _sta , σ _mid σ _end are the standard deviation of the unnormalized two-dimensional Gaussian distribution, σ _sta , α _mid , α _end are positive scalars used to control the value of the standard deviation; 4.1), calculate the weighted cross entropy loss L _stage on the prediction layer:

4.2) Calculate the predicted value of the start position and end position of the video slice of the zth candidate segment

and match score predictions

in,

are the video slice start position, middle position, and end position of the zth candidate segment, respectively,

are the stage features of the corresponding positions of the stage feature sequences r ^sta , r ^mid , and r ^end obtained in step (3), respectively; 4.3), calculate the boundary regression loss L _regress :

Among them, Z is the total number of candidate fragments; 4.4), calculate the matching score weighted cross entropy loss L _match :

Among them, y _z is the starting position of the video segment where the zth candidate segment and the sentence are located

to the end position

The coincidence of the video; 4.5), calculate the cross-entropy loss L _word of masked word prediction L _word = -log p ^mask Among them, p ^mask is based on the sentence feature sequence

the probability of a word predicted to be masked; 4.6), calculate the loss L _total of the entire network for training the multi-stage aggregated Transformer model L _total =L _stage +L _regress +L _match +L _word 4.7), update the parameters of the entire network Take a piece of data from the video sentence training data set in turn, and update the parameters of the entire network according to the loss L _total until the data in the video sentence training data set is empty, thus obtaining a trained multi-stage aggregate Transformer model; (5), video sentence positioning Input the video and the complete query sentence without masked words, process according to steps (1) to (3), and then calculate the matching score prediction value of each candidate segment and the prediction of the start position and end position of the video slice according to step 4.2). value, and form new candidate fragments, then sort the new candidate fragments according to the matching score from high to low, and then use non-maximum suppression to remove the new candidate fragments that overlap more than 70%, and return to the top 1 or top 5 A new candidate segment is used as the final positioned video segment.

Images