CN114494314A - Timing boundary detection method and timing perceptron - Google Patents

Abstract

A time sequence boundary detection method and a time sequence perceptron are based on a transformation decoder structure and an attention mechanism, a universal non-category time sequence action detection model is established, a small amount of hidden feature query quantity is introduced into a coder of the detection model, input features are compressed to fixed dimensionality through a cross attention mechanism, and the features are decoded by the transformation decoder, so that the sparse detection of the universal non-category time sequence boundary is realized. The invention effectively solves the time sequence redundancy problem of the long video through the feature compression, and reduces the complexity of the quadratic model to the linear level; constructing two implicit characteristic query quantities, namely a boundary query quantity and a context query quantity, so as to correspondingly process a non-semantic-coherent boundary region and a coherent context region in the video and fully utilize the semantic structure of the video; an alignment loss function based on cross attention calculation is provided, so that the network can be converged quickly and stably; and the boundary position is sparsely coded by using a transform decoder, so that complex post-processing is avoided, and the generalization performance of the model is improved.

Description

Timing Boundary Detection Method and Timing Perceptron

technical field

The invention belongs to the technical field of computer software, relates to video timing boundary detection, and relates to a timing boundary detection method and a timing sensor.

Background technique

Due to the explosive growth of video data on the Internet, video content understanding has become an important problem in the field of computer vision. In the past literature, the exploration of long video understanding is still insufficient. Class-free temporal boundary detection is an effective technique for bridging the gap between long and short video understanding, which aims to segment long videos into a series of video segments. Classless temporal boundaries are naturally generated temporal boundaries due to semantic discontinuity. They are not constrained by any pre-defined semantic categories. Existing datasets include classless temporal boundaries of different granularities such as sub-action level, event level, and scene level. . For the detection of class-free temporal boundaries with different granularities, different levels of information are required to obtain temporal structure and context at different scales.

Currently, due to differences in temporal boundary semantics and granularity, research on category-free temporal boundary detection is divided into several distinct tasks. The goal of the temporal action segmentation task is to detect sub-action-level class-free temporal boundaries that split an action instance into multiple distinct sub-action segments. Universal temporal boundary detection aims to locate class-free temporal boundaries at the event level, i.e. moments of action/subject/environment change. Movie scene segmentation detects class-free temporal boundaries at the scene level, i.e. transitions between movie scenes that mark high-level plot turns. The target videos for these tasks have the same semantic structure, and their boundary detection paradigms exhibit similar characteristics. Previous work on these tasks has mainly focused on feature encodings carefully crafted for specific boundaries and reduced the boundary detection problem to a dense prediction problem. During the prediction process, these works employ sophisticated post-processing techniques to eliminate the large number of false positives that repeatedly predict the same true value in the results. Such complex design and post-processing modules are highly related to specific boundary types, so they cannot be well generalized to different types of class-free boundary detection and lack generalization.

SUMMARY OF THE INVENTION

The problem to be solved by the present invention is that the existing paradigms of unbounded temporal boundary detection have similar properties, but are scattered in different tasks due to differences in boundary semantics and granularity. Existing related works mainly focus on well-designed feature encoding for specific boundaries, and since the dense prediction paradigm employs sophisticated post-processing techniques to eliminate false positives, it does not generalize well to different types of classless boundary detection.

The technical scheme of the present invention is: a time sequence boundary detection method, constructing a classless time sequence boundary detection network to detect the time sequence boundary of the video, the detection network includes a backbone network and a detection model, and the implementation is as follows:

以每一帧生成一个视频段，第i段视频段为由第i帧图像f_i的前后连续k帧组成的图像序列，由骨干网络对输入的视频段生成视频特征

和连续性打分

F_i和S_i分别为视频段i的RGB特征和连续性打分；1) The detection sample is generated by the backbone network: the video image sequence is obtained by sampling the video interval

A video segment is generated for each frame, and the i-th video segment is an image sequence composed of consecutive k frames before and after the i-th frame image f _i . The backbone network generates video features for the input video segment.

and continuity score

F _i and S _i are the RGB features and continuity scores of video segment i, respectively;

2) based on the video feature F and the continuity score S, the detection model is used to perform unclassified sequential action detection, and the detection model includes the following configurations:

2.1) Encoder: Encoder: Encoder E includes N _e layers concatenated transform decoding layers, each layer contains a multi-head self-attention layer, a multi-head cross-attention layer and a linear mapping layer, self-attention layer, cross-attention layer The force layer and the linear mapping layer respectively have a residual structure, introduce M latent feature query quantities Q _e into the encoder, sort the video features F in descending order based on the continuity score S, and then input them into the encoder. The video feature is compressed into the compressed feature H of M frames, and the initial compressed feature H ₀ is 0. In the jth layer of the transformation and decoding layer, the hidden feature query quantity Q _e is added to the compressed feature H _j of the current layer, and after the self-attention layer and Its residual structure interacts with the reordered video features in the cross-attention layer, and then goes through the residual structure-linear mapping layer-residual structure transformation to obtain the compressed feature H _j+1 , j∈[1,(N _e - 1)], after stacking N _e coding layers, the compression and coding of the input features are realized, and the compressed features are obtained

Among them, the generation of latent feature query volume is: hidden feature query volume Q _e is divided into M _b boundary query volume and M _c context query volume, initialized randomly, and generated with training samples in the process of training the detection model; boundary query The amount corresponds to the boundary area feature in the video feature, and the context query amount corresponds to the context area feature in the video feature. The first M _b features after reordering in the video feature are the boundary area feature, and the others are the context feature;

2.2) Decoder: Decoder D includes N _d layers of concatenated decoding layers, each layer contains a multi-head self-attention layer, a multi-head cross-attention layer and a linear mapping layer, self-attention layer, cross-attention layer and linear The mapping layer has a residual structure respectively; for the compressed feature H obtained by the encoder, the decoder performs timing boundary point analysis by transforming the decoder structure, the decoder defines N _p nominated query quantities Q _d , and the nominated query quantities Q _d are The hidden feature query volume is the same, it is randomly initialized and then learned and generated during training, and the boundary nomination B ₀ is initialized to 0. In the jth layer, the nominated query volume Q _d is added to the boundary nomination B _j , after the self-attention layer and a time Residual structure, in the interaction between the cross attention layer and the compressed feature H, after the residual structure-linear mapping layer-residual structure transformation, the updated boundary nomination B _j+1 is obtained; after passing through the stacked N _d decoding layers, Realize the parsing of compressed features and get the nominated representation of timing boundaries

2.3) Generation and scoring of time-series boundary-free boundaries: For the obtained time-series boundary nominated representation B, it is sent to two different fully-connected layer branches: the positioning branch and the classification branch, which are respectively used to output the time-series boundary-free moments. and confidence scores;

2.4) Assign training labels: adopt a strict one-to-one training label matching strategy: according to the defined matching cost C, use the Hungarian algorithm to obtain a set of optimal one-to-one matches, each of which is assigned to a class-free boundary truth value. The predictions all obtain positive sample labels, and the corresponding boundary truth values are the training targets; the matching cost C consists of two parts, the position cost and the classification cost. The position cost is defined based on the absolute value of the distance between the prediction moment and the boundary truth moment, and the classification cost is based on the prediction. Confidence definition;

2.5) Submission of time-series classless boundaries: After generating a series of time-series class-free boundaries, the most credible time-series classless-boundary moments are screened out through the confidence score threshold γ, and submitted for subsequent performance measurement;

3) Training phase: The configured model is trained with training samples, using cross entropy, L1 distance and log function as loss functions, using AdamW optimizer, and updating network parameters through backpropagation algorithm, repeating steps 1) and Step 2), until the number of iterations is reached;

4) Detection: Input the video feature sequence and continuity score of the data to be tested into the trained detection model, generate time-series unclassified boundary moments and scores, and then use the method of 2.3) to obtain the time-series classless time sequence for performance measurement. Boundary time series.

The present invention also provides a timing sensor, which has a computer storage medium, and a computer program is configured in the computer storage medium, and the computer program is used for implementing the above-mentioned classless timing boundary detection network, and the computer program is implemented when executed. The above-mentioned timing boundary detection method.

The present invention proposes a general and unified architecture to process different types of classless time series boundary detection, which can compress redundant video input into stable and reliable feature representation based on attention mechanism and video semantic structure, and reduce model complexity; From a global context perspective, the location and confidence scores of arbitrary class-free temporal boundaries are sparsely, efficiently, and accurately given.

Compared with the prior art, the present invention has the following advantages

The present invention proposes a general classless time series boundary detection paradigm, and based on the transformation structure and attention mechanism, provides an efficient and unified method for any classless time series boundary detection.

The present invention introduces a small number of learnable latent feature queries as anchor boxes to compress redundant video inputs. Through the cross-attention mechanism, the latent feature query volume compresses the input into a fixed-size latent feature space while ensuring important boundary information, and reduces the spatiotemporal complexity of the model with the squared input length to linear complexity.

The invention constructs an effective latent feature query volume composition for timing boundary detection, including boundary query volume and context query volume. In order to better utilize the semantic structure composed of boundary and context in video, video features are also divided into boundary region features and context region features. The boundary query volume extracts the features of the boundary area in a targeted manner; the context query volume clusters the context area, and compresses the redundant context content into multiple context clustering centers.

The invention uses the alignment loss function to align the boundary query amount and the boundary area feature one-to-one, effectively reducing the training difficulty, shortening the convergence time, forming stable compression features, and improving the positioning prediction performance.

The invention utilizes the transform decoder and the one-to-one corresponding training label matching strategy, effectively utilizes the global context information for boundary prediction, sparsely and efficiently generates more accurate location-free time series boundary positions, and does not require complex post-processing techniques.

The invention has the characteristics of generality, high efficiency, accuracy and the like in the task of timing boundary detection. Compared with the existing methods, the present invention achieves better prediction accuracy and faster inference speed on sub-action level, event level and scene level unclassified time series boundary data sets, which reflects the generalization performance of the model.

Description of drawings

FIG. 1 is a system frame diagram used in the present invention.

FIG. 2 is a schematic diagram of frame extraction processing of video according to the present invention.

FIG. 3 is a schematic diagram of an encoder of the present invention.

FIG. 4 is a schematic diagram of a decoder of the present invention.

FIG. 5 is a schematic diagram of border nomination generation and submission of the present invention.

FIG. 6 shows the model efficiency comparison results of the present invention on an example MovieNet dataset.

Figure 7 shows the results of the present invention compared with previous work on the MovieNet dataset example.

Figure 8 shows the results of the present invention compared with previous work on the Kinetics-GEBD and TAPOS dataset samples.

Figure 9 shows the result visualization of the present invention on an example MovieNet dataset.

FIG. 10 is a schematic diagram of the overall flow of the present invention.

Detailed ways

The invention constructs a classless time sequence boundary detection network to detect the time sequence boundary of the video, and the detection network includes a backbone network and a detection model. The detection model of the present invention is a Temporal Perceiver, which is a general classless time-series boundary detection framework. A small number of latent feature query quantities are introduced as anchor boxes, and redundant inputs are compressed into a fixed dimension through a cross-attention mechanism to realize Class-free temporal boundary detection task. The method of the present invention includes a sample generation stage, a network configuration stage, a training stage, and a testing stage, as shown in FIG. 10 , and the specific description is as follows.

在视频图像序列L_f上采集N_f个长度为2k帧的视频段，其中第i个视频段为由第i帧图像的前后各连续k帧组成的图像序列

N_f为视频图像序列的长度，同时也为视频段的数目。每个视频段图像序列L_s,i被送入骨干网络，经过预训练和微调参数的卷积层、池化层和全连接层，输出第i帧对应的D维RGB特征F_i和连续性打分S_i。将不同视频段的特征和打分分别按照时间顺序拼接起来，得到整个视频特征

和连续性打分

其中，采样间隔τ表示在全局进行时间划分的细粒程度；视频段长度2k的大小表示特征的局部感受野范围。为减小时间复杂度的同时保留更多的局部信息，本发明实施例优选τ取3，k取5，D取2048。具体实施如下：1) Sample generation stage: Use the backbone network based on ResNet50 and time-series convolutional layers to generate samples for training and test videos, and the samples include video feature F and continuity score S. For each video, all the corresponding picture frames are sampled at intervals of τ frames to obtain a video image sequence

Collect N _f video segments with a length of 2k frames from the video image sequence L _f , wherein the ith video segment is an image sequence composed of consecutive k frames before and after the ith frame image

N _f is the length of the video image sequence, and is also the number of video segments. Each video segment image sequence L _s,i is sent to the backbone network, after pre-training and fine-tuning parameters of the convolution layer, pooling layer and fully connected layer, the D-dimensional RGB feature F _i and continuity corresponding to the i-th frame are output Score S _i . The features and scores of different video segments are spliced together in chronological order to obtain the entire video features

and continuity score

Among them, the sampling interval τ represents the fine-grained degree of global time division; the size of the video segment length 2k represents the local receptive field range of the feature. In order to reduce the time complexity while retaining more local information, in the embodiment of the present invention, τ is preferably 3, k is 5, and D is 2048. The specific implementation is as follows:

所有视频段组成的视频段序列记为

将视频段L_s,i∈R^10×224×224送入骨干网络，经过预训练和微调参数的ResNet50网络，得到中间特征F_mid,i∈R^10×2048，F_mid,i再经过时序卷积层和池化层，得到视频段的RGB特征F_i∈R²⁰⁴⁸，最后将F_i输入全连接层，得到连续性打分S_i∈R。将不同视频段的特征F_i和打分S_i分别按照时间顺序拼接起来，得到整个视频的特征

和连续性打分

视频特征和连续性打分通过滑动窗口法形成一系列片段输入模型处理，其中窗口长度N_ws为100，滑窗时无重叠取框。具体如下：Use denseflow to extract picture frames from the original video, and sample all picture frames of the video at 3-frame intervals to obtain a video image sequence L _{f of length N f .} Call the transforms package of the torchvision library to scale each picture frame to obtain a corresponding image with a scale of 224*224. N _f video segments with a length of 2k=10 frames are collected from the video image sequence L _f , wherein the ith video segment L _s,i consists of 5 consecutive frames before and after the ith frame image of the video, that is,

The video segment sequence composed of all video segments is denoted as

The video segment L _s,i ∈ R ^10×224×224 is sent to the backbone network, and after pre-training and fine-tuning the parameters of the ResNet50 network, the intermediate feature F _mid,i ∈ R ^10×2048 is obtained, and F _mid,i goes through the time series volume The accumulation layer and the pooling layer are used to obtain the RGB feature F _i ∈ R ²⁰⁴⁸ of the video segment. Finally, the F _i is input to the fully connected layer to obtain the continuity score S _i ∈ R . The features F _i and scoring S _i of different video segments are spliced together in chronological order to obtain the features of the entire video.

and continuity score

Video features and continuity scores are processed by a series of segment input models through the sliding window method, where the window length N _ws is 100, and there is no overlapping frame during sliding window. details as follows:

1. The overall video segment sequence obtained after frame extraction and sampling is as follows:

L _s,i ={f _i-5 ,f _i-4 ,f _i-3 ,f _i-2 ,f _i-1 ,f _i+1 ,f _i+2 ,f _i+3 ,f _i+4 ,f _i+5 ,}

Wherein V _f represents a video segment sequence, which consists of N _f image sequence segments L _s,i , and each image sequence segment contains 2k=10 images.

2. The backbone network processes the input video image sequence as follows:

F _mid,i =Resnet50(L _s,i )

F _i =MaxPooling(Tconv(F _mid,i ))

S _i =FC(F _i )

Among them, F _mid,i represents the intermediate feature of the input video segment processed by the Resnet50 network, F _i is the video segment feature obtained by the time series convolution of F _mid,i , S _i is the continuity scoring result, and F is the different video segments according to The feature sequence spliced in time sequence, S is the continuous scoring sequence spliced together by different video segments in time sequence.

2) In the network configuration stage, based on the transform decoder structure and attention mechanism, a general category-free time series action detection model Temporal Perceiver is established. The model includes the following configurations:

2.1) Encoder: Based on the continuity score S generated in 1), the video features F generated in 1) are projected and sorted in descending order to obtain F _rerank , and M learnable latent feature query quantities Q _e are introduced and initialized to 0 The compressed feature H ₀ of , uses the transform decoder structure to compress the reordered feature into the compressed feature H of M frames. The encoder E includes a series of transform and decoding layers of N _e layers, each layer is denoted as Encoder _j , j represents the coding layer number, j∈[0,(N _e -1)]. The input of Encoder _j is H _j and the output is H _j+1 . Each Encoder _j contains a multi-head self-attention layer MSA _j , a multi-head cross-attention layer MCA _j , a linear mapping layer FFN _j and three residual structures formed by addition and layer normalization operations.

The input of the multi-head self-attention layer MSA _j is the key parameter K, the query parameter Q and the value parameter V, and the key and query of the self-attention mechanism are the same input. The key parameter and the query parameter are self-multiplied and normalized by the Softmax function to obtain the weight matrix A _s , and the value parameter is multiplied by the weight matrix to obtain the output. The multi-head structure divides the input into multiple parts along the parameters, respectively inputs the self-attention layer, and finally combines the results along the channel. The input of the multi-head cross-attention layer MCA _j includes key, query and value parameters. The key parameter and the query parameter are multiplied and Softmax normalized to obtain the cross-attention weight A _c , and the value parameter is multiplied by the cross-attention weight A _c , get the output. The multi-head structure divides the input into multiple parts along the parameters, respectively enters the cross-attention layer, and finally combines the results along the channel.

In the present invention, it is preferred that N _e =6, M=60, the number of branches of the multi-head self-attention layer is 8, and the number of branches of the multi-head cross-attention layer is 8. The key and query parameters of MSA _j in the encoder are the result of the addition of the latent feature query quantity Q _e and the compressed feature H _j , the value parameter is the compressed feature H _j , the weight matrix A _s ∈ R ^M×M , and MSA _j passes through the residual error. The output after the structure is _H'j . The key parameter of MCA _j is the sum of the _Frerank video feature and the corresponding position code _Frerank +P _rerank , the value parameter is _Frerank , the query parameter is the sum of the output H′ _j of MSA _j and the latent feature query quantity Q _e , and the weight The matrix A _c ∈ R ^N×M , denote the output of MCA _j after the residual structure is H″ _j . The encoder E realizes the compression and encoding of input features by stacking N _e = 6 encoding layers, and the last layer of encoding layer The output of Encoder ₅ is the compressed feature H=H ₆ output by encoder E. The specific calculation is as follows:

1. Reordering transformation of video features and position encoding:

F _rerank = sort(fc _proj (F), S)

_Prerank = sort(P,S)

Among them, P is the position code formed after the relative position of the time series corresponding to F takes the sin function, which corresponds to F one-to-one; fc _proj is the linear projection used by the video feature from the input dimension D=2048 to the model dimension D _model =512 Layer, the subscript proj is the abbreviation of projection.

2. The encoding process of a certain layer of encoding layer Encoder _j in the encoder:

H′ _j =LayerNorm(H _j +MSA _j (H _j ,Q _e ))

H″ _j =LayerNorm(MCA _j (H′ _j ,Q _e ,F _rerank ,P _rerank )+H′ _j )

H _j+1 =LayerNorm(FFN _j (H″ _j )+H″ _j )

3. The encoding process of the multi-head self-attention layer MSA:

SA _h (x _h ,q _h )=x _h W _v,h A _s,h

A _s,h =Softmax((x _h +q _h )Wk,h·(x _h +q _h )W _q,h )

Among them, N _h represents the number of head branches in the multi-head self-attention layer, W _k,h , W _q,h , W _v,h and W _o are the projection matrices corresponding to keys, queries, values and outputs, and the h subscript Refers to different branches, and the projection matrix parameters of each layer and each branch are not shared. SA _h represents a single-head self-attention layer, x _h , q _h are slices of the feature x and its position encoding q along the channel dimension, respectively, there are N _h slices in total, A _{s, h} represent the h-th branch of the self-attention matrix .

4. The encoding process of the multi-head cross-attention layer MCA:

CA _h (x _h ,q _x,h ,y _h ,q _y,h )=y _h W _v,h A _c,h

A _c,h =Softmax((y _h +q _y,h )W _k,h ·(x _h +q _x,h )W _q,h )

Among them, W _k,h , W _q,h , W _v,h and W _o are the projection matrices corresponding to keys, queries, values and outputs. The projection matrix parameters of each layer and each branch are not shared, and cross attention The layer's projection matrix is also not shared with the self-attention layer. CA _h represents a single-head cross-attention layer, x _h , y _h , q _{x, h} , q _{y, h} are slices of x, y and the corresponding position encoding q _x , q _y along the channel dimension, respectively, A _{c, h} represent The cross-attention matrix of the hth branch.

5. The encoding process of the encoder:

H _j+1 =Encoder _j (H _j ; Q _e ,F _rerank ,P _rerank )

H=H6=Encoder ₅ (Encoder ₄ (Encoder ₃ (Encoder ₂ (Encoder ₁ (Encoder ₀ (H _{0 )} ))))))

2.2) The present invention introduces the latent feature query quantity Q _e to the encoder, and its generation is specifically: in order to make better use of the semantic structure of the video, the latent feature query quantity Q _e is divided into M _b boundary query quantities and M _c Context query amount, M _b =48, M _c =12; the reordering features are divided into boundary area features and context area features, and the first M _b features are boundary area features. The definition of boundary area features and context area features is based on the continuity score S. After the video features are reordered according to the descending order of S, the top M _b features with higher scores form boundary area features, and the rest are context area features. The implicit boundary query quantity Q _e of the present invention is a learnable parameter. According to the above description, the implicit boundary query quantity Q _e is randomly initialized and generated by learning during the model training process. The first M _b query volumes are defined as boundary query volumes, and the boundary region features are combined one-to-one in the compression process; the remaining M _c query volumes are context query volumes, and the context region features are clustered during the compression process. , to obtain M _b cluster centers representing context information.

2.3) Further, in order to improve the coding effect of the hidden boundary query quantity Q _e on the video features, the hidden feature query quantity and the video features are aligned during training: in order to speed up the model convergence and obtain more stable compression features, the present invention introduces additional supervision. Constraints, the alignment loss function is calculated based on the cross-attention matrix of the last layer. The calculation of the loss function is to take the negative logarithm of the sum of the diagonal weights of the front M _b ×M _b regions of the matrix, and maximize the diagonal weight by minimizing the loss function, so as to achieve the one-to-one ratio of boundary query volume and features. Align. The specific calculation is as follows:

The calculation process of the alignment loss function L _align :

Among them, α _align =1 is the parameter of the alignment loss function, and the alignment loss function is only calculated for the last cross-attention layer of the encoder.

2.4) Decoder: For the compressed feature H obtained in 2.1), adopt N _p learnable nominated query quantities Q _d and the boundary nominated representation B ₀ initialized to 0, and perform time-series boundary point analysis by transforming the decoder structure. The query quantity Q _d is the same as the latent feature query quantity, which is randomly initialized and then learned and generated during training; the decoder D includes N _d layers of transform decoding layers, the j-th decoding layer is denoted as Decoder _j , and the input is the boundary nominated representation B _j , The output is B _j+1 . Each decoding layer consists of a multi-head self-attention layer, a multi-head cross-attention layer, a linear mapping layer, and three residual structures formed by addition and layer normalization operations. In the jth layer, the nominated query volume Q _d is added to the boundary nomination B _j , after the self-attention layer and the primary residual structure, the cross-attention layer interacts with the compressed feature H, and the residual structure-linear mapping layer-residual The updated boundary nomination B _j+1 is obtained after the difference structure transformation. After passing through the stacked N _d decoding layers, the compressed features are analyzed, and the temporal boundary nominated representation B is obtained.

记MSA_j经过残差结构之后的输出为B′_j。MCA_j的键参数为压缩特征H和作为压缩位置编码的隐特征查询量Q_e之和H+Q_e，值参数为H，查询参数为MSA_j的输出B′_j和提名查询量Q_d之和，权重矩阵

记MCA_j经过残差结构之后的输出为B″_j。解码器D和编码器E对称，通过堆叠N_d＝6个编码层，实现边界提名表示的解码，最后一层解码层Decoder₅的输出B₆＝B，将会作为解码器最终输出放入全连接分支内进行边界位置和置信度的预测。具体计算如下:In the present invention, preferably N _d =6, N _p =10, the number of branches of the multi-head self-attention layer is 8, and the number of branches of the multi-head cross-attention layer is 8. The key and query parameters of MSA _j are the result of the addition of the nominated query quantity Q _d and the boundary nominated representation B _j , the value parameter is the boundary nominated representation B _j , and the weight matrix

Denote the output of MSA _j after passing through the residual structure as B′ _j . The key parameter of MCA _j is the sum H+Q _e of the compressed feature H and the latent feature query quantity Q _e as the compressed position code, the value parameter is H, and the query parameter is the sum of the output _B'j of MSA _j and the nominated query quantity Q _d and, the weight matrix

Denote the output of MCA _j after the residual structure is B″ _j . The decoder D and the encoder E are symmetrical. By stacking N _d = 6 coding layers, the decoding of the boundary nominated representation is realized, and the output of the last decoding layer Decoder ₅ B ₆ =B, it will be put into the fully connected branch as the final output of the decoder to predict the boundary position and confidence. The specific calculation is as follows:

1. The decoding process of a certain layer of decoding layer Decoder _j in the decoder:

B′ _j =LayerNorm(B _j +MSA _j (B _j ,Q _d ))

B″ _j =LayerNorm(MCA _j (B′ _j ,Q _d ,H,Q _e )+B′ _j )

B _j+1 =LayerNorm(FFN _j (B″ _j )+B″ _j )

2. The decoding process of the decoder:

B _j+1 =Decoder _j (B _j ; Q _d , H, Q _e )

B=B6=Decoder ₅ (Decoder ₄ (Decoder ₃ (Decoder ₂ (Decoder ₁ (Decoder ₀ (B _{0 )} ))))))

2.5) Generation and scoring of temporal classless boundaries: For the temporal boundary nomination representation B obtained in 2.4), it is sent to two different branches of the fully connected layer: the positioning branch Head _loc and the classification branch Head _cls , the two branches are respectively used for Output time series moments and confidence scores without class boundaries. The predicted boundary moment is a decimal between 0 and 1, indicating the relative position in the current segment; the confidence score is two scores, including positive class confidence and negative class confidence, higher scores represent more Most likely the corresponding category. The classification branch is composed of a fully connected layer, and the input and output feature dimensions are 512 and 2, respectively; the localization branch is composed of a multilayer perceptron composed of three fully connected layers and a sigmoid activation function, and the input and output dimensions are 512 and 512, respectively. , 512 and 512, 512, 1. The specific calculation is as follows:

1. Prediction of classless action localization time t:

t=sigmoid(fc ₂ (fc ₁ (fc ₀ (B))))

Among them, the three-layer fully connected layers of the positioning branch are respectively fc ₂ , fc ₁ , and fc ₀ , and the input and output dimensions are 512, 512, 512 and 512, 512, and 1, respectively.

2. Generation of binary classification confidence score p _pos :

p _pos ,p _neg =fc(B)

Among them, the fully connected layer of the confidence branch is fc, the positive example score p _pos is generally taken as the confidence score, the input dimension is 512, and the output dimension is 2.

2.6) Assign training labels: adopt a strict one-to-one training label matching strategy: According to the defined matching cost C, use the Hungarian algorithm to obtain a set of optimal one-to-one matches, each of which is assigned to a class-free boundary truth value. The predictions all obtain positive sample labels, and the corresponding boundary truth values are the training targets; the matching cost C consists of two parts: the position cost and the classification cost. Confidence definition. The specific calculation is as follows:

1. The optimization index of the Hungarian algorithm:

The optimization index is denoted as C. The optimization index has two components, the location cost and the classification cost, and each component has a corresponding weight, denoted as α _loc and α _cls respectively. In the present invention, α _loc =5 and α _cls =1 are preferred.

2. Definition of optimized components:

L _cls,n =-p _pos,n

的距离绝对值来衡量；第n个预测的分类分量L_cls,n由预测出的该时刻为边界的置信度p_pos,n来度量，由于C为最小化目标，所以对置信度取负；σ(·)是一个从预测到边界真值的映射，σ(n)为第n个预测所对应的动作真值。In the optimization component, the nth predicted localization component L _loc,n is determined by the predicted boundary time t _n and the position of the corresponding boundary truth

Measured by the absolute value of the distance; the nth predicted classification component L _cls,n is measured by the predicted confidence p _pos,n of the boundary at this moment. Since C is the minimization target, the confidence is negative; σ(·) is a mapping from prediction to boundary truth, and σ(n) is the action truth corresponding to the nth prediction.

2.7) Submission of time-series unclassified boundaries: After generating a series of time-series unclassified boundaries, the most credible time-series unclassified boundary moments are filtered through the confidence score threshold γ, and submitted for subsequent performance measurement. When p _pos,n ≥γ, the predicted position is submitted as the result, and when p _pos,n <γ, the predicted position is discarded. In the present invention, γ=0.9 is preferred.

3) Training phase: The configured model is trained with training samples, using cross entropy and L1 distance as the loss function based on the final result, log function as the loss function based on the intermediate result, using the AdamW optimizer, through the back propagation algorithm to update the network parameters, and repeat steps 1) and 2) until the number of iterations is reached;

4) Test phase: Input the video feature sequence and continuity score of the data to be tested into the Temporal Perceiver model that has been trained, generate time series without category boundary time and score, and then obtain the time series for performance measurement through the method of 2.5). A class-free time series.

The present invention proposes a temporal perceptron model, a general classless temporal boundary detection framework. Further description will be given below through specific embodiments. After training and testing on the TAPOS dataset, the Kinetics-GEBD dataset and the MovieNet/MovieScenes dataset, high inference speed and high accuracy are achieved. The Python3.8.8 programming language is preferably used, and the PyTorch 1.7.0 deep learning framework is implemented.

Fig. 1 shows the system frame diagram used in the present invention, and the specific implementation steps are as follows:

1) In the preparation stage for generating samples, as shown in Figure 2, both training data and test data are processed in the same way. Use denseflow to extract picture frames from the video, sample the picture frame sequence at an interval of τ=3, use torchvision's transforms package to scale and transform the picture frames to a scale of 224*224, and finally convert them into tensor form and normalize. Taking each frame in the picture frame sequence as the center, take k=5 frames before and after it and input it into the backbone network to obtain the corresponding feature and continuity score of the frame, and combine along the time sequence dimension to obtain the video feature and video continuity score. Video-level features and continuity scores are split into a series of video window segments of the same length and fed into the model.

2) In the configuration stage of the model, first score based on continuity, as shown in Figure 3, for the extracted video features, the program first projects the video features to a lower feature dimension, and encodes the corresponding features according to the continuity score. Sort descending, enter the encoder. The encoder consists of alternating multi-head self-attention modules, multi-head cross-attention modules, linear mapping layers, and an additive-union layer normalized residual structure. A set of learnable latent feature query volumes, including boundary query volumes and contextual query volumes, are fed into the encoder. The compressed features are initialized to 0, participate in the calculation of the coding layer and accumulate the features extracted from the input features by each coding layer. The latent feature query volume can be viewed as a positional encoding of compressed features. Compressed features and position encoding are added to obtain position information and participate in attention calculation.

First, the compressed feature and latent feature query volume are added as the key and query input to the self-attention layer, which is input as a value parameter separately. In the self-attention layer, the relationship between features is compressed by global self-attention modeling, and mutual information is extracted for feature renew. The compressed features are added after the update and before the update and normalized to get the intermediate representation. Afterwards, the intermediate representation and the latent feature query amount as the position encoding are added again as the query amount and input to the cross-attention module. The cross-attention layer extracts boundary features and clusters context features from the sorted video features, calculates the dot product activation of the input video sequence through the compressed feature and latent feature query volume to extract useful video features, updates the compressed features, and achieves feature extraction and integration. The effect of reducing complexity. Finally, the results of the cross-attention layer go through the residual structure, the projection of the linear mapping layer and another residual structure to obtain the cumulative updated compressed features.

The step of decoding the encoded feature to obtain the final result representation is the aforementioned step 2.4), as shown in Figure 4. The input nomination query volume and boundary nominations are fed into the multi-head self-attention layer for reinforcement of the nomination representation, and then fed into the cross-attention layer. The learned latent feature query amount is used as the positional encoding of the compressed feature and participates in the calculation of cross attention. The cross-attention layer extracts the boundary position information for the compressed features, finds the weight of each time dimension position of the compressed features through the cross-attention matrix multiplied by the compressed features and the boundary nomination, and extracts the corresponding features accordingly. The boundary nomination accumulates the extracted boundary representations at each layer, and finally obtains the decoding result. There is an additive normalized residual transform between the self-attention layer and the cross-attention layer, a second residual transform after the cross-attention layer, a linear projection layer projection and a third residual transform .

The decoding and submission of timing boundary nominations is shown in Figure 5. The boundary nomination representations are fed into the localization and classification branches represented by fully connected layers to obtain location and confidence scores, respectively. The localization branch consists of three fully connected layers (fc) and a sigmoid activation function to finally get the position. The classification branch is binary classification, including a layer of fc layer, and obtains the confidence score corresponding to the current predicted position. Confidence scores are screened and if it is above the threshold γ=0.9, then the prediction is submitted as the final prediction.

3) In the training phase, this example uses cross-entropy, L1 distance and negative logarithmic function as the loss function, uses the AdamW optimizer, and sets the batch size to 64, that is, each training takes 64 window samples in the training set for training, and the total training The number of epochs is set to 100, the initial learning rate is 2e-4, the learning rate has no decay strategy, and the model is trained on an NVIDIA RTX 2080ti GPU. The process from the original video to the time series boundary results is divided into two stages. In the first stage, the backbone network is fine-tuned on the dataset based on the pre-training parameters, and the video features and continuity scores are obtained; in the second stage, the TemporalPerceiver is trained and tested. In the stage of assigning positive and negative samples, the model adopts a strict one-to-one correspondence matching strategy to reduce the occurrence of false positives, so that the model can sparsely and efficiently predict class-free time series boundaries.

4) Test phase

其中召回率指预测正确的样本数占总真值数的比例，精准率为预测正确的样本数占所有预测为真值的样本数的比例，预测误差在相对距离内的样本即可算为预测正确。AP(Average Precision)的计算依据是召回值从0到1对应的平均精度值。M_iou为预测边界和真值边界的距离和真值场景长度的交并比加权和。AP和M_iou指标均要求预测结果必须和真值位置一致才能算作正确预测。The preprocessing of the test set is the same as the training data, which is compressed and transformed into a size of 224*224 after the frame is extracted, and the RGB feature extraction is performed using the backbone network based on ResNet50. The test metrics used are different on the datasets of different tasks: the f1 scores under different relative distances (Rel.Dis.) are used as the sub-action unclassified temporal boundary and the event unclassified temporal boundary as the metric, and the scene-level unclassified time sequence boundary adopts AP and M _iou as indicators. The F1 score is calculated based on recall and precision.

The recall rate refers to the ratio of the number of correctly predicted samples to the total number of true values, the precision is the ratio of the number of correctly predicted samples to the total number of samples predicted to be true values, and the samples whose prediction error is within the relative distance can be regarded as the correct prediction . AP (Average Precision) is calculated based on the average precision value corresponding to the recall value from 0 to 1. M _iou is the weighted sum of the intersection ratio of the distance between the prediction boundary and the ground truth boundary and the length of the ground truth scene. Both AP and M _iou indicators require that the prediction result must be consistent with the true value position to be counted as a correct prediction.

On three randomly selected videos from the MovieNet dataset, as shown in Figure 6, TemporalPerceiver has 7 times faster scene inference speed per second and nearly 200 times smaller floating-point calculation times than the classic work LGSS, reflecting the model The advantages of sparse prediction and no post-processing module; compared with the Transformer variant, TemporalPerceiver also has faster scene inference speed per second and fewer floating-point operations, reflecting the role of feature compression in model lightweighting, It further proves the efficiency of Temporal Perceiver. In terms of prediction accuracy, compared with previous work, TemporalPerceiver has achieved a huge improvement in all indicators of all datasets, reflecting the generality and generalization of the model: as shown in Figure 7, on the MovieScenes dataset, Temporal Perceiver is in AP and M. The _iou indicators surpass the classic work LGSS by nearly 3%; as shown in Figure 8, in the Kinetics-GEBD and TAPOS datasets, based on all indicators of f1@Rel.Dis., the performance of Temporal Perceiver is higher than the previous state-of-the- art work PC, where on the Kinetics-GEBD dataset, based on the f1@0.05 metric, Temporal Perceiver outperforms the former state-of-the-art work PC by 12.6%; on the TAPOS dataset, based on the average f1 score, Temporal Perceiver Perceiver outperformed the former state-of-the-art working PC by 9%, demonstrating the more flexible and accurate predictions of Temporal Perceiver. A more specific prediction visualization on the MovieNet dataset is shown in Figure 9. Temporal Perceiver's prediction avoids false positives near the true value and accurately predicts the true value of each scene.

Claims (8)

1. The time series boundary detection method is characterized by constructing a classless time series boundary detection network to detect the time series boundary of the video. The detection network includes a backbone network and a detection model. The implementation methods are as follows: 1) The detection sample is generated by the backbone network: the video image sequence is obtained by sampling the video interval

A video segment is generated for each frame, and the i-th video segment is an image sequence composed of consecutive k frames before and after the i-th frame image f _i . The backbone network generates video features for the input video segment.

and continuity score

F _i and S _i are the RGB features and continuity scores of video segment i, respectively; 2) based on the video feature F and the continuity score S, the detection model is used to perform unclassified sequential action detection, and the detection model includes the following configurations: 2.1) Encoder: The encoder _E includes a series of transform decoding layers of Ne layers, each layer includes a multi-head self-attention layer, a multi-head cross-attention layer and a linear mapping layer, self-attention layer, cross-attention layer and The linear mapping layer has a residual structure respectively, introduces M latent feature query quantities Q _e to the encoder, sorts the video features F in descending order based on the continuity score S, and then inputs them into the encoder, and the encoder compresses the sorted video features. is the compression feature H of M frames, and the initial compression feature H ₀ is 0. In the jth layer of transformation and decoding layer, the latent feature query quantity Q _e is added to the compressed feature H _j of the current layer, and after the self-attention layer and its residual Structure, the cross attention layer interacts with the reordered video features, and then the compressed feature H _j+1 , j∈[0,(N _e -1)] is obtained after the residual structure-linear mapping layer-residual structure transformation , after stacking N _e coding layers, the compression and coding of the input features are realized, and the compressed features are obtained

Among them, the generation of latent feature query volume is: hidden feature query volume Q _e is divided into M _b boundary query volume and M _c context query volume, initialized randomly, and generated with training samples in the process of training the detection model; boundary query The amount corresponds to the boundary area feature in the video feature, and the context query amount corresponds to the context area feature in the video feature. The first M _b features after reordering in the video feature are the boundary area feature, and the others are the context feature; 2.2) Decoder: Decoder D includes N _d layers of concatenated decoding layers, each layer contains a multi-head self-attention layer, a multi-head cross-attention layer and a linear mapping layer, self-attention layer, cross-attention layer and linear The mapping layer has a residual structure respectively; for the compressed feature H obtained by the encoder, the decoder performs timing boundary point analysis by transforming the decoder structure, the decoder defines N _p nominated query quantities Q _d , and the nominated query quantities Q _d are The hidden feature query volume is the same, it is randomly initialized and then learned and generated during training, and the boundary nomination B ₀ is initialized to 0. In the jth layer, the nominated query volume Q _d is added to the boundary nomination B _j , after the self-attention layer and a time Residual structure, in the interaction between the cross attention layer and the compressed feature H, after the residual structure-linear mapping layer-residual structure transformation, the updated boundary nomination B _j+1 is obtained; after passing through the stacked N _d decoding layers, Realize the parsing of compressed features and get the nominated representation of timing boundaries

2.3) Generation and scoring of time-series boundary-free boundaries: For the obtained time-series boundary nominated representation B, it is sent to two different fully-connected layer branches: the positioning branch and the classification branch, which are respectively used to output the time-series boundary-free moments. and confidence scores; 2.4) Assign training labels: adopt a strict one-to-one training label matching strategy: according to the defined matching cost C, use the Hungarian algorithm to obtain a set of optimal one-to-one matches, each of which is assigned to a class-free boundary truth value. The predictions all obtain positive sample labels, and the corresponding boundary truth values are the training targets; the matching cost C consists of two parts, the position cost and the classification cost. The position cost is defined based on the absolute value of the distance between the prediction moment and the boundary truth moment, and the classification cost is based on the prediction. Confidence definition; 2.5) Submission of time-series classless boundaries: After generating a series of time-series class-free boundaries, the most credible time-series classless-boundary moments are screened out through the confidence score threshold γ, and submitted for subsequent performance measurement; 3) Training phase: The configured model is trained with training samples, using cross entropy, L1 distance and log function as loss functions, using AdamW optimizer, and updating network parameters through backpropagation algorithm, repeating steps 1) and Step 2), until the number of iterations is reached; 4) Detection: Input the video feature sequence and continuity score of the data to be tested into the trained detection model, generate time-series unclassified boundary moments and scores, and then use the method of 2.3) to obtain the time-series classless time sequence for performance measurement. Boundary time series. 2. timing boundary detection method according to claim 1, is characterized in that when training detection model, align latent feature query volume and video feature: boundary query volume is aligned by the boundary region feature alignment of the alignment loss function and the video feature, and the alignment loss function Based on the calculation of the cross attention map of the last layer, using the condition that the number of boundary queries and the number of features in the boundary area are consistent, the diagonal attention weights formed by the corresponding relationship between the two are merged to take the negative logarithm, and the value of the alignment loss function is obtained. , by minimizing the alignment loss function, to maximize the diagonal weight, and to ensure that the boundary query volume corresponds to the extraction of boundary region features during cross-attention. 3. according to claim 1 and 2 described time series boundary detection methods, it is characterized in that backbone network is based on ResNet50 and time series convolution layer, and video is sampled and generated, for all picture frames corresponding to each video, take τ frame as Interval sampling forms a video image sequence, and the video image sequence is

It is divided into N _f video segments with a length of 2k frames, N _f is the length of the video image sequence, and is also the number of video segments, and the i-th video segment is an image sequence consisting of consecutive k frames before and after the i-th frame image.

The image sequence L _s,i is sent to the backbone network. After pre-training and fine-tuning parameters of the convolution layer, pooling layer and fully connected layer, the output obtains the RGB feature F _i and the continuity score S _i . and scores are spliced together in chronological order to obtain the entire video features.

and continuity score

Among them, the sampling interval τ represents the fine-grained degree of global time division, and the size of the video segment length 2k represents the local receptive field range of the feature. 4. timing boundary detection method according to claim 3, it is characterized in that first utilize denseflow library to carry out frame extraction process to all video and obtain picture frame, take τ frame as interval sampling, τ takes 3, and carry out following processing: call torchvision The transforms package of the library scales each picture frame to a size of 224*224, converts it into a tensor form, and finally normalizes it to obtain a video image sequence L _f ; traverse the image sequence of the entire video, take the k frames before and after each frame, and k takes 5. Obtain N _f video segments, and then use the stacked spatial convolution layer, maximum pooling layer and time series convolution layer to extract the features of each video segment; the features are then scored continuously through the fully connected layer, and the features and The scores are spliced in chronological order, respectively, to obtain the video-level feature F and the continuity score S. 5. the time sequence boundary detection method according to claim 1 and 2, it is characterized in that in the configuration of step 2), the convolution layer of backbone network is made up of convolution operation, batch normalization operation and ReLU activation function, encoder For the transform decoder structure used as the encoder, the decoder is also a transform decoder structure. 6. The time sequence boundary detection method according to claim 1 and 2 is characterized in that the encoder layer number N _e takes 6, and in the jth layer, the compressed feature H _j is used as a value parameter, the hidden feature query quantity Q _e and the compressed feature After H _j is added, it is input into the self-attention module of 8 branches as the key and query parameter. The attention matrix is formed by the self-multiplication of the key and the query parameter and Softmax normalization, and the output is obtained by multiplying it with the value parameter. In the residual structure H j ′ is added with H _j and normalized to obtain H _j ′; H _j ′ and the latent feature query quantity Q _e are added as query parameters, and the video feature F is used as a value parameter after descending reordering based on the continuity score S, It is added to the same reordered position code as the key parameter and input to the cross-attention module. In the module, the key and query parameters are cross-multiplied and Softmax normalized to form an attention matrix and multiplied with the value parameter. The residual structure and H′ _j are added to normalize, linearly map and re-residual structure to obtain the updated compressed feature H _j+1 . 7. The method for detecting time sequence boundary according to claim 1 or 2, characterized in that the decoder layer number N _d takes 6, and at the jth layer, the boundary nominates B _j as a value parameter, and the nominated query quantity Q _d and the boundary nominate B After _j is added, it is input into the self-attention module of 8 branches as the key and query parameter, and the attention matrix is formed by the self-multiplication and Softmax normalization of the key and the query parameter, and the output is obtained by multiplying it with the value parameter. In the residual structure Then add with B _j and normalize to get B _j ′; B _j ′ and the nominated query quantity Q _d are added as query parameters, the compressed feature H obtained in 2.1) is used as the value parameter, and the hidden value of the compressed position encoding is used as the query parameter. The feature query quantity Q _e is added as a key parameter and input to the cross-attention module. In the module, the key and query parameters are cross-multiplied and Softmax normalized to form an attention matrix and multiplied with the value parameter. After the residual structure and B _j ′ is added and normalized to obtain the updated boundary nomination B _j+1 . 8. A timing sensor, characterized in that it has a computer storage medium, and a computer program is configured in the computer storage medium, and the computer program is used to implement the classless timing boundary detection network of claims 1-7, and the computer When the program is executed, the timing boundary detection method described in claims 1-7 is implemented.

Images