CN115098728A - Video retrieval method and device - Google Patents

Abstract

The invention provides a video retrieval method and a video retrieval device, wherein the method comprises the following steps: splitting the long reference video to obtain a plurality of candidate intervals; inputting a query video and the candidate intervals into a video retrieval model to obtain candidate intervals matched with the query video in the candidate intervals; the video retrieval model is trained on the basis of local semantic similarity and global semantic similarity among a plurality of sample candidate intervals obtained by splitting the sample query video and the sample long reference video. The method enables the video retrieval model to simultaneously consider the space-time dimension of the video, calculates and learns the semantic similarity between the videos from the local frame level and the global angle respectively, and greatly improves the accuracy of the positioning of the video retrieval task segments.

Description

Video retrieval method and device

technical field

The present invention relates to the technical field of image processing, in particular to a video retrieval method and device.

Background technique

With the widespread popularity of video sharing nowadays, people increasingly expect to find the video content they are interested in in an efficient and accurate way. In order to meet this demand, research on video retrieval technology was born. Researchers have developed a number of different video retrieval methods with the goal of finding the most relevant videos from the candidate video set that are most relevant to the query information. However, in practical applications, the videos are not directly provided in the form of clips as in the experimental datasets, and long unedited videos can be found everywhere. These videos usually contain complex content, mostly irrelevant to the query information provided, and only a small part satisfy the query description. Therefore, another topic, video moment retrieval, has become an emerging research focus in recent years.

Video Moment Retrieval has the ability to locate the exact start and end points of the video clips relevant to the query from a long, unedited video. Unlike retrieving videos from candidate sets, video moment retrieval provides more fine-grained temporal localization in long videos, avoiding the hassle of manual searching.

Currently, video moment retrieval methods are mainly based on natural language queries, which aim to locate the start and end time points of video clips related to a given natural language query in long videos. However, using text as a query method limits the richness and complexity of the information contained in the query. Video-based video moment retrieval (VQ-VMR) has gradually become an emerging research topic due to the advantages of video in information expression, that is, video contains both visual content of each frame and semantic content between frames.

Given a query video clip, the VQ-VMR task aims to find the video clip semantically corresponding to the query video from a long reference video. In this task, precise retrieval of target moments requires a semantic correlation measure between the query video and reference video clips. Since the information contained in videos is comprehensive and complex, it is difficult to measure the correlation between them. Currently, there is still a huge technical gap in building a framework that can effectively measure the correlation between videos.

SUMMARY OF THE INVENTION

The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention provides a video retrieval method and device.

Specifically, the present invention provides the following technical solutions:

In a first aspect, an embodiment of the present invention provides a video retrieval method, including:

Split the long reference video to obtain multiple candidate intervals;

Input the query video and the plurality of candidate intervals into the video retrieval model, and obtain the candidate intervals that match the query video in the plurality of candidate intervals;

Wherein, the video retrieval model takes multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video as input, and uses the sample candidate intervals obtained by splitting the sample long reference video with the sample The sample candidate interval matched by the query video is obtained after training the initial machine learning model, and the video retrieval model is based on the sample query video and the sample long reference video. The local semantic similarity and global semantic similarity are trained.

Further, input the query video and the multiple candidate intervals into the video retrieval model, and obtain the candidate intervals matching the query video among the multiple candidate intervals, including:

The query video and the plurality of candidate intervals are subjected to video encoding and then input into the video retrieval model to obtain candidate intervals that match the query video in the plurality of candidate intervals;

Wherein, the video coding process of the query video and the multiple candidate intervals includes:

Use the C3D model to perform feature extraction on the query video and the multiple candidate intervals respectively;

The extracted features are aggregated by using two long-term and short-term memory units to obtain the video coding results of the query video and the multiple candidate intervals.

Further, the video retrieval model includes a local semantic similarity module, and the local semantic similarity module is used to perform local semantic similarity processing on the query video and the plurality of candidate intervals;

The local semantic similarity processing process includes:

Using Hadamard product operation to perform feature matching on the query video and the multiple candidate intervals;

Based on the attention mechanism, matched feature representation pairs are enhanced and mismatched feature pairs are suppressed.

Further, the video retrieval model includes a global semantic similarity module, and the global semantic similarity module is used to perform global semantic similarity processing on the query video and the plurality of candidate intervals;

The global semantic similarity processing process includes:

encoding the query video and any candidate interval pairwise embedding into a three-dimensional tensor;

Taking the three-dimensional tensor as input, considering the temporal dimension of the video, the semantic similarity between the query video and the candidate interval is learned from a global perspective.

Further, the global semantic similarity module includes a combination of an attention layer and a convolution layer;

The three-dimensional tensor is input into the global semantic similarity module, and the input three-dimensional tensor is passed through a combination of multiple convolution layers and attention layers, and the convolution kernel in the global semantic similarity module is gradually Find the matching embedding from each layer to get the global matching result;

The global matching result is subjected to global draw pooling to obtain a single vector as the result of the inter-video matching representation.

Further, the training process of the video retrieval model includes:

training the video retrieval model with triple loss and regression loss as loss functions;

Wherein, the convergence direction of the triplet loss includes: for a triplet consisting of a query video, a positive candidate interval sample, and a negative candidate interval sample, assigning a higher matching score to a positive matching video pair, and assigning a higher matching score to a negative of video pairs assigned lower match scores;

The regression loss includes: the predicted relative offset of the center point and the length of the candidate interval and the actual relative offset of the center point and the length of the candidate interval.

In a second aspect, an embodiment of the present invention further provides a video retrieval device, including:

The splitting module is used to split the long reference video to obtain multiple candidate intervals;

a retrieval module, configured to input the query video and the plurality of candidate intervals into a video retrieval model, and obtain a candidate interval that matches the query video in the plurality of candidate intervals;

Wherein, the video retrieval model takes multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video as input, and uses the sample candidate intervals obtained by splitting the sample long reference video with the sample The sample candidate interval matched by the query video is obtained after training the initial machine learning model, and the video retrieval model is based on the sample query video and the sample long reference video. The local semantic similarity and global semantic similarity are trained.

Further, the retrieval module is specifically used for:

The query video and the plurality of candidate intervals are subjected to video encoding and then input into the video retrieval model to obtain candidate intervals that match the query video in the plurality of candidate intervals;

Wherein, the video coding process of the query video and the multiple candidate intervals includes:

Use the C3D model to perform feature extraction on the query video and the multiple candidate intervals respectively;

The extracted features are aggregated by using two long-term and short-term memory units to obtain the video coding results of the query video and the multiple candidate intervals.

In a third aspect, an embodiment of the present invention further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, the processor implementing the first program when executing the program The steps of the video retrieval method described in the aspect.

In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the video retrieval method according to the first aspect.

According to the above technical solutions, the video retrieval method and device provided in the embodiments of the present invention enable the video retrieval model to simultaneously consider the spatiotemporal dimensions of the video, and calculate and learn the semantic similarity between the videos from the local frame level and the global perspective respectively. Improves the accuracy of segment location for video retrieval tasks.

It is to be noted that additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are For some embodiments of the present invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flowchart of a video retrieval method provided by an embodiment of the present invention;

2 is a schematic diagram of an implementation principle of a video retrieval method provided by an embodiment of the present invention;

3 is a schematic diagram of an experimental effect of a video retrieval method provided by an embodiment of the present invention;

4 is a schematic diagram of an implementation process of a video retrieval method provided by an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments These are some embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

FIG. 1 shows a schematic flowchart of a video retrieval method provided by an embodiment of the present invention. Referring to FIG. 1 , the video retrieval method provided by an embodiment of the present invention includes:

Step 101: splitting the long reference video to obtain multiple candidate intervals;

Step 102: input the query video and the plurality of candidate intervals into the video retrieval model, and obtain the candidate intervals that match the query video in the plurality of candidate intervals;

Wherein, the video retrieval model takes multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video as input, and uses the sample candidate intervals obtained by splitting the sample long reference video with the sample The sample candidate interval matched by the query video is obtained after training the initial machine learning model, and the video retrieval model is based on the sample query video and the sample long reference video. The local semantic similarity and global semantic similarity are trained.

According to the above technical solutions, the video retrieval method provided by the embodiment of the present invention enables the video retrieval model to simultaneously consider the spatiotemporal dimensions of the video, and calculate and learn the semantic similarity between the videos from the local frame level and the global perspective respectively, which greatly improves the performance of the video retrieval model. Accuracy of segment localization for video retrieval tasks.

Based on the content of the above embodiment, in this embodiment, the query video and the multiple candidate intervals are input into the video retrieval model, and the candidate intervals matching the query video among the multiple candidate intervals are obtained, including:

The query video and the plurality of candidate intervals are subjected to video encoding and then input into the video retrieval model to obtain candidate intervals that match the query video in the plurality of candidate intervals;

Wherein, the video coding process of the query video and the multiple candidate intervals includes:

Use the C3D model to perform feature extraction on the query video and the multiple candidate intervals respectively;

The extracted features are aggregated by using two long-term and short-term memory units to obtain the video coding results of the query video and the multiple candidate intervals.

Based on the content of the above embodiment, in this embodiment, the video retrieval model includes a local semantic similarity module, and the local semantic similarity module is used to perform local semantic similarity between the query video and the multiple candidate intervals. degree processing;

The local semantic similarity processing process includes:

Using Hadamard product operation to perform feature matching on the query video and the multiple candidate intervals;

Based on the attention mechanism, matched feature representation pairs are enhanced and mismatched feature pairs are suppressed.

Based on the content of the above embodiment, in this embodiment, the video retrieval model includes a global semantic similarity module, and the global semantic similarity module is used to perform global semantic similarity between the query video and the multiple candidate intervals. degree processing;

The global semantic similarity processing process includes:

encoding the query video and any candidate interval pairwise embedding into a three-dimensional tensor;

Taking the three-dimensional tensor as input, considering the temporal dimension of the video, the semantic similarity between the query video and the candidate interval is learned from a global perspective.

Based on the content of the above embodiment, in this embodiment, the global semantic similarity module includes a combination of an attention layer and a convolution layer;

The three-dimensional tensor is input into the global semantic similarity module, and the input three-dimensional tensor is passed through a combination of multiple convolution layers and attention layers, and the convolution kernel in the global semantic similarity module is gradually Find the matching embedding from each layer to get the global matching result;

The global matching result is subjected to global draw pooling to obtain a single vector as the result of the inter-video matching representation.

Based on the content of the above embodiment, in this embodiment, the training process of the video retrieval model includes:

training the video retrieval model with triple loss and regression loss as loss functions;

Wherein, the convergence direction of the triplet loss includes: for a triplet consisting of a query video, a positive candidate interval sample, and a negative candidate interval sample, assigning a higher matching score to a positive matching video pair, and assigning a higher matching score to a negative of video pairs assigned lower match scores;

The regression loss includes: the predicted relative offset of the center point and the length of the candidate interval and the actual relative offset of the center point and the length of the candidate interval.

Considering the deficiencies of the existing VQ-VMR technology in effectively measuring the correlation between videos, the embodiment of the present invention proposes a new framework to measure the semantic correlation between the query video and the reference video segment. By using the convolution structure and attention mechanism, the model of the embodiment of the present invention considers the spatiotemporal dimensions of the video, calculates and learns the semantic similarity between the videos from the local frame level and the global perspective, and greatly improves the VQ-VMR task segment. positioning accuracy. The following describes a video retrieval method provided by an embodiment of the present invention with reference to the schematic diagram shown in FIG. 2 .

Step 1: Video encoding.

查询视频Q与任意一个候选区间p_k作为构建模型的一对输入，分别进行视频编码。视频编码过程分为两个步骤：First, the TAG candidate interval generation algorithm is used to split the long reference video R into N candidate interval segments, denoted as:

The query video Q and any candidate interval p _k are used as a pair of inputs to construct the model, and video coding is performed respectively. The video encoding process is divided into two steps:

①Use the C3D model to extract the features of the query video Q and the candidate interval p _k respectively.

表示候选区间的第j个特征；m，n分别表示查询视频和候选区间提取的特征总数。where qi represents the _ith feature of the query video,

Represents the jth feature of the candidate interval; m, n represent the total number of features extracted from the query video and the candidate interval, respectively.

② Since the above features only consider video features in a short range, two long-short-term memory units (LSTMs) are used in the embodiment of the present invention to further aggregate the extracted features.

记为

For simplicity, in the following content, the embodiments of the present invention will

marked as

Step 2: Calculate the semantic similarity between videos at the local frame level.

利用哈达玛积运算完成特征匹配操作。任意两个视频特征之间，本发明实施例计算二者的相关性并记为γ，公式如下：① Feature matching. Based on the obtained video features

The feature matching operation is completed using the Hadamard product operation. Between any two video features, the embodiment of the present invention calculates the correlation between the two and denote it as γ, and the formula is as follows:

Among them, ⊙ represents the Hadamard product operation; D represents the fully connected layer.

② attention mechanism. An attention mechanism is introduced, aiming to effectively encourage matching feature representation pairs while suppressing mismatched feature pairs. For each pair of video features, the embodiment of the present invention calculates its attention weight α:

α _i,j =σ(W ^T D ₄ (ri _,j ))

是一个可学习参数。具体来说，视频特征之间的相关性越高,得到的权重α值应该更大；相反，特征之间的相关性越低,权重α值则更小。where: σ represents the sigmoid function,

is a learnable parameter. Specifically, the higher the correlation between the video features, the larger the weight α value should be; on the contrary, the lower the correlation between the features, the smaller the weight α value.

Therefore, the correlation between a pair of video features is finally expressed as: t _i,j =αi _,j γi _,j

③ three-dimensional tensor. This embodiment of the present invention encodes all pairwise embeddings between two video sequences into a three-dimensional tensor, denoted as T:

Step 3: Calculation of global semantic similarity between videos.

Taking the obtained 3D tensor T as input, the semantic similarity between the query video and the candidate interval is learned from a global perspective. The matching results obtained in the previous section only calculate the relationship between the query video and the candidate interval at the local frame level, without considering the temporal dimension of the video. In the VQ-VMR task, matched videos should share semantically similar frames in contiguous time frames. Multiple isolated high matching points cannot represent a high degree of matching between the videos as a whole. Therefore, in this section, embodiments of the present invention introduce another attention mechanism to eliminate isolated high matching values to obtain better correlation metrics.

The embodiment of the present invention proposes a special convolutional layer called an attention layer. It is a single-channel convolutional layer, denoted by Att(T). The embodiment of the present invention uses a combination of convolutional layer and attention layer, and expresses it as:

Conv(T) σ(ATT(T))

where T represents the input tensor. In the combination of embodiments of the present invention, the width and height of the attention layer are the same as that of the previous convolutional layer. After the attention layer passes the sigmoid function, it is multiplied by the previous convolutional layer (as shown in the formula), which is equivalent to assigning the corresponding attention weight to each position of the previous convolutional layer.

First, the embodiment of the present invention transmits the input tensor T through a combination of multiple convolution layers and attention layers, and its learnable convolution kernel can gradually find matching embeddings from each layer. Specifically, the embodiment of the present invention applies three groups of combinations, which are expressed as:

T ^k =Conv(k)(T ^k-1 )·σ(Att(k)(T ^k-1 ))

where k can be 1, 2, 3, T ⁰ is initialized as T obtained from the previous section, Conv(k) represents the kth convolutional layer, and Att(k) represents the kth attention layer.

In the embodiment of the present invention, ^Tk is globally pooled to obtain a single vector as the result of the matching representation between videos, which is denoted as T _out .

Step 4: Similarity score and location information regression.

Taking the resulting vector T _out as input, the regression network has two sibling output layers. The first output is the semantic similarity score between the query video and the candidate interval and denoted as S. The embodiment of the present invention uses tanh as the activation function, so that the final score value S generated by the network is in the range of [-1, 1].

The second output gets the candidate interval position regression offset. In the embodiment of the present invention, the position regression offset is designed as (T _c , T _l ), where T _c and T _l represent the offset of the center point of the candidate interval and the offset of the length, respectively. Since the candidate interval may be too long or too short compared to the true interval, the regression process helps to find better location information.

Step 5: Training.

The proposed model structure contains two sibling outputs, one for correlation measure and the other for boundary regression. The embodiment of the present invention designs a multi-task loss L to jointly train the model of the embodiment of the present invention. The loss function is specifically expressed as:

L=L _tr + μ·L _reg

where L _tr represents the triple loss; L _reg represents the regression loss; μ represents the weight value that measures the contribution of the two losses.

① The triplet loss L _tr . The training samples applied to triplet loss include positive candidate interval samples and negative candidate interval samples. In the embodiment of the present invention, the training data set is reorganized into video triples (q, p, n), where q, p, n represent query video, positive candidate interval samples and negative candidate interval samples, respectively. In the embodiment of the present invention, a video segment is randomly selected as a query video from the training set, and a candidate interval obtained from a video with the same label as the query video is used as a positive sample. Similarly, suggestions obtained from videos with different labels from the query video are called negative samples.

The role of _Ltr is to assign a higher matching score to a positive matching video pair (q, p) and a lower matching score to a negative video pair (q, n).

where γ is a marginal parameter to ensure a sufficiently large difference between the matching score of the positive sample pair (q, p) and the matching score of the negative sample pair (q, n); λ is a regularization parameter that prevents Overfitting of the model; N is the number of triplets contained in a batch of samples.

② regression loss L _reg

Use only positive candidate interval samples to train the regression loss. The regression loss _Lreg is expressed as:

和

为候选区间中心点和长度的真实相对偏移量。Among them, L ₁ is the smooth L ₁ function, T _c and T _l are the predicted relative offsets of the center point and length of the candidate interval, respectively,

and

is the true relative offset of the center point and length of the candidate interval.

和

表示对应真实的中心坐标和长度。Among them, loc _i and len _i respectively represent the center coordinate and length of the ith candidate interval;

and

Indicates the corresponding real center coordinates and length.

Step 6: Predict.

After the training is completed, the embodiment of the present invention takes the query video and the candidate interval obtained from the reference video as input, and uses the proposed framework to predict the position of the retrieved segment. The embodiment of the present invention selects the candidate interval with the highest similarity score, which is specifically expressed as:

i=arg maxS(q, p _i )

where i represents the ith candidate interval.

After obtaining the candidate interval with the highest matching score, the embodiment of the present invention refines the boundary of the selected candidate interval according to the regression offset, and the specific formula is as follows:

Among them, T _c (i) and T _l (i) represent the center point and length offset of the selected i-th candidate interval, respectively. s, e represent the start and end points of the final positioning segment, respectively.

Step 7: Verify.

The following will verify the patented video clip-based video moment retrieval technology. For the generated three-dimensional tensor T, the embodiment of the present invention hopes that its width and height can be kept consistent, so that it can be used as the input of the subsequent convolution layer for further correlation learning. However, the video lengths of the query video and the candidate interval may be different, resulting in different numbers of extracted features. To solve this problem, the embodiment of the present invention limits the maximum number of features of a video to 40. If the video is too long and the number of features is greater than 40, the embodiment of the present invention selects 40 equidistant features.

The experiments use the Adam optimizer to train the model, the initial learning rate is set to 10 ⁻⁴ , and the batch size is set to 32. For the regularization operation, the embodiment of the present invention applies a BN (batch normalization) operation to each fully connected layer. In the embodiment of the present invention, the triple loss and the regression loss have the same contribution, so the weight μ in the multi-task loss L is set to 0.5. The sizes of the fully connected layers D ₁ , D ₂ , D ₃ , and D ₄ in the model are all set to 512. The setting details of different convolutional layers and attention layers in the model are shown in Table 1. For the parameters in the formula, λ=0.0005 and γ=1.5 are respectively set in the embodiment of the present invention.

The embodiments of the present invention mainly use videos in the Thumos14 and ActivityNet datasets as training and test samples to evaluate the models proposed in the embodiments of the present invention. Tables 2 and 3 respectively show the quantitative comparison results between the method of the embodiment of the present invention and other methods on the Thumos14 and ActivityNet datasets. Figure 3 shows the qualitative results of the method of an embodiment of the present invention. It can be seen from the results that the results of the embodiments of the present invention have better positioning accuracy compared with other methods. Compared with other methods, the models proposed in the embodiments of the present invention have higher adaptability under different data sets, and can perform retrieval tasks well on both data sets.

Table 1 Convolutional layer and attention layer setting details

name convolution kernel step size number of channels Conv1 3*3 1 256 Att1 3*3 1 1 Conv2 3*3 1 256 Att2 3*3 1 1 Conv3 3*3 2 256 Att3 3*3 2 1 Average Pooling 17*17 17 256

Table 2 Comparison results of objective indicator mAP under different tIoU thresholds in Thumos14 dataset

Table 3 Comparison results of objective indicator mAP under different tIoU thresholds in ActivityNet dataset

It can be seen from FIG. 4 that an embodiment of the present invention proposes a new framework to measure the semantic correlation between a query video and a reference video segment. By using the convolution structure and attention mechanism, the model of the embodiment of the present invention considers the spatiotemporal dimensions of the video, calculates and learns the semantic similarity between the videos from the local frame level and the global perspective, and greatly improves the VQ-VMR task segment. positioning accuracy. The video moment retrieval algorithm proposed in the embodiment of the present invention utilizes the convolution structure and the attention mechanism, and simultaneously considers the spatiotemporal dimension of the video, and calculates and learns the semantic similarity between the videos from the local frame level and the global perspective respectively. Effectively improve the accuracy of VQ-VMR task segment location.

Based on the same inventive concept, another embodiment of the present invention provides a video retrieval device, including:

The splitting module is used to split the long reference video to obtain multiple candidate intervals;

a retrieval module, configured to input the query video and the plurality of candidate intervals into a video retrieval model, and obtain a candidate interval that matches the query video in the plurality of candidate intervals;

Wherein, the video retrieval model takes multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video as input, and uses the sample candidate intervals obtained by splitting the sample long reference video with the sample The sample candidate interval matched by the query video is obtained after training the initial machine learning model, and the video retrieval model is based on the sample query video and the sample long reference video. The local semantic similarity and global semantic similarity are trained.

Based on the content of the foregoing embodiment, in this embodiment, the retrieval module is specifically used for:

The query video and the plurality of candidate intervals are subjected to video encoding and then input into the video retrieval model to obtain candidate intervals that match the query video in the plurality of candidate intervals;

Wherein, the video coding process of the query video and the multiple candidate intervals includes:

Use the C3D model to perform feature extraction on the query video and the multiple candidate intervals respectively;

The extracted features are aggregated by using two long-term and short-term memory units to obtain the video coding results of the query video and the multiple candidate intervals.

Since the video retrieval apparatus provided in this embodiment can be used to execute the video retrieval method described in the foregoing embodiment, its working principle and beneficial effects are similar, and the specific content can refer to the foregoing embodiment, which will not be repeated here.

Based on the same inventive concept, another embodiment of the present invention provides an electronic device, see FIG. 5 , the electronic device specifically includes the following: a processor 301, a memory 302, a communication interface 303, and a communication bus 304;

Wherein, the processor 301, the memory 302, and the communication interface 303 complete the communication with each other through the communication bus 304; the communication interface 303 is used to realize the transmission between related devices; the processor 301 is used to call The computer program in the memory 302, when the processor executes the computer program, implements all the steps of the above-mentioned video retrieval method, for example, when the processor executes the computer program, implements the following steps: performing a long reference video Splitting to obtain a plurality of candidate intervals; inputting the query video and the plurality of candidate intervals into a video retrieval model to obtain candidate intervals that match the query video in the plurality of candidate intervals; wherein, the video retrieval model It is to take as input multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video, and split the sample long reference video to obtain the sample candidate interval that matches the sample query video. As an output, obtained after training the initial machine learning model, the video retrieval model is based on the local semantic similarity and global semantic similarity between multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video Semantic similarity for training.

Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, all steps of the above video retrieval method are implemented , for example, the processor implements the following steps when executing the computer program: splitting the long reference video to obtain multiple candidate intervals; inputting the query video and the multiple candidate intervals into the video retrieval model to obtain the The candidate interval that matches the query video among the plurality of candidate intervals; wherein, the video retrieval model takes as input a plurality of sample candidate intervals obtained by splitting the sample query video and the sample long reference video, and uses the sample long reference video as input. The sample candidate interval that matches the sample query video among the multiple sample candidate intervals obtained by reference video splitting is used as the output, and obtained after training the initial machine learning model, and the video retrieval model is based on the sample query video and the sample query video. The local semantic similarity and global semantic similarity between multiple sample candidate intervals obtained by splitting the sample long reference video are used for training.

In addition, the above-mentioned logic instructions in the memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solutions of the embodiments of the present invention. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the video retrieval method described in each embodiment or some part of the embodiment.

In the present invention, terms such as "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

Furthermore, in the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply existence between these entities or operations any such actual relationship or sequence. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

In addition, in the description of this specification, reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" and the like means description in conjunction with the embodiment or example. A particular feature, structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. a video retrieval method, is characterized in that, comprises: Split the long reference video to obtain multiple candidate intervals; Input the query video and the plurality of candidate intervals into the video retrieval model, and obtain the candidate intervals that match the query video in the plurality of candidate intervals; Wherein, the video retrieval model takes multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video as input, and uses the sample candidate intervals obtained by splitting the sample long reference video with the sample The sample candidate interval matched by the query video is obtained after training the initial machine learning model, and the video retrieval model is based on the sample query video and the sample long reference video. The local semantic similarity and global semantic similarity are trained. 2 . The video retrieval method according to claim 1 , wherein the query video and the multiple candidate intervals are input into a video retrieval model to obtain candidates matching the query video in the multiple candidate intervals. 3 . interval, including: The query video and the plurality of candidate intervals are subjected to video encoding and then input into the video retrieval model to obtain candidate intervals that match the query video in the plurality of candidate intervals; Wherein, the video coding process of the query video and the multiple candidate intervals includes: Use the C3D model to perform feature extraction on the query video and the multiple candidate intervals respectively; The extracted features are aggregated by using two long-term and short-term memory units to obtain the video coding results of the query video and the multiple candidate intervals. 3. The video retrieval method according to claim 1, wherein the video retrieval model comprises a local semantic similarity module, and the local semantic similarity module is used to compare the query video and the multiple candidate intervals Perform local semantic similarity processing; The local semantic similarity processing process includes: Using Hadamard product operation to perform feature matching on the query video and the multiple candidate intervals; Based on the attention mechanism, matched feature representation pairs are enhanced and mismatched feature pairs are suppressed. 4. The video retrieval method according to claim 1, wherein the video retrieval model comprises a global semantic similarity module, and the global semantic similarity module is used to compare the query video and the multiple candidate intervals Perform global semantic similarity processing; The global semantic similarity processing process includes: encoding the query video and any candidate interval pairwise embedding into a three-dimensional tensor; Taking the three-dimensional tensor as input, considering the temporal dimension of the video, the semantic similarity between the query video and the candidate interval is learned from a global perspective. 5. The video retrieval method according to claim 4, wherein the global semantic similarity module comprises a combination of an attention layer and a convolution layer; The three-dimensional tensor is input into the global semantic similarity module, and the input three-dimensional tensor is passed through a combination of multiple convolution layers and attention layers, and the convolution kernel in the global semantic similarity module is gradually Find the matching embedding from each layer to get the global matching result; The global matching result is subjected to global draw pooling to obtain a single vector as the result of the inter-video matching representation. 6. The video retrieval method according to claim 1, wherein the training process of the video retrieval model comprises: training the video retrieval model with triple loss and regression loss as loss functions; Wherein, the convergence direction of the triplet loss includes: for a triplet consisting of a query video, a positive candidate interval sample, and a negative candidate interval sample, assigning a higher matching score to a positive matching video pair, and assigning a higher matching score to a negative of video pairs assigned lower match scores; The regression loss includes: the predicted relative offset of the center point and the length of the candidate interval and the actual relative offset of the center point and the length of the candidate interval. 7. A video retrieval device, comprising: The splitting module is used to split the long reference video to obtain multiple candidate intervals; a retrieval module, configured to input the query video and the plurality of candidate intervals into a video retrieval model, and obtain a candidate interval that matches the query video in the plurality of candidate intervals; Wherein, the video retrieval model takes multiple sample candidate intervals obtained by splitting the sample query video and the sample long reference video as input, and uses the sample candidate intervals obtained by splitting the sample long reference video with the sample The sample candidate interval matched by the query video is obtained after training the initial machine learning model, and the video retrieval model is based on the sample query video and the sample long reference video. The local semantic similarity and global semantic similarity are trained. 8. The video retrieval device according to claim 7, wherein the retrieval module is specifically used for: The query video and the plurality of candidate intervals are subjected to video encoding and then input into the video retrieval model to obtain candidate intervals that match the query video in the plurality of candidate intervals; Wherein, the video coding process of the query video and the multiple candidate intervals includes: Use the C3D model to perform feature extraction on the query video and the multiple candidate intervals respectively; The extracted features are aggregated by using two long-term and short-term memory units to obtain the video coding results of the query video and the multiple candidate intervals. 9. An electronic device, comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements any one of claims 1 to 6 when the processor executes the program The steps of the video retrieval method described in item. 10. A computer-readable storage medium on which a computer program is stored, characterized in that, when the computer program is executed by a processor, the steps of the video retrieval method according to any one of claims 1 to 6 are implemented.

Images