CN118113888A - A cross-modal fine-grained retrieval method based on multi-channel fusion - Google Patents

Abstract

The present invention proposes a cross-modal fine-grained retrieval method based on multi-channel fusion. On the one hand, this method uses a branch network to extract the deep feature information of four modalities. This method can greatly extract the features that are specific to each modality and make full use of the feature information of each modality. On the other hand, after the deep feature information of each modality is extracted, it is divided into four channels and then reorganized, so that each group of reorganized information contains the deep feature information of the four modalities. In this way, when the model is learned, it can not only learn the information of this modality, but also learn the information brought by other modalities in advance, which greatly enhances the information interaction ability between each modality, thereby enhancing the classification ability of the model, providing more accurate classification results for subsequent retrieval tasks, and further improving the cross-modal retrieval ability of the model. This technology can be applied to search engines or public security systems to effectively improve the retrieval accuracy and crime detection efficiency.

Description

A cross-modal fine-grained retrieval method based on multi-channel fusion

Technical Field

The present invention belongs to the field of artificial intelligence, and specifically relates to technologies such as deep learning, cross-modal retrieval, fine-grained retrieval, and channel fusion, and in particular to a cross-modal fine-grained retrieval method based on multi-channel fusion.

Background technique

With the development of social science and technology, the four channels of video, image, text and audio have become the main forms for humans to understand the world and communicate with each other. The rapid growth of multimodal data has brought about a large number of application requirements for cross-modal retrieval, whose goal is to achieve mutual retrieval of cross-modal content. Compared with single-modal retrieval, cross-modal retrieval has higher technical challenges due to the large changes between modalities. However, current cross-modal retrieval tasks are usually focused on coarse granularity, which is far from meeting the needs of practical applications. In contrast, fine-grained retrieval has greater application needs and research value, both in industry and academia. Therefore, cross-modal fine-grained retrieval has become an important research direction, and more cross-modal fine-grained retrieval theories and technologies need to be developed.

Existing cross-modal retrieval methods mainly focus on the modality of image-text pairs, while there are few studies on the four modalities of image, video, audio and text. Multimodal retrieval means that the user gives a sample of any modality as a query sample, and the system retrieves and feedbacks samples of each modality that belong to the same category as the query sample. Multimodal retrieval generally requires that the number of modalities is greater than or equal to two, which also brings great difficulty to the experiment. There are generally two methods in the traditional cross-modal fine-grained retrieval model based on deep learning. One is to use different neural network models to extract feature vectors for different modalities, such as using an image feature extractor and a linear classifier to predict some labels, and jointly train an image encoder and a text encoder to predict the correct pairing of a batch of (image, text) training samples. At test time, the learned text encoder synthesizes a zero-shot linear classifier by embedding the name or description of the class of the target dataset. Another method is to use a backbone network to extract feature vectors of different modalities at the same time, such as using ResNet50 as the basic deep model, with 448*448 as the input size, and after the last convolutional layer, passing through an average pooling layer with a kernel size of 14 and a step of 1. The purpose is to use one network to extract features of all modalities. Some methods propose a dual-path attention model for feature learning, which integrates deep convolutional neural networks, attention mechanisms, and recurrent neural networks to learn cross-modal fine-grained salient features and fully explore the fine-grained semantic correlation between different modal data. Other methods apply strip pooling to image and text modalities, which is a lightweight spatial attention mechanism for capturing modal space semantic information, while exploring second-order covariance pooling to obtain multimodal semantic representations, capture modal channel semantic information, and achieve semantic alignment between image and text modalities. There are also methods that extract salient image region features and word features with context, respectively, construct fine-grained similarity matrices between image regions and text words, as well as image latent spaces and text latent spaces based on attention mechanisms, so as to perform semantic supervision on image region features and corresponding image context features. The method of using a backbone network to extract features focuses on extracting the connections and commonalities between modalities, but the connections and commonalities are only a small part of all the data, resulting in a large loss of effective modal-specific information. The method of using a branch network to extract features of each modality focuses on utilizing modal-specific information, but it is difficult to extract the connections between modalities and the commonalities of samples of different modalities. In addition, most studies focus on finding the connections between images and texts, while ignoring the large amount of information contained in videos and audios.

Summary of the invention

The purpose of the present invention is to solve the problems of heterogeneous gap and semantic gap between various modalities in existing methods, and to provide a cross-modal fine-grained retrieval network based on multi-channel fusion.

To achieve the above purpose, the technical solution adopted by the present invention is as follows:

1) The image modality data is cropped according to the existing bounding box labels, the key frames are extracted from the video modality data by extracting one frame every ten frames, and the audio modality data is converted into a spectrogram using short-time Fourier transform.

2) The image modality, video modality and audio modality data that have been converted into pictures are used to extract features using the ResNet50 network, and finally the modality-specific features of the three modalities are obtained, Res(f _I ), Res(f _V ), and Res(f _A ).

3) Use TextCNN to extract the features of text modality data, and finally obtain the modality-specific features Text(f _T ) of the text modality.

4) Divide the modal specific characteristics of each mode into four equal parts.

5) Use the multi-channel fusion method to cross-channel fuse the four features Res(f _I ), Res(f _V ), Res(f _A ), and Text(f _T ) to obtain new modality-specific features f' _I , f' _V , f' _A , and f' _T .

6) Input the new modality-specific features into the linear classifier for classification, and use the cross entropy loss function and noise contrast estimation loss function to optimize the model effect.

7) Enter any category of any modality in the classified results, and retrieve other modal samples of the same category based on the classification to achieve cross-modal fine-grained retrieval.

Specifically, the step 1) includes:

For image data, since we only need the part of the whole image related to the retrieval object, in order to eliminate the interference of background noise, we crop the image. Cropping is performed according to the pixel coordinates of the retrieval object to obtain the part containing only the object.

For video data, we extract ten frames of images from each video as input samples of the video modality.

For audio data, short-time Fourier transform is used to convert it into the corresponding spectrogram as the input sample of the audio modality.

Further, step 3) includes:

First, the local features of the text are obtained, and the N-Gram information of the text is extracted through different convolution kernel sizes. Then, the maximum pooling operation is used to highlight the most critical information extracted by each convolution operation. After splicing, the features are combined through the fully connected layer, and finally the model is trained through the cross entropy loss function.

Further, step 5) comprises:

Generally speaking, the biggest challenge of cross-modal fine-grained retrieval lies in the heterogeneous gap and semantic gap between modalities. Therefore, the present invention uses the channel fusion method to fuse and reorganize the deep features of the four modalities during the training process.

In order to realize the cross-modal retrieval method of multi-channel fusion, the first part of the image is replaced with the first part of the video modality, the second part of the image modality is replaced with the second part of the audio modality, the third part of the image modality is replaced with the third part of the text modality, and the fourth part of the image modality and the rest of the other modalities remain unchanged.

Further, step 6) includes:

Since the problem addressed by the present invention is fine-grained cross-modal retrieval of 200 different bird species belonging to the same large category of birds, the classification task is very arduous, and therefore it is necessary to use a noise contrast estimation loss function to optimize the model.

The cross-modal fine-grained retriever classifies the features obtained in step 5) and obtains the category prediction loss:

Among them, l(x _k ,y _k ) is the cross entropy loss function, I, V, A, T represent image modality, video modality, audio modality and text modality.

Since the retrieval task of the present invention is to classify first, then cluster, and finally perform the retrieval task, in order to reduce the huge amount of calculation caused by 200 classifications, the present invention uses the noise contrast estimation loss function to reduce the amount of calculation:

G(x,y)＝F(x,y)-logQ(y|x)

Where F(x,y) represents the match between x and y, that is, the output of the model, x and y both represent correctly predicted instances, and y' represents the negative example of x, which is taken from the entire set y. Therefore, the total loss function will be defined as L _total = αL _cls + βL _NCE .

After the above steps 1) to 7), for a use case of a certain category of a certain modality input by the user, the system will retrieve and feedback use cases of other modalities belonging to the same category as the use case.

The beneficial effects of the present invention are as follows:

1) The input of the image modality is cropped according to the bounding box to eliminate the interference of background noise.

2) Convert image, video and audio modal data information into pictures, making network processing more convenient and the structure more unified.

3) Introduce the cross entropy loss function and the noise contrast estimation loss function to guide the effective update of the network structure parameters. The use of the noise contrast estimation loss function can effectively reduce the amount of calculation and improve efficiency.

4) Through multi-channel fusion technology, the information interaction between modalities is strengthened, the heterogeneous gap and semantic gap between modalities are reduced, and the model can learn richer information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic flow chart of the method of the present invention.

FIG. 2 is a schematic diagram of a model framework of the present invention.

FIG3 is a data set used in the experiment of the present invention.

FIG4 is a performance comparison chart of the present invention and other methods in one-to-one modal retrieval on the PKU FGXmedia dataset.

FIG5 is a performance comparison chart of the present invention and other methods in single-to-multimodal retrieval on the PKU FGXmedia dataset.

FIG6 is a performance comparison diagram of different variants of the present invention in single-to-single modality retrieval.

FIG. 7 is a performance comparison diagram of different variants of the present invention in single-to-multimodal retrieval.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The present invention proposes a cross-modal fine-grained retrieval network based on single-modal guidance and multi-channel fusion. The main process of the method is shown in Figure 1. The specific implementation process is as follows:

1) The image modality data is cropped according to the existing bounding box labels, the key frames are extracted from the video modality data by extracting one frame every ten frames, and the audio modality data is converted into a spectrogram using short-time Fourier transform.

2) The image modality, video modality and audio modality data that have been converted into pictures are used to extract features using the ResNet50 network, and finally the modality-specific features of the three modalities are obtained, Res(f _I ), Res(f _V ), and Res(f _A ).

3) Use TextCNN to extract the features of text modality data, and finally obtain the modality-specific features Text(f _T ) of the text modality.

4) Divide the modal specific features of each mode into four channels.

5) Use the multi-channel fusion method to cross-channel fuse the four features Res(f _I ), Res(f _V ), Res(f _A ), and Text(f _T ) to obtain new modality-specific features f' _I , f' _V , f' _A , and f' _T .

6) Input the new modality-specific features into the linear classifier for classification, and use the cross entropy loss function, noise contrast estimation loss function and fine-grained cross-modal center loss to optimize the model effect.

7) Enter any category of any modality in the classified results, and retrieve other modal samples of the same category based on the classification to achieve cross-modal fine-grained retrieval.

Specifically, the step 2) includes:

Compared with the CNN network, we use the more advanced ResNet50 network to process image input. The main feature of ResNet50 is its depth. It has 50 layers of convolutional neural networks. The depth of this model enables it to learn more complex features, thereby improving its accuracy. However, one problem with deep learning models is the disappearance of gradients, which can cause the model to be unable to train. To solve this problem, ResNet50 uses the residual learning method. The idea of residual learning is that if the input and output of a layer are the same, then the layer is an identity mapping. If the input and output of a layer are different, then the layer is a residual mapping. ResNet50 uses residual blocks to implement residual learning. Each residual block contains two convolutional layers and a skip connection. The skip connection passes the input directly to the output, thereby avoiding the problem of gradient disappearance. Another feature of ResNet50 is that it uses a global average pooling layer. This layer takes the average of all pixels of each feature map as the output of the feature map. The role of this layer is to reduce the number of parameters of the model, thereby reducing the risk of overfitting. In general, ResNet50 is a very powerful deep learning model. Its depth and residual learning methods enable it to learn more complex features, thereby improving its accuracy. Its global average pooling layer can reduce the number of parameters of the model, thereby reducing the risk of overfitting. ResNet50 has been widely used in the field of computer vision, such as image classification, object detection, and semantic segmentation.

Further, step 5) comprises:

There are four known modalities M = {I, V, A, T}, where I represents image modality, V represents video modality, A represents audio modality, and T represents text modality. Assume that the category space is K = {k ₁ , k ₂ , k ₃ , ... k _n }, and the sample space is Where/> is a set of instances, indicating that the modality M belongs to the category k _i . During training, we randomly select a category k _m , and for each modality in M we Randomly select an instance to build a multimodal dataset of image-video-audio-text/> Where/> is a randomly selected image sample from category k _m , /> is a randomly selected video sample from category k _m , /> is an audio sample randomly selected from category k _m , /> is a randomly selected image sample from category k _m , that is, /> We will then As the input of the network, the image, video, and audio modalities are passed through the Vision Transformer (ViT) backbone network, and the text modality is extracted through the Bert backbone network to obtain deep feature information/> in

Wherein d is the characteristic size divided by the present invention. In the present invention, the characteristic sizes of each mode are the same, that is, d ^I =d ^V =d ^A =d ^T =d.

In order to realize the cross-modal retrieval method of multi-channel fusion, we first represent the extracted feature information as Then the feature information of each modality is divided into four parts, taking the image modality as an example

The other three modalities are processed in the same way, where L ₁ , L ₂ , L ₃ , L ₄ represent the length of each part, and L ₁ + L ₂ + L ₃ + L ₄ = d. Then we perform a multi-channel feature fusion operation, and the fused image-video-audio-text quad-modal deep feature information is represented as

In order to achieve better results for the model, a Squeeze-and-Excitation Network (SENet) is applied to the mixed deep feature information, so that the effective feature map channels have large weights, and the invalid or ineffective feature map channels have small weights. Finally, a new image-video-audio-text quad-modal deep feature information is obtained. This is fed into a linear classifier as input to get the final prediction.

After the above steps 1) to 7), for a use case of a certain category of a certain modality input by the user, the system will retrieve and feedback use cases of other modalities belonging to the same category as the use case.

Experimental verification

1. Dataset

The data set used in the present invention is the only public data set FG-Xmedia for fine-grained retrieval across four modalities. The data set contains data in four modalities: image, video, audio, and text. The image data comes from CUB-200-2011, which is the most widely used fine-grained image classification data set. It contains 11,788 images of 200 subcategories, belonging to the same basic coarse-grained category as "Bird", wherein the training set contains 5,994 images, and the test set contains 5,794 images. Each image annotation contains an image-level subcategory label, the bounding box of the object, 15 partial positions, and 312 binary attributes. The video source is YouTube Birds, which is a new fine-grained video data set. The same category division method as CUB-200-2011 is used, wherein the training set includes 12,666 videos, and the test set includes 5,684 videos. The audio and text data are obtained from professional websites according to the divided categories. The above four modalities together constitute the public data set FG-Xmedia.

2. Implementation details

The present invention mainly uses two backbone networks. Since the inputs of image, video and audio modalities are converted into images in the data preprocessing stage, they jointly use ResNet50 as the backbone network. TextCNN is used to extract features for the text modality. After preprocessing of the data of the four modalities, the dimension of the data samples is fixed to 448×448×3. The entire program code is written in Pytorch, and the graphics card is RTX 3090. During training, we set the batchsize to 8, the initial learning rate to 0.00005, select AdamW as the optimizer, and adjust the learning rate to the cosine learning rate with 1000 steps of warm-up.

3. Comparative experiment

The present invention compares the proposed method with some representative models and names the proposed method MCF.

(1) ACMR: A novel Adversarial Cross-Modal Retrieval (ACMR) method that seeks an effective common subspace based on adversarial learning.

(2) CMDN: A cross-media multi-deep network that exploits complex cross-media associations through hierarchical learning, learns rich cross-media correlations in two stages, and finally obtains shared representations through a stacked network working approach.

(3) JRL: A novel feature learning algorithm for cross-media data, namely Joint Representation Learning (JRL), can jointly explore relevant information and semantic information under a unified optimization framework.

(4) GSPH: A simple yet effective general hashing framework that can be applied to all different situations while preserving the semantic distance between data points. This method first simultaneously learns the optimal hash codes of the two modalities to preserve the semantic similarity between data points, and then learns a hash function to map features to hash codes.

(5) MHTN: A modality-backward hybrid transfer network (MHTN) aims to achieve knowledge transfer from a unimodal source domain to a cross-modal target domain and learn cross-modal joint representations.

(6) FGCrossNet: A unified deep model that learns four types of media simultaneously without processing them separately. Three constraints are considered together: the classification constraint ensures the learning of discriminative features of fine-grained subcategories, the center constraint ensures the compactness of features of the same subcategory, and the sorting constraint ensures the sparsity of features of different subcategories.

(7) DBFC-Net: A novel dual-branch fine-grained cross-media network (DBFC-Net) that exploits media-specific information to build common features through a unified framework. It also designs an effective distance metric for fine-grained cross-media retrieval.

(8) SAFGCM: An attention space training method to learn the common representation of different media data. Specifically, a local self-attention layer is used to learn the common attention space between different media data.

For fair comparison, this paper uses a unified evaluation index for multimodal fine-grained retrieval tasks, namely mean average precision (mAP). In this experiment, mean average precision refers to first calculating the average accuracy of query samples in each category, and then calculating the average of the average accuracy of 200 categories to obtain mAP.

At the same time, in order to more comprehensively demonstrate the effect of cross-modal fine-grained retrieval, we set a total of 18 evaluation indicators, including the individual retrieval scores of each modality for the other three modalities, respectively expressed as I→V, I→A, I→T, V→I, V→A, V→T, A→I, A→V, A→T, T→I, T→V, T→A, and the average of the above scores. It also includes the common retrieval scores of each modality for all other modalities, namely I→ALL, V→ALL, A→ALL, T→ALL, and the average of the above scores. Observing the data in Figure 5, it can be found that our algorithm has significantly improved performance compared with other methods, and the improvement is more than 110% compared with the FGCrossNet algorithm in the I→T, T→I, T→A, T→V, A→T, V→T, and T→ALL scenarios. Overall, the average retrieval mAP value of one-to-one modal retrieval has increased by 52% compared with the FGCrossNet algorithm, and the average retrieval mAP value of one-to-multimodal retrieval has increased by 33% compared with the FGCrossNet algorithm. This proves the effectiveness of MCF in cross-modal fine-grained retrieval, which can fully integrate information from different modalities, enhance the information interaction of fine-grained objects in different modalities, reduce the differences between different modalities, make full use of the rich semantic information of the text modality, reduce information loss, etc., thereby better improving the retrieval effect.

4. Ablation experiment

Since the MCF proposed in the present invention contains multiple key components, in this section, the present invention will compare the variants of MCF from the following aspects to demonstrate the effectiveness of MCF:

(1) ResNet50: After extracting features using the ResNet network, only the cross entropy loss function and the noise contrast estimation loss function are used to optimize the model.

(2) ResNet+MCF: After using the convolutional neural network used by most networks to extract features, a multi-channel fusion module is added for subsequent processing.

(3) ResNet+MCF+FCCL: It uses the most advanced image feature extractor and text feature extractor to extract features and then directly performs classification. At the same time, it adds single-modal guidance and multi-channel fusion modules to improve the model effect.

Figure 5 shows the effects of various variants of MCF on the PKU FG-Xmedia dataset.

It is not difficult to see from Figure 5 that the performance of ResNet+MCF+FCCL is significantly better than ResNet+MCF and ResNet50, which shows the effectiveness of each submodule on the overall MCF model. In addition, it can be observed that among the three variants, the performance of removing MCF drops most significantly, which shows that channel fusion has the greatest impact on the performance of the MCF model. The use of multi-channel fusion technology can greatly enhance the information interaction between modalities, eliminate the heterogeneous gap and semantic gap, and improve retrieval performance.

Claims (6)

1. A cross-mode fine granularity retrieval method based on multi-channel fusion is characterized in that: the method comprises the following steps: 1) Clipping the image mode data according to the existing boundary frame label, extracting a key frame from the video mode data according to a mode of extracting one frame every ten frames, and converting the audio mode data into a spectrogram by using short-time Fourier transform; 2) Extracting features from the data converted into image mode, video mode and audio mode of the picture by ResNet network to obtain mode specific features of three modes finally, res (f _I),Res(f_V),Res(f_A); 3) Extracting characteristics of the Text mode data by TextCNN to finally obtain mode specific characteristics Text (f _T) of the Text mode; 4) Dividing the mode specific characteristics of each mode into four parts; 5) Performing cross-channel fusion on four features of Res (f _I),Res(f_V),Res(f_A),Text(f_T) by using a multi-channel fusion method to obtain a new mode specific feature f' _I,f'_V,f'_A,f'_T; 6) Inputting the new modal specific features into a linear classifier for classification, and optimizing the model effect by using a cross entropy loss function and a noise contrast estimated loss function; 7) Inputting any type of any mode in the classified results, and searching other mode samples of the same type according to the classification to realize cross-mode fine granularity searching.

2. The cross-modal fine-grained retrieval method based on multi-channel fusion according to claim 1, wherein the step 1) comprises: the input data is clipped by using the existing bounding box labels of the image mode, the interference of background noise is eliminated, meanwhile, in order to process video and audio information by using ResNet, the video is subjected to frame extraction processing, and the audio is converted into pictures by using short-time Fourier transform processing, so that the pictures are taken as the input of ResNet.

3. The cross-modal fine-grained retrieval method based on multi-channel fusion according to claim 1, wherein the step 3) comprises: to obtain local features of a text, N-Gram information of the text is extracted through different convolution kernel sizes, then the most critical information extracted by each convolution operation is highlighted through a maximum pooling operation, the features are combined through a full connection layer after splicing, and finally a model is trained through a cross entropy loss function.

4. The cross-modal fine-grained retrieval method based on multi-channel fusion according to claim 1, wherein the step 5) comprises: for the extracted mode specific features, the invention divides the features of each mode into four channels, and then fusion recombination is carried out to obtain a new mode specific feature f' _I,f'_V,f'_A,f'_T.

5. The cross-modal fine-grained retrieval method based on multi-channel fusion according to claim 1, wherein the step 6) comprises: the cross-mode fine granularity retriever predicts which category the input case specifically belongs to according to the characteristics obtained in the step 5), obtains the category prediction loss of each mode, classifies the case according to the category under the condition of not considering the mode by minimizing the category prediction loss improvement model, and provides preconditions for subsequent retrieval tasks.

6. The cross-modal fine-grained retrieval method based on multi-channel fusion according to claim 1, wherein the step 7) comprises: the cross-modal fine-grained retriever retrieves according to the classification result obtained in the step 6), and the classification in the step 6) is carried out according to the classification without considering the modes, so that the use cases of the same class in different modes can be divided together, after a user inputs a certain class in a certain mode, the system can retrieve and feed back the use cases of other modes which belong to the same class as the input sample, and the cross-modal fine-grained retrieval task is realized.