CN115563316A - Cross-modal retrieval method and retrieval system - Google Patents

Abstract

The present invention provides a cross-modal retrieval method and a retrieval system. The retrieval method includes: using a CLIP pre-training model to encode features to obtain original modal features including original images and text; Perform attention alignment processing to obtain modality alignment data to realize the semantic correlation between the original modalities; pass the modality data formed by the above steps through a weight-sharing multi-layer perceptron to maintain modality invariance; use Arc4cmr The loss function distributes the final feature data onto a normalized hypersphere for category boundary constraints. The cross-modal retrieval method of the present invention makes the common representation of the paired images and texts as similar as possible, and simultaneously enhances the intra-class compactness and inter-class difference.

Description

A cross-modal retrieval method and retrieval system

technical field

The invention relates to the field of cross-modal retrieval with maximum semantic correlation and modal alignment, and specifically relates to a cross-modal retrieval method and retrieval system.

Background technique

Information resources have shown a mixed trend of multi-modal data (text, image, audio, video, etc.), these data are cross-correlated and gradually deeply integrated, and these multimedia data show a trend of rapid growth. How to mine the hidden semantic associations between cross-modal data and realize cross-modal information retrieval is an important prerequisite for making full use of multi-modal data resources.

With the continuous increase of data scale and model scale, deep learning has gradually entered the era of pre-training models, and how to better apply it to downstream tasks has attracted more and more attention, such as CLIP, SimVLM, etc. The existing text image reasoning ability of this type of pre-training model is relatively good for different downstream tasks such as image description (Image Captioning), visual question answering (Visual Question Answering, VQA), cross-modal retrieval (Cross-Modal Retrieval), etc. Good migration ability. Compared with traditional image classification methods, the CLIP model no longer assigns a noun label to each image, but a sentence. Therefore, images that were forcibly classified into the same category in the past have "infinitely fine-grained" labels. Although the pre-training model CLIP of the unsupervised comparative learning method has obtained rich text-image semantics through 40 billion image-text pairs, the pre-encoding stages of CLIP for the two modal data are still independent of each other, and there is still a lack of model. Interaction of information between states. CLIP uses contrastive loss constraints to give the judgment of two modal matches or mismatches, and each image (text) modal information has one and only one text (image) modal information that matches it, ignoring the one-to-many Intra-modal and inter-modal rich semantic information and discrimination information contained in the approximate matching situation.

In view of this, the present invention is proposed.

Contents of the invention

In view of this, the present invention discloses a new type of cross-modal retrieval method. Firstly, the feature representation of one modality is re-expressed by another modality through the Decomposable Attention mechanism, so as to obtain richer semantic information and enhance both Semantic associations of various modalities, and then in the label space, the Arc4cmr loss function is used to distribute the learned multimodal features on the normalized hypersphere, and the corner edge penalty is added between the features and weights to make the categories have clear distinctions. Decision boundary, which achieves simultaneous enhancement of intra-class compactness and inter-class difference.

Specifically, the present invention is achieved through the following technical solutions:

In the first aspect, the present invention discloses a novel cross-modal retrieval method, comprising the following steps:

Use the CLIP pre-training model to encode the features of the image and text samples to obtain the original modality features including the original image and text;

performing attention alignment processing on the original modality features to obtain modality alignment data so as to realize the semantic correlation between the original modality;

The mode alignment data formed by the above steps is passed through a weight-sharing multi-layer perceptron to maintain the invariance of the mode;

The obtained modal data is distributed onto a normalized hypersphere using the Arc4cmr loss function for class boundary constraints.

In the second aspect, the present invention discloses a cross-modal retrieval system, including:

Initial module: used to encode the features of image and text samples using the CLIP pre-training model to obtain the original modal features including the original image and text;

Alignment module: for performing attention alignment processing on the original modality features to obtain modality alignment data so as to realize the semantic correlation between the original modality;

Weight sharing module: used to pass the modal alignment data formed by the above steps through a weight-sharing multi-layer perceptron to maintain the invariance of the modal;

Normalization module: Use the Arc4cmr loss function to distribute the obtained final modal data onto a normalized hypersphere for category boundary constraints.

In a third aspect, the present invention discloses a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the cross-modal retrieval method as described in the first aspect are implemented.

In a fourth aspect, the present invention discloses a computer device, which includes a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the program, the computer program described in the first aspect is implemented. The steps of the modal retrieval method.

The cross-modal retrieval in the existing technology solves the problem of heterogeneity difference by measuring the similarity of different modal data with heterogeneous underlying features and high-level semantic correlation. It can be generally divided into two types: unsupervised retrieval and supervised retrieval. kind.

Unsupervised cross-modal retrieval: Canonical Correlation Analysis (CCA) is essentially a multivariate statistical analysis, which uses the correlation between multiple image and text matching pairs to obtain an unsupervised common subspace, and image features and text features are mapped to the common subspace to obtain a unified representation of different modal data, reflecting the overall correlation between the two modalities, thereby achieving cross-modal retrieval. Kernel correlation analysis (Kernel CCA, KCCA) introduces kernel kernel techniques to improve the situation where CCA may fail when there is a nonlinear correlation between two variables. Correspondence Autoencoders (Corr-AE) utilize autoencoders to consider reconstruction error and correlation loss in cross-modal retrieval.

Supervised cross-modal retrieval: Joint Representation Learning (JRL) integrates sparse and semi-supervised regularization across different media types in a unified framework to jointly explore pairwise correlations and semantic information. Adversarial cross-modal retrieval (ACMR) attempts to classify different modalities through the idea of adversarial learning. Cross-modal correlation learning (CCL) mines coarse-grained information of different media types data through multi-task learning. The deep supervised cross-modal retrieval method (Deep Supervised Cross-Modal Retrieval, DSCMR) in the public representation space by linearly classifying the samples to preserve the semantic distinction, through the weight sharing strategy to maintain the invariance of the modality in the public representation space middle. CLIP4CMR (CLIP for Supervised Cross-Modal Retrieval, CLIP4CMR), which adds class-level association information to the pre-training model CLIP, uses CLIP as the backbone network to generate the original feature representation of each modality, and then sends it to the multi-layer perceptron of each modality to learn common Representation space, for the lack of robustness of unknown categories, by assigning a set of unified prototypes as class agents, and using the nearest prototype (Nearest-Prototype) classification rules for reasoning to solve the problem of lack of robustness of unknown categories, by pre-training Model CLIP adds class-level association information.

However, the traditional approach to cross-modal retrieval in the prior art is to embed text and images into a joint latent space via a two-tower structure model, and then apply a cosine similarity equidistant metric to let the model make the matched text-image With higher similarity, however, there are relatively large representation differences between the two modalities, making direct comparison of the two modalities inherently difficult.

In order to solve the above-mentioned technical problems, the present invention provides a cross-modal retrieval method. Firstly, the features are encoded by the pre-trained model CLIP to obtain the original image and text representation. To further enhance the modal information interaction, the original modal representation is then fed into the attention alignment module. That is, for each query of the image (text) modality (in a batch), pay more attention to the text (image) sample that matches the query in the text (image) modality library of batch size, and realize a single sample aligned with each other. At the same time, the semantic association of the two modal information is enhanced. Finally, the shared multi-layer perceptron is used to share the weight parameters to process the data after the above operations, and to generate a common representation space for each modality data while adding semantic restrictions, so that the public representation of the paired image and text is as close as possible to achieve At the same time, it enhances the intra-class compactness and inter-class difference.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiment. The drawings are only for the purpose of illustrating a preferred embodiment and are not to be considered as limiting the invention. Also throughout the drawings, the same reference numerals are used to designate the same parts. In the attached picture:

FIG. 1 is an overall framework diagram of a cross-modal retrieval method provided by an embodiment of the present invention;

FIG. 2 is a schematic diagram of the operation of the modality alignment method provided by the embodiment of the present invention;

Fig. 3 is the angular space schematic diagram of Arc4cmr loss that the embodiment of the present invention provides;

FIG. 4 is a schematic flowchart of a computer device provided by an embodiment of the present invention;

Fig. 5 is a result diagram of a visualization experiment provided by an embodiment of the present invention.

detailed description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only, and is not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present disclosure, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

The invention discloses a cross-modal retrieval method, as shown in Figure 1, comprising the following steps:

Use the CLIP pre-training model to encode the features of the image and text samples to obtain the original modality features including the original image and text;

performing attention alignment processing on the original modality features to obtain modality alignment data so as to realize the semantic correlation between the original modality;

A multi-layer perceptron for weight sharing of the modality alignment data formed by the above steps to maintain the invariance of the modality;

The resulting final modal data is distributed onto a normalized hypersphere using the Arc4cmr loss function for category boundary constraints.

文本表示为

模态对齐模块将原始图像(文本)特征表示与用文本(图像)重新表示的图像(文本)特征进行相加并做归一化处理，促进两模态信息的交互，增加跨模态数据的同类聚合度以及异类分离度，同时增强两种搜索模式的精度使两种检索结果的精度都能得到均衡的提高。In the solution of the present invention, it is advocated that modality alignment be placed on the encoding of the backbone network to increase the matching degree of cross-modal data of the same type and the separation degree of heterogeneous modality data. A decomposed attention adjustment is performed for each sample of modality 1 (image or text) and all samples in the batch of modality 2 (text or image). Based on the original feature representation obtained through the CLIP encoder, the feature representation of one modality is re-represented by another modality through the Decomposable Attention mechanism to enhance the semantic association of the two modalities. In the process of modality alignment, a single query of a modality can obtain multiple approximate matching information of another modality, so as to obtain richer semantic information. In order to prevent the loss of information due to the large proportion of irrelevant modes in the new feature representation, the output features after modality alignment are added to the original features, and then to Layer Normalization to ensure the stability of the data feature distribution during the optimization process , speed up the convergence of the model, that is, the final image is expressed as

text expressed as

The modality alignment module adds the original image (text) feature representation and the image (text) feature re-expressed with text (image) and performs normalization processing to promote the interaction of two modal information and increase the cross-modal data. The degree of aggregation of the same type and the degree of separation of different types, and the accuracy of the two search modes are enhanced at the same time, so that the accuracy of the two retrieval results can be improved in a balanced manner.

Specifically, as shown in Figure 2, in fact, the original features of the images in the batch (striped grid on the left, using colors as intervals to distinguish multiple images in the batch) are used as the query Q for each image and all the text in the batch The original feature calculates the similarity (one-to-many relationship, one Q is multiplied by multiple K to get the attention weight, which is the blue bar of different lengths between K and V in the figure, the longer the representative, the more similar), and then Multiply the attention weight with the specific feature value V of the original text feature to obtain a new text feature to represent the image feature, that is, the aligned text feature representation (striped grid on the right), in order to prevent semantic inconsistency in the new aligned text feature representation The weight distribution of relevant original text features is too large, resulting in the loss of information represented by the image, and then the original image features and aligned text features are added and layer normalized LayerNormalization processing.

The above process refers to the specific adjustment process of decomposing attention when modality 1 is the original image and modality 2 is the original text, that is, the operation that exists in the image (modal 1) retrieval text (modal 2) process. What is involved here is the mutual retrieval between the two modalities. Similarly, the text (modal 1) retrieval image (modal 2) is to change the specific input of QKV. Its core is actually a matrix operation, so the alignment principle is the same, only the attention weight matrix obtained in the image retrieval text process is transposed to be used for text retrieval images.

In addition, the cross-modal retrieval task requires increasing the intra-class similarity and aggregation and increasing the inter-class difference and inconsistency as much as possible. In order to increase the intra-class compactness and inter-class separation while satisfying the classification, and eliminate the boundary ambiguity problem, the additive angular margin loss (ArcFace) is applied to the field of cross-modal retrieval, and it is named Arc4cmr loss. The specific process is: directly enforcing between the nearest classes in the angle space to maximize the classification bounds, L2 regularizes the feature x _i and the corresponding weight W _yi so that ||W _yi ||=1, after normalization The feature of is multiplied by a rescaled rescale parameter s, so that || _xi ||=s, that is, the embedded features are distributed on a hypersphere with a radius of s; on the other hand, between the feature x _i and the target weight W _yi A custom additive angular margin m is added to replace the original cosθ _yi with cos(θ _yi +m), and the rest remain unchanged. In fact, each weight w here provides a category center, which is changed to θ _yi +m by adding an angular interval, so that the original corresponding output is smaller and the spatial angle becomes larger, thus increasing the difficulty of training and more towards The clustering of class centers, the normalization step of features and weights make the predictions only depend on the angle between features and weights; finally, an angle edge penalty m is added between _xi and W _yi , while enhancing intra-class compactness and Differences between classes. The specific formulas expressed as formulas 1 and 2 are restrictive conditions.

为特征x_i与其对应权重W_yi的余弦夹角，m为角边缘惩罚，n为类别数目，即k＝1，2，...，n，W_k为各个类别的权重，θ_k为将输入特征x_i误判为非y_i类的其他k类别(对应k类权重为W_k)的余弦夹角。对于不同的检索需求公式1、2仅作输入上的变化。对于图像检索文本I2T的损失函数L_I，输入为

相应的正则化为

即

对于文本检索图像T2I的损失函数L_T，输入变为

相应的正则化为

即

综上，所提的SMR-MA模型的目标函数为L_Arc4cmr＝L_I+L_T。Arc4cmr损失的角度空间示意图如图3所示，其中不同的颜色代表不同的类别，圆圈代表图像模态，三角代表文本模态。In the above formula, the batch size is N, that is, i=1, 2, ..., N, x _i is the feature input, and its category label is y _i ,

is the cosine angle between the feature x _i and its corresponding weight W _yi , m is the angle edge penalty, n is the number of categories, that is, k=1, 2,..., n, W _k is the weight of each category, θ _k is the The input feature x _i is misjudged as the cosine angle of other k categories (corresponding to k category weight W _k ) that is not y _i category. For different retrieval requirements, formulas 1 and 2 only change the input. For the loss function L _I of image retrieval text I2T, the input is

The corresponding regularization is

which is

For the loss function LT of text retrieval image _T2I , the input becomes

The corresponding regularization is

which is

In summary, the objective function of the proposed SMR-MA model is L _Arc4cmr = L _I + L _T . The schematic diagram of the angular space of Arc4cmr loss is shown in Fig. 3, where different colors represent different categories, circles represent image modalities, and triangles represent text modalities.

In addition, in addition to providing a cross-modal retrieval method, the present invention also provides a cross-modal retrieval system, which specifically includes:

Initial module: used to encode the features of image and text samples using the CLIP pre-training model to obtain the original modal features including the original image and text;

Alignment module: for performing attention alignment processing on the original modality features to obtain modality alignment data so as to realize the semantic correlation between the original modality;

Weight sharing module: used to pass the modal alignment data formed by the above steps through a weight-sharing multi-layer perceptron to maintain the invariance of the modal;

Normalization module: used to distribute the obtained final modal data to the normalized hypersphere by using the Arc4cmr loss function for category boundary constraints.

During specific implementation, each of the above modules may be implemented as an independent entity, or may be combined arbitrarily as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

Experimental example 1

The overall performance of the cross-modal retrieval method implemented by the embodiment of the present invention was compared with 8 representative baseline methods in the prior art, including 4 traditional methods, CCA, KCCA, Corr-AE, JRL, and 4 A deep learning based method, ACMR, CCL, DSCMR and CLIP4CMR. Taking the mean Average Precision (mAP) of the cross-modal retrieval standard as the evaluation index, the mAP scores of text retrieval from image (I2T) and image retrieval from text (T2I) were compared and verified.

Table 1 Comparison of mAP values between SMR-MA and baseline methods on benchmark datasets

Comprehensive analysis experiments on three benchmark datasets show that the proposed method has good performance in cross-modal retrieval tasks, compared with the current baseline methods that achieve the best results on Wikipedia, PascalSentence and NUS-WIDE , the proposed SMR-MA improves the mAP by 9.4%, 0.7%, 3.4% and 8.7%, respectively, and achieves the effect of SOTA (state-of-the-art), so it has higher application value.

In order to intuitively observe the effectiveness of the maximum semantic relevance and modality alignment model (SMR-MA), observe whether the representation of high-dimensional image and text samples in the shared representation space has achieved good separability, through t-SNE (t-distributed Stochastic Neighbor Embedding), a non-linear dimensionality reduction algorithm, projects the original 1024-dimensional high-dimensional data into a 2-dimensional space for visualization. Use the Wikipedia dataset for visualization experiments. Figure 5(d) and Figure 5(e) show the original feature distribution of the image and text obtained by the CLIP visual encoder and text encoder respectively. It can be seen from the two figures that the degree of separation between classes and the degree of aggregation within classes are both Low, resulting in low accuracy of direct matching. Figure 5(a) and Figure 5(b) respectively show the distribution of image and text representations through SMR-MA, both of which can effectively discriminate samples of different semantic categories and divide them into corresponding semantic discrimination clusters. Figure 5(c) demonstrates how much the feature embedding distributions of the two modalities overlap in the common representation space, which shows that the method has a significant effect on eliminating modality differences.

FIG. 4 is a schematic structural diagram of a computer device disclosed in the present invention. 4, the computer device 400 includes at least a memory 402 and a processor 401; the memory 402 is connected to the processor through a communication bus 403, and is used to store computer instructions executable by the processor 401, and the processing The device 401 is configured to read computer instructions from the memory 402 to implement the steps of the cross-modal retrieval method described in any of the above-mentioned embodiments.

As for the above device embodiments, since they basically correspond to the method embodiments, for relevant parts, please refer to part of the description of the method embodiments. The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed solution. It can be understood and implemented by those skilled in the art without creative effort.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

A final note: While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as primarily describing features of particular embodiments of particular inventions. Certain features that are described in this specification in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function in certain combinations as described above and even be initially so claimed, one or more features from a claimed combination may in some cases be removed from that combination and the claimed A protected combination can point to a subcombination or a variant of a subcombination.

Similarly, while operations are depicted in the figures in a particular order, this should not be construed as requiring that those operations be performed in the particular order shown, or sequentially, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can often be integrated together in a single software product in, or packaged into multiple software products.

Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the present disclosure within the scope of protection.

Claims (9)

1. A cross-modal retrieval method is characterized by comprising the following steps:

coding the features by adopting a CLIP pre-training model to obtain original modal features comprising original images and texts;

performing attention alignment processing on the original modal characteristics to obtain modal alignment data so as to realize semantic correlation between the original modalities;

keeping the modal invariance of the modal alignment data formed in the previous step through a weight-sharing multilayer perceptron;

and distributing the finally obtained modal data to a normalized hypersphere by utilizing an Arc4cmr loss function to carry out class boundary constraint.

2. The cross-modality retrieval method according to claim 1, wherein the attention-alignment processing method comprises:

through the decommissible Attention mechanism, each sample of the original image (text) contained in the modality 1 is readjusted by the text (image) contained in all the modality 2 samples in the batch, that is, the modality 1 data is re-represented by the modality 2 data.

3. The cross-modality retrieval method according to claim 2, further comprising, after the attention-alignment process:

performing Add operation on the output characteristics subjected to mode alignment and the characteristics of the original mode, and performing Normalization processing on the output characteristics subjected to mode alignment and the characteristics of the original mode to accelerate convergence of the model to obtain image mode characteristic data of the final characteristics

Text modal characteristic data of

4. The cross-modality retrieval method of claim 3, wherein the modality alignment method comprises: when the mode 1 is an original image and the mode 2 is an original text, the original features of the images in the batch are used as query Q, similarity between each image and all original features K of the texts in the batch is calculated, attention weight is obtained, and then the attention weight is multiplied by the specific feature value V of the original features of the texts to obtain the output features which are subjected to mode alignment.

5. The cross-modal search method of claim 2, wherein the method for performing class boundary constraint by distributing the finally obtained modal data onto a normalized hypersphere using the Arc4cmr loss function comprises:

will feature x _i And corresponding weight W _yi L2 regularization is performed such that | | | W _yi I | =1, then the normalized feature is multiplied by a rescale parameter s, so that | | x _i I | = s, i.e. such that the embedded features are distributed on a hypersphere with radius s;

at feature x _i And target weight

Adds a self-defined additive angle edge distance m cos (theta) _yi + m) instead of cos θ _yi 。

6. The cross-modal search method of claim 5, wherein the method of distribution to the normalized hypersphere represents a specific formula:

in the above formula, the batch size is N, i.e., i =1,2 _i For feature input, its class label is y _i ，

Is a characteristic x _i Corresponding weight W to _yi M is the angular edge penalty, n is the number of classes, i.e. k =1,2 _k For the weight of each class, θ _k To input feature x _i Wrongly judging as other k types other than yi types, wherein the weight of the corresponding k types is W _k The cosine angle of (c);

loss function L for image retrieval text I2T _I Input is as

Corresponding regularization

Namely, it is

Loss function L for text retrieval image T2I _T Input becomes

Corresponding regularization

Namely, it is

Then the maximum word mentionedThe objective function used by the semantic correlation and modal alignment model is L _Arc4cmr ＝L _I +L _T 。

7. A retrieval system using the cross-modality retrieval method according to any one of claims 1 to 6, characterized by comprising:

an initial module: the method comprises the steps of coding the characteristics of an image and a text sample by adopting a CLIP pre-training model to obtain original mode characteristics comprising an original image and a text;

an alignment module: the original modal characteristics are subjected to attention alignment processing to obtain modal alignment data so as to realize semantic correlation between the original modalities;

the weight sharing module: the multi-layer perceptron is used for sharing the modal alignment data formed in the previous step through weight so as to keep the invariance of the modality;

a normalization module: and the method is used for distributing the obtained final modal data to a normalized hypersphere by utilizing an Arc4cmr loss function to carry out class boundary constraint.

8. A computer-readable storage medium, on which a computer program is stored, which, when executed, carries out the steps of the cross-modality search method of any one of claims 1 to 6.

9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the cross-modality retrieval method according to any one of claims 1-6 are implemented when the program is executed by the processor.

Images