CN112148839A - Image-text matching method, device and storage medium - Google Patents

Abstract

The present disclosure relates to a method, device and storage medium for image and text matching. The graphic-text matching method includes: acquiring an image to be matched with graphic-text; inputting the image into a pre-trained graphic-text encoding model, encoding the image, and obtaining an image vector; A text vector similar to an image vector; wherein, the image-text encoding model further includes a text encoding sub-network for encoding text in the training process, and the pre-stored text vector is preset by the text encoding sub-network. Determining text encoding; determining the text corresponding to a preset number of text vectors similar to the image vector as the text matching the image. Through the present disclosure, the image-text matching efficiency of the image-text matching server can be improved, and the system delay of the image-text matching server can be reduced.

Description

Image-text matching method, device and storage medium

technical field

The present disclosure relates to the field of computer technology, and in particular, to a method, an apparatus, and a storage medium for image-text matching.

Background technique

Multimodal retrieval is a new retrieval method, which can realize data retrieval between different modalities. For example, during image and text retrieval, the user can input an image to retrieve the description text matching the image, or the user can input a text to retrieve the image described by the sentence.

Taking the retrieval of text based on images as an example, the server needs to roughly retrieve candidate texts that are relatively closely related to the input images in the text library according to the images input by the user. After the candidate texts are obtained, the text encoding model is used to encode the candidate texts. , obtain the text vector of the candidate text, and finally determine the matching degree with the user input image according to the text vector of the candidate text.

When the image and text matching is performed in the above manner, the server needs to go through several steps such as preliminary retrieval, coding, and calculation of the matching degree to realize the image and text matching, and the processing efficiency of the server is low.

SUMMARY OF THE INVENTION

In order to overcome the problems existing in the related art, the present disclosure provides a picture-text matching method, device and storage medium.

According to a first aspect of the embodiments of the present disclosure, there is provided a graphic-text matching method, the graphic-text matching method includes: acquiring an image to be subjected to graphic-text matching; inputting the image into a pre-trained graphic-text encoding model, Encoding to obtain an image vector; determining a text vector similar to the image vector from a pre-stored text vector; wherein, the image-text encoding model also includes a text encoding sub-network for encoding the text during the training process , the pre-stored text vector is determined by encoding the preset text through the text encoding sub-network; the text corresponding to a preset number of text vectors similar to the image vector is determined as the text matching the image.

In an example, the determining a text vector similar to the image vector from a pre-stored text vector includes:

respectively determine the cosine distance between each text vector in the pre-stored text vector and the image vector;

According to the calculated cosine distance, obtain the text vector whose cosine distance from the image vector is greater than the set distance threshold, and

A text vector whose cosine distance is greater than the set distance threshold is determined as a text vector similar to the image vector.

In an example, the pre-stored text vector is determined by the text encoding sub-network to encode the preset text: the image and text encoding model is called; the preset text is input into the image and text encoding model, and the The text encoding sub-network included in the image-text encoding model encodes the input preset text to obtain the pre-stored text vector.

In an example, the image-text encoding model is obtained by training based on a first training sample pair and a second training sample pair, and the first training sample pair includes the preset text and an image matching the preset text samples, and the second training sample pair includes the preset text and image samples that do not match the preset text.

According to a second aspect of the embodiments of the present disclosure, there is provided a method for training an image-text encoding model. The method for training an image-text encoding model includes: determining a preset text sample; determining an image matching the preset text sample, and obtaining a first training sample pairs, and determining images that do not match the preset text samples to obtain a second training sample pair; and training based on the first training sample pair and the second training sample pair to obtain an image-text encoding model.

In an example, the image and text encoding model includes an image encoding sub-network and a text encoding sub-network, and the image and text encoding model obtained by training based on the first training sample pair and the second training sample pair includes: The image coding sub-network separately extracts the image vector of each image sample in the first training sample pair and the second training sample pair, and through the text coding sub-network, respectively extracts the first training sample pair and the second training sample pair. 2. The text vector of each text sample in the training sample pair; the first cosine distance and the second cosine distance are determined based on the image vector of the image sample and the text vector of the text sample, wherein the first cosine distance is The distance is the cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is the cosine distance between the text vector and the image vector in the second training sample; according to The first cosine distance, the second cosine distance, and the loss function adjust the training parameters of the image encoding sub-network and the text encoding sub-network to obtain the image-text encoding model that satisfies the loss value; Wherein, the image-text encoding model satisfies that the first cosine distance is greater than a preset first distance threshold, and the second cosine distance is less than a preset second distance threshold.

According to a third aspect of the embodiments of the present disclosure, there is provided an image-text matching apparatus, the image-text matching apparatus includes: an acquiring unit configured to acquire an image to be subjected to image-text matching; a processing unit configured to input the image into a pre-trained image-text encoding model, which encodes the image to obtain an image vector; the determining unit is configured to determine a text vector similar to the image vector from the pre-stored text vector; wherein, the image-text encoding The model also includes a text encoding sub-network for encoding text during the training process, and the pre-stored text vector is determined by the text encoding sub-network to encode the preset text; the determining unit is further configured to Text corresponding to a preset number of text vectors similar to the image vector is determined as the text matching the image.

In an example, the determining unit determines a text vector similar to the image vector from the pre-stored text vectors in the following manner: respectively determining the cosine distance between each text vector in the pre-stored text vector and the image vector According to the calculated cosine distance, obtain the text vector whose cosine distance with the image vector is greater than the set distance threshold, and determine the text vector with the cosine distance greater than the set distance threshold as the text vector similar to the image vector.

In an example, the determining unit determines the pre-stored text vector by encoding the preset text through the text encoding sub-network in the following manner: calling the image-text encoding model; inputting the preset text into the The image-text encoding model encodes the input preset text through a text encoding sub-network included in the image-text encoding model to obtain the pre-stored text vector.

In an example, the image-text encoding model is obtained by training based on a first training sample pair and a second training sample pair, and the first training sample pair includes the preset text and an image matching the preset text samples, and the second training sample pair includes the preset text and image samples that do not match the preset text.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an apparatus for training an image-text encoding model, the apparatus for training an image-text encoding model includes: a determining unit configured to determine a preset text sample, and determine a text sample corresponding to the preset text sample. matching images, obtain a first training sample pair, and determine an image that does not match a preset text sample, obtain a second training sample pair; a training unit is configured to be based on the first training sample pair and the second training sample pair The sample pairs are trained to obtain the image-text encoding model.

In an example, the image and text encoding model includes an image encoding sub-network and a text encoding sub-network, and the training unit obtains the image and text encoding based on the first training sample pair and the second training sample pair in the following manner: Model: through the image coding sub-network, respectively extract the image vector of each image sample in the first training sample pair and the second training sample pair, and through the text coding sub-network, respectively extract the first training sample pair and The text vector of each text sample in the second training sample pair; the first cosine distance and the second cosine distance are determined based on the image vector of the image sample and the text vector of the text sample, wherein the A cosine distance is the cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is the cosine distance between the text vector and the image vector in the second training sample distance; according to the first cosine distance, the second cosine distance, and the loss function, adjust the training parameters of the image encoding sub-network and the text encoding sub-network to obtain the image and text satisfying the loss value An encoding model; wherein, the image-text encoding model satisfies that the first cosine distance is greater than a preset first distance threshold, and the second cosine distance is less than a preset second distance threshold.

According to a fifth aspect of the present disclosure, there is provided an image-text matching apparatus, the image-text matching apparatus comprising: a memory configured to store an instruction. and a processor configured to invoke an instruction to execute the image-text matching method in the foregoing first aspect or any example of the first aspect.

According to a sixth aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions, when executed by a processor, execute the aforementioned first The image-text matching method in one aspect or any example of the first aspect.

According to a seventh aspect of the present disclosure, there is provided an apparatus for training an image-text encoding model. The apparatus for training an image-text encoding model includes: a memory configured to store instructions. and a processor, configured to invoke an instruction to execute the image-text encoding model training method in the second aspect or any example of the second aspect.

According to an eighth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, where the non-transitory computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions, when executed by a processor, execute the aforementioned first The second aspect or the image-text encoding model training method in any example of the second aspect.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects: the server for image and text matching acquires the image to be matched with the image and text, and inputs the image to be matched with the image and text into a pre-trained image and text encoding model, The image to be matched is encoded, and after the image vector of the image to be matched is obtained, a text vector similar to the image vector of the image to be matched is determined from the pre-stored text vector, and a text vector similar to the image vector of the image to be matched is determined from the pre-stored text vectors. The text corresponding to a preset number of text vectors that are similar to the image vector of the matching image is determined as the text matching the image. As a result, the server for image-text matching does not need to retrieve multiple candidate texts matching the user input image from the preset text first, and does not need to encode the candidate texts in real time to obtain a text vector, thereby improving the image-text matching of the image-text matching server. Matching efficiency, reducing the system delay of the server for image and text matching.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

FIG. 1 is a flow chart of a method for image-text matching according to an exemplary embodiment.

Fig. 2 is a flow chart of a method for image-text matching according to an exemplary embodiment.

FIG. 3 is a flow chart of training a text encoding model and an image encoding model according to an exemplary embodiment.

Fig. 4 is a block diagram of an apparatus for image-text matching according to an exemplary embodiment.

Fig. 5 is a block diagram of an apparatus for training an image-text encoding model according to an exemplary embodiment.

Fig. 6 is a block diagram of an apparatus according to an exemplary embodiment.

Detailed ways

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

The technical solutions of the exemplary embodiments of the present disclosure can be applied to an application scenario in which text matching the image is returned to the image-text matching system according to the image input by the user to the image-text matching system. In this scenario, the image-text matching system may include, for example, a terminal for inputting images to be matched with images and texts and a server for performing text matching on the input images to be matched with images and texts, that is, a server for image-text matching. The user terminal includes, but is not limited to, fixed or mobile electronic devices such as smart phones, tablet computers, notebook computers, desktop computers, and e-book readers. The server used for image and text matching may be an independent application service device, or a service cluster composed of multiple servers. In practical applications, it may be a cloud server, a cloud host, a virtual center, etc. The structure and its realization form are not limited.

In the related art, multi-modal retrieval is used. For example, when retrieving the description text matching the image according to the input image, the user inputs the image whose text needs to be matched on the terminal, and the server for image-text matching receives the user input. After obtaining the image, it is necessary to retrieve multiple candidate texts that match the user input image in the text library. After obtaining the candidate text, encode the candidate text based on the network model to obtain the text vector of the candidate text, and then according to the text vector of the candidate text, The text matching the user-input image in the candidate text is determined, and the text matching the user-input image is fed back to the terminal for the user to select.

Since the server used for image and text matching needs to go through the above-mentioned steps in the process of performing image and text matching, the image and text matching can be realized, so that the server used for image and text matching has low efficiency and serious system delay when performing operations.

The embodiments of the present disclosure provide a method for matching image and text. In the image-text matching method of the present disclosure, a server for image-text matching acquires an image to be matched with image and text, and inputs the image to be matched with image-text into a pre-trained image-text encoding model, and a The image is encoded, and after obtaining the image vector of the image to be matched, a text vector similar to the image vector of the image to be matched is determined from the pre-stored text vectors, and the image vector of the image to be matched is determined. The text corresponding to the similar preset number of text vectors is determined as the text matching the image. As a result, the server for image-text matching does not need to retrieve multiple candidate texts matching the user input image from the preset text first, and does not need to encode the candidate texts in real time to obtain a text vector, thereby improving the image-text matching of the image-text matching server. Matching efficiency, reducing the system delay of the server for image and text matching.

FIG. 1 is a flow chart of a method for image-text matching according to an exemplary embodiment. As shown in FIG. 1 , the image-text matching method includes the following steps.

In step S11, an image to be image-text matching is acquired.

In the present disclosure, the image to be image-text matching may be one or several pictures input by the user, or may be an image frame extracted from the video after inputting the video to the user.

In step S12, the image is input into a pre-trained image and text encoding model, and the image is encoded to obtain an image vector.

The image and text encoding model in the present disclosure can encode images and/or texts according to input images and/or texts, and output image vectors and/or text vectors.

Furthermore, in the present disclosure, the image to be image-text matched is input into the image-text encoding model, and the image to be image-text matched is encoded by the image coding sub-network to obtain an image vector of the image to be image-text matched.

In step S13, a text vector similar to the image vector is determined from the pre-stored text vectors.

In practical applications, since images and texts come from two heterogeneous spaces, in order to directly measure the similarity between images and texts, the image and text can be mapped into one space, and the image vector of the image and the text vector of the text can be used to Measure how similar an image is to text.

Based on the vector similarity principle, in one embodiment, the present disclosure may retrieve text vectors similar to image vectors from pre-stored text vectors, and determine the text corresponding to a preset number of text vectors similar to the image vector as the same as the image vector. Image matching text.

The pre-stored text vector is obtained by encoding the preset text based on the text encoding sub-network included in the image-text encoding model.

In one embodiment, the text encoding sub-network may be, for example, a neural network using Bidirectional Encoder Representations from Transformers (BERT) from the transformer.

In step S14, the text corresponding to a preset number of text vectors similar to the image vector is determined as the text matching the image.

In the present disclosure, after acquiring an image to be matched with images, a text vector similar to the image vector of the image to be matched can be retrieved from a pre-stored text vector according to the image vector of the image to be matched. The text corresponding to the preset number of text vectors similar to the image vector to be matched with the image and text is returned to the user for selection by the user.

For example, after the user inputs a landscape image, the image vector of the landscape image is obtained by encoding the landscape image through the image-text encoding model, and then the text that is similar to the image vector of the input landscape image is retrieved from the pre-stored text vector. vector, and texts corresponding to a preset number of text vectors similar to the image vector of the landscape image, such as ancient poems, beautiful words, etc., are returned to the user for selection by the user. Based on the returned, for example, ancient poems, beautiful words, etc. are returned to the user, which greatly reduces the time for the user to match the text to the scenery. In addition, the user can also use a preset amount of text that is similar to the returned image vector of the landscape image to be used for further processing of the image in the later stage.

In an exemplary embodiment of the present disclosure, a server for image-text matching acquires an image to be matched with image and text, and inputs the image to be matched into a pre-trained image-text encoding model, and a The image is encoded, and after obtaining the image vector of the image to be matched, a text vector similar to the image vector of the image to be matched is determined from the pre-stored text vectors, and the image vector of the image to be matched is determined. The text corresponding to the similar preset number of text vectors is determined as the text matching the image. As a result, the server for image-text matching does not need to retrieve multiple candidate texts matching the user input image from the preset text first, and does not need to encode the candidate texts in real time to obtain a text vector, thereby improving the image-text matching of the image-text matching server. Matching efficiency, reducing the system delay of the server for image and text matching.

FIG. 2 is a flow chart of a method for image-text matching according to an exemplary embodiment. As shown in FIG. 2 , the image-text matching method includes the following steps.

In step S21, an image to be image-text matching is acquired.

In step S22, the image is input into a pre-trained image-text encoding model, and the image is encoded to obtain an image vector.

In step S23, the cosine distance between each text vector in the pre-stored text vector and the image vector is respectively determined, and according to the calculated cosine distance, a text vector whose cosine distance from the image vector is greater than the set distance threshold is obtained, and the cosine distance is Text vectors larger than the set distance threshold are determined as text vectors similar to image vectors.

Based on the principle of vector similarity, in one embodiment, the present disclosure can measure the similarity between vectors by measuring the cosine distance between two vectors, that is, the cosine value of the angle between the two vectors. When the cosine distance between the two vectors approaches 1, the two vectors are characterized as similar vectors. When the cosine distance between the two vectors approaches 0, the two vectors are characterized as dissimilar vectors.

According to the characteristics of the above cosine distance, the distance threshold of the cosine distance similar to the image vector can be set, for example, the distance threshold of the cosine distance similar to the image vector is set to 0.98. That is, when the cosine distance between the text vector in the pre-stored text vector and the image vector is greater than 0.98, the text vector in the pre-stored text vector whose cosine distance from the image vector is greater than 0.98 is determined as a text vector similar to the image vector. Furthermore, the text corresponding to a preset number of text vectors similar to the image vector can be determined as the text matching the image.

For example, obtain an image of mountains and flowing water input by the user, call the image-text encoding model to encode the image of mountains and flowing water to obtain the image vector of the image of mountains and flowing water, and determine the cosine distance between each text vector in the pre-stored text vector and the image vector of mountains and flowing water, respectively, According to the calculated cosine distance, a text vector with a cosine distance greater than 0.98 from the image vector is obtained, and a text vector with a cosine distance greater than 0.98 is determined as a text vector similar to the image vector. Text corresponding to a preset number of text vectors similar to the image vector of the landscape image, such as ancient poems including mountains and rivers, and beautiful texts related to mountains and rivers are returned to the user for selection by the user. Based on the image input by the user, the text matching the image is returned to the user, which greatly reduces the time for the user to match the text to the high mountain and flowing water image. In addition, the user can also use a preset amount of text that is similar to the returned image vector of the mountain and flowing water image to be used for further processing of the image in the later stage.

For another example, when acquiring a video input by the user, since the content of adjacent video frames is not very different, the input video may be sampled every preset number of video frames to obtain each image frame. After calling the image-text encoding model to encode each image frame to obtain the image vector of the image frame, determine the cosine distance between each text vector in the pre-stored text vector and the image vector of each image frame, and obtain the cosine distance according to the calculated cosine distance. A text vector whose cosine distance of the image vector of each image frame is greater than 0.98, and a text vector whose cosine distance is greater than 0.98 is determined as a text vector similar to the image vector of each image frame. Text corresponding to a preset number of text vectors similar to the image vector of each image frame image, for example, including beautiful text related to each image frame, is returned to the user for selection by the user.

In step S24, the text corresponding to the preset number of text vectors similar to the image vector is determined as the text matching the image.

In an exemplary embodiment of the present disclosure, a server for image-text matching obtains an image to be matched with image and text, and by invoking a preset image-text encoding model, encodes the image to be matched with image and text, and obtains an image vector. , by respectively determining the cosine distance between each text vector in the pre-stored text vector and the image vector, and according to the calculated cosine distance, the text vector whose cosine distance is greater than the set distance threshold is determined as a text vector similar to the image vector. As a result, the server for image-text matching does not need to first retrieve multiple candidate texts matching the user input image in the text library, and does not need to encode the candidate texts in real time to obtain text vectors, which improves the image-text matching of the image-text matching server. efficiency, reducing the processing delay of image and text matching.

In order to improve the accuracy of image-text matching and reduce the processing delay for image-text matching, in a real-time manner, the present disclosure can use an end-to-end training method to train a graphics-text encoding model.

Fig. 3 is a flow chart of training a text encoding sub-network and an image encoding model according to an exemplary embodiment, as shown in Fig. 3, including the following steps.

In step S31, a first training sample pair and a second training sample pair are obtained.

In one embodiment, the present disclosure may train the image-text matching model based on the first training sample pair and the second training sample pair. The first training sample pair includes preset text and image samples matching the preset text, and the second training sample pair includes preset text and image samples that do not match the preset text. And, for example, 70% of the sample pair data in the first training sample pair and the second training sample pair can be used as the training data set to train the image-text matching model, and 30% of the sample pair in the first training sample pair and the second training sample pair can be used. The data is used as a test dataset to verify the image-text matching model.

For the convenience of description in the present disclosure, a training sample pair composed of a preset text sample and an image sample matching the preset text is referred to as a first training sample pair. The training sample pair composed of the preset text and the image samples that do not match the preset text is called the second training sample pair.

In one embodiment, the first training sample pair and the second training sample pair in the present disclosure are determined, for example, in the following manner:

Acquire preset text, where the preset text may be one sentence or a combination of multiple sentences with complete semantics. The sentences mentioned here may be sentences in any natural language, for example, it may include a large amount of texts such as Tang poems, Song poems, lyrics, famous quotes, classic movie lines and so on.

An image matching the text in the preset text is constructed to obtain a first pair of training samples, and an image that does not match the preset text is constructed to obtain a second pair of training samples.

In step S32, the image-text matching model is trained, and the image-text matching model is optimized.

In the present disclosure, the image-text matching model may be a model including an image encoding sub-network and a text encoding sub-network, wherein the image encoding sub-network may be a ResNet or a VGG network. The text encoding sub-network can be, for example, a neural network using Bidirectional Encoder Representations from Transformers (BERT) from the Transformer.

In order to avoid the situation of missing and incomplete retrieval when retrieving multiple candidate texts matching the user input image from the text retrieval database, and in order to avoid the situation of processing delay caused by retrieving candidate texts, in one embodiment, this It is disclosed that the image-text matching model can be trained by the end-to-end training method, that is, by jointly training the image coding sub-network and the text coding sub-network, it is possible to perform image-text matching on images with matching text from a global perspective, and improve the performance of image-text matching. accuracy, improve the processing efficiency of image and text matching, and reduce the processing delay of the server for image and text matching.

Since the first training sample pair is a training sample pair composed of the text in the text retrieval database and the image samples matching the text in the text retrieval database, the cosine distance between the text vector and the image vector in the first training sample pair It needs to tend to 1, and the second training sample pair is a training sample pair composed of the text in the text retrieval library and the image samples that do not match the text in the text retrieval library. Therefore, the text vector and image vector in the second training sample pair are The cosine distance between them should tend to 0.

Therefore, when training the image-text matching model including the image encoding sub-network and the text encoding sub-network in an end-to-end manner, the first training sample pair and the second training sample pair are input into the image-text matching model, and the image-text matching model The image encoding sub-network encodes each image sample in the first training sample pair and the second training sample pair, outputs the image vector of the image sample through the image-text encoding model, and uses the text encoding sub-network in the image-text matching model to train the first training sample. Each text sample in the sample pair and the second training sample pair is encoded, and the text vector of the text sample is output through the image-text encoding model.

Then, based on the image vector of the image sample and the text vector of the text sample, the cosine distance between the text vector and the image vector in the first training sample pair is determined, and the distance between the text vector and the image vector in the second training sample is determined. Cosine distance.

In the present disclosure, for the convenience of description, the cosine distance between the text vector and the image vector in the first training sample pair is called the first cosine distance, and the cosine distance between the text vector and the image vector in the second training sample pair is called the cosine distance. is the second cosine distance.

Adjust the parameters of the graphic encoding model according to the first cosine distance, the second cosine distance, and the loss function, that is, adjust the training parameters of the image encoding sub-network and the text encoding sub-network included in the graphic encoding model respectively, so that the graphic encoding The model satisfies that the first cosine distance is greater than the preset first distance threshold, and the second cosine distance is less than the preset second distance threshold.

For example, the preset first distance threshold is 0.97 and the preset second distance threshold is 0.02. The cross-entropy loss function is used to adjust the training parameters of the image encoding sub-network and the text encoding sub-network respectively, so that the image vector and text vector output from the image and text encoding model satisfy the difference between the text vector and the image vector in the first training sample pair. The first cosine distance is greater than 0.97, and the second cosine distance between the text vector and the image vector in the second pair of training samples is less than 0.02. So far, the trained image and text encoding model can be obtained.

In step S33, the test data set is input into the trained image-text matching model for verification, and a verified image-text matching model is obtained.

In the present disclosure, the test data set can be input into the trained image-text matching model for verification, and a verified image-text matching model can be obtained.

Therefore, after the image-text encoding model is trained, the text encoding sub-network included in the trained image-text encoding model is used to encode all the text in the preset text, and then when the user needs to perform image-text matching on the image, The image to be matched with image and text is encoded by the trained image-text encoding model, and after the image vector is obtained, based on the cosine distance between the image vector and the text vector, the image to be matched with the image and text can be directly retrieved from the pre-stored text vector. For the text vector similar to the image vector, the text corresponding to the preset number of text vectors similar to the image vector to be image-text matching is determined as the text matching the image, and is output for the user to select.

Moreover, in order to quickly retrieve the text vector similar to the image vector to be matched with the image and text, after the text encoding sub-network included in the image-text encoding model encodes all the texts in the preset text to obtain the text vector, the encoded Text vectors can be used to build a vector retrieval library through similarity searches such as faiss. Since faiss supports billion-level retrieval, the vector retrieval library constructed based on faiss can greatly improve the retrieval efficiency while ensuring the search accuracy when retrieving text vectors similar to image vectors, thereby improving the efficiency of image-text matching.

In the exemplary embodiment of the present disclosure, the image-text matching model including the image coding sub-network and the text-coding sub-network is trained in an end-to-end manner, so that the trained image-text coding model satisfies the requirement that the first training sample pair is centered. The cosine distance between the text vector and the image vector is greater than the preset first distance threshold, and the second cosine distance is that the cosine distance between the text vector and the image vector in the second training sample is less than the preset second distance threshold. Then, when performing image-text matching on the acquired image, it is only necessary to input the image into the image-text encoding model, encode the acquired image through the image-text encoding model, and obtain the image vector based on the text encoding sub-network included in the image-text matching model. For all the text vectors determined by the preset text encoding, it can be determined to obtain text vectors similar to the image vectors. It is not necessary to select candidate texts in the text retrieval library first, and encode the candidate texts in real time. Based on the encoded candidate text vectors, the The image vector is further matched, which improves the image-text matching efficiency of the image-text matching server and reduces the processing delay of image-text matching.

Based on the same concept, an embodiment of the present disclosure also provides a picture-text matching apparatus.

It can be understood that, in order to realize the above-mentioned functions, the image-text matching apparatus provided by the embodiments of the present disclosure includes corresponding hardware structures and/or software modules for executing each function. Combining with the units and algorithm steps of each example disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of the embodiments of the present disclosure.

Fig. 4 is a block diagram of an apparatus for image-text matching according to an exemplary embodiment. Referring to FIG. 4 , the image-text matching apparatus 400 includes an acquisition unit 401 , a processing unit 402 and a determination unit 403 .

Wherein, the acquiring unit 401 is configured to acquire an image to be matched with graphics and text; the processing unit 402 is configured to input the image into a pre-trained graphics and text encoding model, encode the image, and obtain an image vector; determine Unit 403, configured to determine a text vector similar to the image vector from a pre-stored text vector; wherein, the image-text encoding model also includes a text encoding sub-network for encoding the text in the training process, The pre-stored text vector is determined by encoding the preset text through the text encoding sub-network; the determining unit 403 is further configured to determine the text corresponding to a preset number of text vectors similar to the image vector as Text that matches the image.

In an example, the determining unit 403 determines a text vector similar to the image vector from the pre-stored text vector in the following manner: respectively determining the cosine of each text vector in the pre-stored text vector and the image vector distance; according to the calculated cosine distance, obtain a text vector whose cosine distance from the image vector is greater than the set distance threshold, and determine the text vector whose cosine distance is greater than the set distance threshold as a text vector similar to the image vector .

In an example, the determining unit 403 determines the pre-stored text vector for the preset text encoding through the text encoding sub-network in the following manner: calling the image-text encoding model; inputting the preset text into any The image-text encoding model is used to encode the input preset text through a text encoding sub-network included in the image-text encoding model to obtain the pre-stored text vector.

In an example, the image-text encoding model is obtained by training based on a first training sample pair and a second training sample pair, and the first training sample pair includes the preset text and an image matching the preset text samples, and the second training sample pair includes the preset text and image samples that do not match the preset text.

Fig. 5 is a block diagram of an apparatus for training an image-text encoding model according to an exemplary embodiment. Referring to FIG. 5 , the image-text coding model training apparatus 500 includes a determining unit 501 and a training unit 502 .

The determining unit 501 is configured to determine a preset text sample, and determine an image matching the preset text sample to obtain a first pair of training samples, and determine an image that does not match the preset text sample to obtain a first pair of training samples. Two pairs of training samples; the training unit 502 is configured to obtain an image-text encoding model based on the first pair of training samples and the second pair of training samples.

In an example, the image and text encoding model includes an image encoding sub-network and a text encoding sub-network, and the training unit 502 uses the following method to train the image and text based on the first training sample pair and the second training sample pair. Coding model: through the image coding sub-network, respectively extract the image vector of each image sample in the first training sample pair and the second training sample pair, and through the text coding sub-network, respectively extract the first training sample pair and the text vector of each text sample in the second training sample pair; based on the image vector of the image sample and the text vector of the text sample, determine the first cosine distance and the second cosine distance, wherein the The first cosine distance is the cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is the distance between the text vector and the image vector in the second training sample. Cosine distance; according to the first cosine distance, the second cosine distance, and the loss function, adjust the training parameters of the image encoding sub-network and the text encoding sub-network to obtain the graph that satisfies the loss value A text encoding model; wherein, the image and text encoding model satisfies that the first cosine distance is greater than a preset first distance threshold, and the second cosine distance is less than a preset second distance threshold.

Regarding the apparatus in the above-mentioned embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the method, and will not be described in detail here.

FIG. 6 is a block diagram of an apparatus 600 for image-text matching according to an exemplary embodiment. For example, the apparatus 600 may be provided as a server. 6, apparatus 600 includes a processing component 622, which further includes one or more processors, and a memory resource, represented by memory 632, for storing instructions executable by processing component 622, such as applications. An application program stored in memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute the instructions to perform the above-described image-text matching method.

Device 600 may also include a power component 626 configured to perform power management of device 600 , a wired or wireless network interface 660 configured to connect device 600 to a network, and an input output (I/O) interface 668 . Device 600 may operate based on an operating system stored in memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

It should be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And/or", which describes the association relationship of the associated objects, means that there can be three kinds of relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. The character "/" generally indicates that the associated objects are an "or" relationship. The singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise.

It is further understood that the terms "first", "second", etc. are used to describe various information, but the information should not be limited to these terms. These terms are only used to distinguish the same type of information from one another, and do not imply a particular order or level of importance. In fact, the expressions "first", "second" etc. are used completely interchangeably. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, without departing from the scope of the present disclosure.

It should be further understood that, unless otherwise specified, "connection" includes a direct connection between the two without other components, and also includes an indirect connection between the two with other elements.

It is further to be understood that, although the operations in the embodiments of the present disclosure are described in a specific order in the drawings, it should not be construed as requiring that the operations be performed in the specific order shown or the serial order, or requiring Perform all operations shown to obtain the desired result. In certain circumstances, multitasking and parallel processing may be advantageous.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or techniques in the technical field not disclosed by the present disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

It is to be understood that the present disclosure is not limited to the precise structures described above and illustrated in the accompanying drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (16)

1. A method for matching images and texts, the method comprising:

acquiring an image to be subjected to image-text matching;

inputting the image into a pre-trained image-text coding model, and coding the image to obtain an image vector;

determining a text vector similar to the image vector from pre-stored text vectors;

the image-text coding model further comprises a text coding sub-network used for coding a text in the training process, and the pre-stored text vector is determined by the text coding sub-network according to a preset text code;

and determining the texts corresponding to the preset number of text vectors similar to the image vectors as texts matched with the image.

2. The teletext matching method according to claim 1, wherein determining a text vector from pre-stored text vectors that is similar to the image vector comprises:

respectively determining cosine distances between the text vectors in the pre-stored text vectors and the image vectors;

according to the calculated cosine distance, obtaining a text vector of which the cosine distance with the image vector is greater than a set distance threshold, and

and determining the text vector with the cosine distance larger than a set distance threshold as the text vector similar to the image vector.

3. The teletext matching method according to claim 1, wherein the pre-stored text vector is determined by the text encoding sub-network for a pre-set text encoding, comprising:

and inputting the preset text into the image-text coding model, and coding the input preset text through a text coding sub-network included in the image-text coding model to obtain the pre-stored text vector.

4. The method of claim 1, wherein the teletext coding model is trained on a first pair of training samples and a second pair of training samples, the first pair of training samples including the predetermined text and image samples matching the predetermined text, and the second pair of training samples including the predetermined text and image samples not matching the predetermined text.

5. A method for training a teletext model, the method comprising:

determining a preset text sample;

determining images matched with the preset text samples to obtain a first training sample pair, and determining images not matched with the preset text samples to obtain a second training sample pair;

and training based on the first training sample pair and the second training sample pair to obtain a graphic coding model.

6. The method of claim 5, wherein the teletext model comprises an image coding sub-network and a text coding sub-network, and wherein the training based on the first training sample pair and the second training sample pair to obtain the teletext model comprises:

respectively extracting image vectors of the image samples in the first training sample pair and the second training sample pair through an image coding sub-network, and

respectively extracting text vectors of the text samples in the first training sample pair and the second training sample pair through a text coding sub-network;

determining a first cosine distance and a second cosine distance based on the image vector of the image sample and the text vector of the text sample, wherein the first cosine distance is a cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is a cosine distance between the text vector and the image vector in the second training sample;

adjusting training parameters of the image coding sub-network and the text coding sub-network according to the first cosine distance, the second cosine distance and the loss function to obtain the image-text coding model meeting the loss value;

the image-text coding model enables the first cosine distance to be larger than a preset first distance threshold value, and the second cosine distance to be smaller than a preset second distance threshold value.

7. An apparatus for matching graphics and text, the apparatus comprising:

an acquisition unit configured to acquire an image to be subjected to image-text matching;

the processing unit is configured to input the image into a pre-trained image-text coding model, and code the image to obtain an image vector;

a determination unit configured to determine a text vector similar to the image vector from among pre-stored text vectors;

the image-text coding model further comprises a text coding sub-network used for coding a text in the training process, and the pre-stored text vector is determined by the text coding sub-network according to a preset text code;

the determining unit is further configured to determine a text corresponding to a preset number of text vectors similar to the image vector as a text matching the image.

8. Teletext matching arrangement according to claim 7, wherein the determination unit determines a text vector similar to the image vector from pre-stored text vectors in the following way:

respectively determining cosine distances between the text vectors in the pre-stored text vectors and the image vectors;

according to the calculated cosine distance, obtaining a text vector of which the cosine distance with the image vector is greater than a set distance threshold, and

and determining the text vector with the cosine distance larger than a set distance threshold as the text vector similar to the image vector.

9. Teletext matching arrangement according to claim 7, wherein the determination unit determines the pre-stored text vector for a pre-set text encoding by the sub-network of text encodings as follows:

calling the image-text coding model;

and inputting the preset text into the image-text coding model, and coding the input preset text through a text coding sub-network included in the image-text coding model to obtain the pre-stored text vector.

10. The device according to claim 7, wherein the teletext coding model is trained on a first pair of training samples and a second pair of training samples, the first pair of training samples comprising the predetermined text and image samples matching the predetermined text, and the second pair of training samples comprising the predetermined text and image samples not matching the predetermined text.

11. An apparatus for training a teletext model, the apparatus comprising:

a determination unit configured to determine a preset text sample, an

Determining images matched with the preset text samples to obtain a first training sample pair, and determining images not matched with the preset text samples to obtain a second training sample pair;

a training unit configured to train to obtain a teletext coding model based on the first pair of training samples and the second pair of training samples.

12. The apparatus of claim 11, wherein the teletext model comprises an image coding sub-network and a text coding sub-network, and wherein the training unit is configured to train the teletext model based on the first pair of training samples and the second pair of training samples in the following manner:

respectively extracting image vectors of the image samples in the first training sample pair and the second training sample pair through an image coding sub-network, and

respectively extracting text vectors of the text samples in the first training sample pair and the second training sample pair through a text coding sub-network;

determining a first cosine distance and a second cosine distance based on the image vector of the image sample and the text vector of the text sample, wherein the first cosine distance is a cosine distance between the text vector and the image vector in the first training sample pair, and the second cosine distance is a cosine distance between the text vector and the image vector in the second training sample;

adjusting training parameters of the image coding sub-network and the text coding sub-network according to the first cosine distance, the second cosine distance and the loss function to obtain the image-text coding model meeting the loss value;

the image-text coding model enables the first cosine distance to be larger than a preset first distance threshold value, and the second cosine distance to be smaller than a preset second distance threshold value.

13. An image-text matching device, characterized by comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: -performing the teletext matching method of any one of claims 1-4.

14. A non-transitory computer readable storage medium having instructions therein which, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the teletext matching method according to any one of claims 1-4.

15. A device for training a graphic coding model, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to: performing the teletext code model training method of any one of claims 5-6.

16. A non-transitory computer readable storage medium having instructions that, when executed by a processor of a mobile terminal, enable the mobile terminal to perform the teletext code model training method of any one of claims 5-6.

Images