CN117094365A - Training method and device for image-text generation model, electronic equipment and medium - Google Patents

Abstract

This application discloses a training method, device, electronic equipment and medium for a graphic and text generation model, which belongs to the field of artificial intelligence. The method includes: inputting a first training sample pair in a first training sample pair set to a first image and text generation model, and outputting a second training sample pair, where the first training sample pair includes a first image and a first training sample pair used to describe the first image. The first text of the image content, the second training sample pair includes a second image and a second text, the second image is an image obtained by image-text conversion of the first text, and the second text is an image obtained by image-text conversion of the first image Text; Based on the first training sample pair and the second training sample pair, generate M training sample pairs; replace the first training sample pair in the first training sample pair set with the third training sample pair to obtain the second training sample pair The third training sample pair is the training sample pair with the highest image and text similarity among the M training sample pairs; the first image and text generation model is trained based on the second training sample pair set to obtain the target image and text generation model.

Description

Training method, device, electronic device and medium for image-text generation model

Technical Field

The present application belongs to the field of artificial intelligence, and specifically relates to a training method, device, electronic equipment and medium for a graphic and text generation model.

Background Art

At present, with the rise and continuous development of artificial intelligence generated content (AIGC), image-text generation models, such as the image-text diffusion model in the field of AI painting, have been widely used in the fields of wallpaper, avatar, game, animation design, etc., with the advantages of high efficiency and high degree of automation. In related technologies, text can be input into the image-text generation model to output the corresponding image of the text.

However, there is still a problem of low training accuracy in the model training process of the above-mentioned text-image-text generation model.

Summary of the invention

The purpose of the embodiments of the present application is to provide a training method, device, electronic device and medium for a graphic-text generation model, which can improve the model training accuracy of the graphic-text generation model.

In a first aspect, an embodiment of the present application provides a training method for a graphic-text generation model, the training method for the graphic-text generation model comprising: inputting a first training sample pair in a first training sample pair set into a first graphic-text generation model, and outputting a second training sample pair, wherein the first graphic-text generation model is obtained by training based on the first training sample pair set, the first training sample pair comprises a first image and a first text for describing the image content of the first image, the second training sample pair comprises a second image and a second text, the second image is an image obtained by graphic-text conversion of the first text, and the second text is a text obtained by graphic-text conversion of the first image; based on the first training sample pair and the second training sample pair, M training sample pairs are generated, the M training sample pairs include at least the first training sample pair and the second training sample pair, and M is an integer greater than 1; the first training sample pair in the first training sample pair set is replaced by a third training sample pair to obtain a second training sample pair set, the third training sample pair is the training sample pair with the highest graphic-text similarity among the M training sample pairs; the first graphic-text generation model is trained based on the second training sample pair set to obtain a target graphic-text generation model.

In a second aspect, an embodiment of the present application provides a training device for a graph-text generation model, the training device for the graph-text generation model comprising: a processing module, a generation module, a replacement module, and a training module;

The processing module is used to input the first training sample pair in the first training sample pair set into the first image-text generation model and output the second training sample pair. The first image-text generation model is obtained by training based on the above-mentioned first training sample pair set. The first training sample pair includes a first image and a first text for describing the image content of the first image. The second training sample pair includes a second image and a second text. The second image is an image obtained by image-text conversion of the above-mentioned first text, and the second text is a text obtained by image-text conversion of the above-mentioned first image; the above-mentioned generation module is used to generate M training sample pairs based on the above-mentioned first training sample pair and the above-mentioned second training sample pair. The M training sample pairs at least include the first training sample pair and the second training sample pair, and M is an integer greater than 1; the above-mentioned replacement module is used to replace the first training sample pair in the above-mentioned first training sample pair set with a third training sample pair to obtain the second training sample pair set. The third training sample pair is the training sample pair with the highest image-text similarity among the multiple training sample pairs generated by the above-mentioned generation module; the above-mentioned training module is used to train the first image-text generation model based on the second training sample pair set replaced by the above-mentioned replacement module to obtain the target image-text generation model.

In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

In an embodiment of the present application, a first training sample pair in a first training sample pair set is input into a first image-text generation model, and a second training sample pair is output. The first image-text generation model is obtained by training based on the first training sample pair set, the first training sample pair includes a first image and a first text for describing the image content of the first image, and the second training sample pair includes a second image and a second text, the second image is an image obtained by image-text conversion of the first text, and the second text is a text obtained by image-text conversion of the first image; based on the first training sample pair and the second training sample pair, M training sample pairs are generated, and the M training sample pairs include at least the first training sample pair and the second training sample pair, and M is an integer greater than 1; the first training sample pair in the first training sample pair set is replaced with the third training sample pair to obtain a second training sample pair set, and the third training sample pair is the training sample pair with the highest image-text similarity among the M training sample pairs; the first image-text generation model is trained based on the second training sample pair set to obtain a target image-text generation model. In this way, one or more training sample pairs in the training sample set are input into the image-text generation model for image-text conversion to obtain new training sample pairs. Then, based on the image-text similarity between the text and the image in each training sample, the training sample pair set is continuously updated using training sample pairs with higher image-text similarity, so that the image-text generation model is trained based on the updated training sample pair set, thereby improving the model training accuracy of the image-text generation model and further improving the consistency of the image and text content generated by the image-text generation model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a training method for a graph-text generation model provided in an embodiment of the present application;

FIG2 is a schematic diagram of an example of a first image in a training method for a picture-text generation model provided in an embodiment of the present application;

FIG3 is a second flow chart of a training method for a graph-text generation model provided in an embodiment of the present application;

FIG4 is a third flow chart of a training method for a graph-text generation model provided in an embodiment of the present application;

FIG5 is a fourth flow chart of a training method for a graph-text generation model provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of a training device for a picture-text generation model provided in an embodiment of the present application;

FIG7 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;

FIG. 8 is a second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated with each other are in an "or" relationship.

The following is an explanation of some concepts and/or terms involved in the training method, device, electronic device and medium of the image-text generation model provided in the embodiments of the present application.

In the following, in conjunction with the accompanying drawings, the training method, device, electronic device and medium of the image-text generation model provided in the embodiments of the present application are described in detail through specific embodiments and their application scenarios.

At present, with the rise and continuous development of artificial intelligence generated content (AIGC), image-text generation models, such as the image-generated diffusion model in the field of AI painting, have been widely used in the fields of wallpaper, avatar, game, animation design, etc., with the advantages of high efficiency and high degree of automation. Among them, the image-text generation model is used for image-text conversion, that is, converting the input text into the image corresponding to the text, or converting the input image into the text corresponding to the image.

In the related art, in the process of training the image-text generation model, the training samples can be input into the image-text generation model to perform denoising on the noisy sample images to generate denoised sample images, wherein texts with different text contents correspond to different noisy sample images, and then, according to the first representation vector of the denoised sample image and the second representation vector of the sample text, a first text-image alignment score is obtained, and based on the first text-image alignment score, a first training sample is selected from multiple training samples of the current batch, and according to the original sample image corresponding to the first training sample and the denoised sample image, a first loss function of the image-text generation model is determined, and based on the first loss function, the image-text generation model is adjusted. Then, the image-text generation model is trained using the next batch of training samples until the training is completed.

However, since the current conventional image-text generation model can only generate images from text content, the training process of the above-mentioned image-text generation model can only convert the text content, resulting in low accuracy of the overall training process.

In an embodiment of the present application, by inputting the first training sample pair in the first training sample pair set into the first image-text generation model, the second training sample pair is output, the first image-text generation model is trained based on the first training sample pair set, the first training sample pair includes a first image and a first text for describing the image content of the first image, the second training sample pair includes a second image and a second text, the second image is an image obtained by image-text conversion of the first text, and the second text is a text obtained by image-text conversion of the first image; based on the first training sample pair and the second training sample pair, M training sample pairs are generated, the M training sample pairs include at least the first training sample pair and the second training sample pair, and M is an integer greater than 1; the first training sample pair in the first training sample pair set is replaced by the third training sample pair to obtain the second training sample pair set, the third training sample pair is the training sample pair with the highest image-text similarity among the M training sample pairs; the first image-text generation model is trained based on the second training sample pair set to obtain the target image-text generation model. In this way, one or more training sample pairs in the training sample set are input into the image-text generation model for image-text conversion to obtain new training sample pairs. Then, based on the image-text similarity between the text and the image in each training sample, the training sample pair set is continuously updated using training sample pairs with higher image-text similarity, so that the image-text generation model is trained based on the updated training sample pair set, thereby improving the model training accuracy of the image-text generation model and further improving the consistency of the image and text content generated by the image-text generation model.

The present application embodiment provides a training method for a graph-text generation model. FIG1 shows a flow chart of a training method for a graph-text generation model provided by the present application embodiment. The method can be applied to a training device for a graph-text generation model. In the present application embodiment, the training device for a graph-text generation model can be an electronic device as an example.

As shown in FIG. 1 , the training method of the image-text generation model provided in the embodiment of the present application may include the following steps 201 to 204 .

Step 201: The electronic device inputs a first training sample pair in a first training sample pair set into a first image-text generation model, and outputs a second training sample pair.

In an embodiment of the present application, the first image-text generation model is obtained by training based on a first training sample pair set.

In an embodiment of the present application, the above-mentioned first image and text generation model can be a convolutional neural network model.

In the embodiment of the present application, the first training sample pair set may be automatically acquired by the electronic device or selected by the user.

In the embodiment of the present application, the first training sample pair set includes N training sample pairs, each training sample pair includes: an image and a text for describing the image, wherein N is a positive integer.

In the embodiment of the present application, the first training sample pair includes a first image and a first text for describing the image content of the first image.

For example, as shown in FIG2 , if the image content of the first image is FIG2 , the first text may be a kitten, a kitten fishing, or fishing.

It should be noted that the above-mentioned first training sample pair is any training sample pair in the first training sample pair set.

In the embodiment of the present application, the second training sample pair includes a second image and a second text.

In the embodiment of the present application, the second image is an image obtained by converting the first text into text. For example, taking the first text as a kitten, the second image can be any image related to a kitten.

In the embodiment of the present application, the second text is the text obtained by converting the first image into text. For example, taking the image content of the first image as Figure 2 as an example, the second text can be a kitten, a kitten fishing, or fishing.

Step 202: The electronic device generates M training sample pairs based on the first training sample pair and the second training sample pair.

In the embodiment of the present application, the M training sample pairs include at least a first training sample pair and a second training sample pair, and M is an integer greater than 1.

In the embodiment of the present application, each of the M training sample pairs includes an image and text for describing the image content of the image.

Optionally, in an embodiment of the present application, the M training sample pairs include: the first training sample pair, the second training sample pair, the fourth training sample pair and the fifth training sample pair.

In an embodiment of the present application, the first training sample pair includes a first image and a first text, the second training sample pair includes a second image and a second text, the fourth training sample pair includes a first image and a second text, and the fifth training sample pair includes a second image and a first text.

Step 203: The electronic device replaces the first training sample pair in the first training sample pair set with the third training sample pair to obtain a second training sample pair set.

In the embodiment of the present application, the third training sample pair is the training sample pair with the highest image-text similarity among the M training sample pairs.

In an embodiment of the present application, the electronic device calculates the image-text similarity between the text and the image in each training sample pair in the above-mentioned M training samples, and then, based on the image-text similarity between the text and the image in each training sample pair in the above-mentioned M training samples, selects the training sample pair with the highest image-text similarity from the above-mentioned M training samples as the third training sample pair.

In the embodiment of the present application, the image-text similarity between the text and the image in the above training sample pair is calculated by using the overall loss function in the image-text generation model.

Optionally, in an embodiment of the present application, the overall loss function in the above-mentioned first image-text generation model can be constructed based on at least one of the following: a text encoder model, an image encoder model, a diffusion model, a text decoder model, etc. in the above-mentioned first image-text generation model.

Optionally, in an embodiment of the present application, each time the training of the image-text generation model is completed, a new overall loss function is constructed based on the trained image-text generation model, and the next time the image-text generation model is trained, the new overall loss function can be used to calculate the image-text similarity between the text and the image in the training sample pair.

In this way, the overall loss function is constructed through the characteristics of different modules in the image-text generation model, so that the constructed overall loss function can fit the image-text generation model, and the image-text similarity of the training sample pairs calculated by the electronic device using the overall loss function is more accurate.

Step 204: The electronic device trains the first image-text generation model based on the second training sample pair set to obtain a target image-text generation model.

Optionally, in an embodiment of the present application, the electronic device continuously constructs an overall loss function based on the second training sample pair set to perform gradient backpropagation on each training sample pair, thereby updating the parameters of the image and text generation model. When the second training sample pair set has been fully input into the first image and text generation model, it indicates that a complete training cycle is currently completed. If the number of the current training cycle reaches the preset number of times, the target image and text generation model is directly output; if the number of the current training cycle does not reach the preset number of times, it means that the target image and text generation model is not yet mature. Therefore, the electronic device continues to replace the current training sample pair set for training until the training cycle reaches the preset number of times, and finally uses the image and text generation model as the target image and text generation model.

Optionally, in the embodiment of the present application, the above step 204 specifically includes steps 204a to 204e:

Step 204a: The electronic device trains the first image-text generation model based on the second training sample pair set to obtain a second image-text generation model.

In an embodiment of the present application, the second image-text generation model is obtained by training based on a second training sample pair set.

In an embodiment of the present application, the second image and text generation model may be a convolutional neural network model.

Exemplarily, the electronic device inputs the training sample pairs in the second training sample pair set into the first image-text generation model, constructs an overall loss function, performs gradient backpropagation on each training sample pair in the second training sample pair set, updates the parameters of the first image-text generation model, and obtains the second image-text generation model.

Step 204b: When the number of model training times does not reach a preset threshold, the electronic device inputs the sixth training sample pair in the second training sample pair set into the second image-text generation model, and outputs the seventh training sample pair.

In the embodiment of the present application, the sixth training sample pair is any training sample pair in the second training sample pair set.

In the embodiment of the present application, the above-mentioned preset threshold is set by the electronic device or set by the user.

Step 204c: The electronic device generates N training sample pairs based on the sixth training sample pair and the seventh training sample pair.

In the embodiment of the present application, the N training sample pairs include at least the sixth training sample pair and the seventh training sample pair, and N is an integer greater than 1.

Step 204d: The electronic device replaces the sixth training sample pair in the second training sample pair set with the eighth training sample pair to obtain a third training sample pair set.

In the embodiment of the present application, the eighth training sample pair is the training sample pair with the highest image-text similarity among the N training sample pairs.

Step 204c: the electronic device trains the second image-text generation model based on the third training sample pair set to obtain a third image-text generation model, iterates the above process until the number of model training times reaches a preset threshold, and then uses the image-text generation model obtained by the last training as the target image-text generation model.

The following takes the first image as M1, the first text as T1, the second image as M2, and the second text as T2 as an example to exemplify the model training method provided in the embodiment of the present application. Specifically, the above-mentioned model training method may include the following steps B1 to B6:

Step B1: The electronic device selects a training sample pair <T1, M1> from a training sample pair set.

Step B2: The electronic device uses the image-text generation model that has completed one cycle of training to perform image-text conversion processing on the training sample pair <T1, M1>, and outputs the training sample pair <T2, M2>.

Step B3, the electronic device uses the overall loss function to calculate the image-text similarity between the images and texts in the four groups of training sample pairs <T1, M1>, <T1, M2>, <T2, M1>, and <T2, M2>. At the same time, by constructing a new overall loss function, the gradient backpropagation is performed on the four groups of training sample pairs.

Step B4: The electronic device selects a training sample pair with the highest image-text similarity, assuming it is <T2, M1>, and uses the training sample pair to update and replace <T1, M1>.

Step B5: When the electronic device traverses the entire set of training sample pairs, a training cycle is completed.

Step B6: The electronic device determines whether the current number of training cycles reaches the maximum number of training cycles. If so, the final image and text generation model is directly output; if not, steps B1 to B5 are repeated.

In this way, the image-text generation model is trained by continuously replacing the set of training sample pairs, thereby improving the consistency of the image and text output by the image-text generation model.

In the training method of the image-text generation model provided in the present application, one or more training sample pairs in the training sample set are input into the image-text generation model for image-text conversion to obtain new training sample pairs, and then, based on the image-text similarity between the text and the image in each training sample, the training sample pair set is continuously updated using training sample pairs with higher image-text similarity, so that the image-text generation model is trained based on the updated training sample pair set, thereby improving the model training accuracy of the image-text generation model, and thereby improving the consistency of the image and text content generated by the image-text generation model.

Optionally, in an embodiment of the present application, the above-mentioned image-text generation model includes at least: a text encoder model, an image encoder model, a diffusion model, an image decoder model, and a text decoder model.

Exemplarily, the above-mentioned text encoder model is used to encode text, that is, to extract text feature information of the text.

Exemplarily, the above-mentioned text encoder model includes: a tokenizer module and a self-attention module.

In one example, the Tokenizer module is used to convert text into a numerical vector corresponding to the text.

In one example, the self-attention module is used to extract feature information from text.

Exemplarily, the above-mentioned image encoder model is used to encode an image, that is, to extract image feature information of the image.

Exemplarily, the above-mentioned image encoder model includes: a first convolution module, a second convolution module, and a third convolution module.

Exemplarily, the above diffusion model is used to convert text feature information into image feature information.

In one example, the first convolution module includes: a first convolution layer, a second convolution layer, and a third convolution layer; the second convolution module includes: a first convolution layer, a second convolution layer, and a third convolution layer; the third convolution module includes: a first convolution layer, a second convolution layer, and a third convolution layer.

In one example, the convolution kernel of the first convolution layer is 3×3, the stride is 2, and the number of output channels is 8; the convolution kernel of the second convolution layer is 3×3, the stride is 1, and the number of output channels is 8; the convolution kernel of the third convolution layer is 3×3, the stride is 1, and the number of output channels is 8.

Exemplarily, the above diffusion model includes: a diffusion model encoder, a diffusion model intermediate layer, and a diffusion model decoder.

In one example, the diffusion model encoder includes: a first encoding module, a second encoding module and a third encoding module.

In an example, the diffusion model decoder includes: a first decoding module, a second decoding module and a third decoding module.

It should be noted that the above-mentioned first encoding module, second encoding module, third encoding module, first decoding module, second decoding module and third decoding module all include: CrossAttention operator (CrossAttention), Add&Norm, and FeedForward.

Exemplarily, the above-mentioned image decoder model is used to convert image feature information into a final image.

In one example, the image decoder model includes: a first deconvolution module, a second deconvolution module, and a third deconvolution module.

In one example, the first deconvolution module includes: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer; the second deconvolution module includes: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer; the third deconvolution module includes: a first deconvolution layer, a second deconvolution layer, and a third deconvolution layer.

In one example, the deconvolution kernel of the first deconvolution is 3×3, the step size is 2, and the number of output channels is 16; the deconvolution kernel of the second deconvolution layer is 3×3, the step size is 1, and the number of output channels is 8; the deconvolution kernel of the third deconvolution layer is 3×3, the step size is 1, and the number of output channels is 8.

Exemplarily, the text decoder is used to convert image feature information into text.

Optionally, in the embodiment of the present application, the above step 201 specifically includes the following steps 201a to 201f:

Step 201a: The electronic device inputs a first training sample pair in a first training sample pair set into a first image-text generation model.

Step 201b: The electronic device inputs the first text into a text encoder model for feature extraction to obtain first text feature information of the first text.

Exemplarily, the first text feature information may be represented in the form of a numerical vector.

For example, as shown in FIG3, taking the first text as an example, first, the electronic device inputs the first text into the Tokenizer module to obtain a first numerical vector with a vector dimension of [128,768]. Among them, 128 represents the maximum number of character tokens, and 768 is the representation vector of each character. Then, the electronic device inputs the first numerical vector of [128,768] dimensions as the QKV value into the self-attention module, extracts the feature vector in the first numerical vector through the self-attention operator (SelfAttention), and obtains the key text feature information of the first text, calculates the mean and standard deviation corresponding to the key text feature information through the vector addition & normalization operator (Add&Norm), and performs feature information fusion processing on the mean and standard deviation corresponding to the key text feature information through the forward feedback operator (FeedForward), repeats the calculation process 12 times, and obtains a [128,768]-dimensional text encoding vector (Text Encoder Vector, TCV), that is, obtains a more fully described first text feature information.

Exemplarily, the above QKV value contains three input vectors of the attention mechanism, representing Query, Key and Value respectively. In the self-attention module SelfAttention, these three vectors come from the same input mapping; in CrossAttention, Query is mapped as an independent input, and Key and Value come from the same input mapping.

Step 201c: The electronic device inputs the first image into an image encoder model to perform feature extraction to obtain first image feature information of the first image.

Exemplarily, the image of the first image can be expressed as [X, Y, Z], where X represents the number of color channels of the first image, and Y and Z represent the numerical values of each pixel of the image.

Exemplarily, the first image feature information may be represented in the form of a vector matrix.

For example, as shown in Figure 4, for the first convolution module in the image encoder model, the electronic device inputs the first image with a vector dimension of [3,512,512] into the first convolution module, and calculates the feature matrix of [8,256,256] through the first convolution layer. Then, the feature matrix with a vector dimension of [8,256,256] passes through an activation function Relu, and still obtains a feature matrix of [8,256,256] dimensions. Next, the electronic device inputs the feature matrix of [8,256,256] dimensions into the second and third convolution layers in sequence. The calculation process is the same as that of the first convolution layer, but the convolution step size is changed to 1, so that the final feature matrix of [8,256,256] dimensions is obtained.

It should be noted that the electronic device inputs the [8, 256, 256]-dimensional feature matrix obtained by the output of the first convolution module into the second convolution module and the third convolution module in sequence. The calculation process is the same as that of the first convolution module, and finally obtains a [32, 64, 64]-dimensional image feature encoding matrix (Image Encoder Matrix, IEM), that is, the above-mentioned first image feature information.

Optionally, after obtaining the first text feature information, that is, after step 201b, the electronic device may perform time mapping processing on the first text feature information to obtain a text condition control vector (TCV).

For example, in conjunction with Figure 3, the electronic device first passes the first text feature information [128,768] dimension through the linear mapping layer Project to obtain a text coding vector (Text Project Vector, TPV) with a dimension of [320,768]. Then the initialization time Embedding passes through the linear mapping layer Project to obtain a time mapping Embedding with a dimension of [320,768]. Finally, the TPV and the time mapping Embedding are added to obtain a TCV with a dimension of [320,768].

Optionally, after obtaining the first image feature information, that is, after step 201c, the electronic device may add noise processing to the first image feature information, and perform convolution calculation on the processed first image feature information to obtain an image coding mapping vector (Image Project Vector, IPV).

Exemplarily, the above-mentioned “add noise processing” (ADDNoise processing) can be to add a random Gaussian noise matrix (Gaussian Noise Matrix, GNM) to the above-mentioned first image feature information to obtain the image latent matrix Latent after noise addition.

Exemplarily, the electronic device performs a convolution operation of Conv on the above Latent to obtain a matrix with a dimension of [320, 64, 64]. Here, the convolution kernel size of the convolution operation is 3x3, the step size is 1, and the number of output channels is 320. The matrix is then reshaped by a reshape function to obtain an IPV with a dimension of [320, 64, 64].

It should be noted that the above two examples are not sequentially processed. The first text feature information may be processed first, and then the first image feature information. The first image feature information may be processed first, and then the first text feature information. The two examples may also be processed simultaneously. This application does not limit this.

Step 201d: input the first text feature information and the first image feature information into the diffusion model to perform a cross attention mechanism and convolution calculation to obtain the second image feature information.

Exemplarily, the second image feature information is the image feature information corresponding to the first text.

For example, for the first encoding module in the diffusion model encoder in the diffusion model, the processing process of the first encoding module may include the following S1 to S7:

S1) The electronic device uses the first image feature information as a query and the first text feature information as a key and value to input into the first encoding module, performs a cross-attention mechanism calculation, extracts the associated image feature information from the first image feature information and the first text feature information through CrossAttention, and then calculates the mean and standard deviation values corresponding to the associated image feature information through the vector addition & normalization operator (Add&Norm). Then, the mean and standard deviation values corresponding to the associated image feature information are processed by the forward feedback operator (FeedForward) for feature information fusion. After repeating the calculation twice, a vector with a vector dimension of [320,64,64] is output. Next, the vector is subjected to a convolution operation with a convolution kernel of 3×3, a step size of 2, and an output channel of 640 to obtain a vector with a dimension of [640,32,32]. Then, the vector is used as the query of the second encoding module for another calculation. The calculation process is the same as that of the first encoding module. After repeating the calculation twice, a vector with a dimension of [320,64,64] is obtained.

S2) Perform convolution operation on the vector of dimension [320,64,64] again, and output a vector of dimension [640,32,32], which is used as the query of the third encoding module for further operation. The operation process is the same as that of the first encoding module. After repeating the calculation twice, a vector of dimension [640,32,32] is obtained. Perform convolution operation again to output a vector of dimension [1280,16,16].

S3) Input the vector with the dimension [1280, 16, 16] into the middle layer of the diffusion model. The calculation process of the middle layer of the diffusion model is the same as that of the first encoding module. It is calculated by CrossAttention, Add&Norm, and FeedForward in turn, and outputs a vector with the dimension [1280, 16, 16].

S4) The vector output by the third encoding module and the vector output by the intermediate layer of the diffusion model are averaged as the query, and the first text feature information as the key and value are input into the first decoding module for calculation. The calculation process is the same as that of the first encoding module. After repeating the calculation twice, a vector of [1280,16,16] dimensions is obtained. Then, the vector of [1280,16,16] dimensions is subjected to a deconvolution operation with a deconvolution kernel of 3×3, a step size of 2, and a channel number of 640, and the output vector dimension is a vector of [640,32,32].

S5) The vector output by the first decoding module and the vector output by the second encoding module are averaged and used as the query, and the first text feature information as the key and value are input into the second decoding module for calculation. The calculation process is the same as that of the first encoding module. After repeating the calculation twice, a vector of [640, 32, 32] dimensions is obtained. A deconvolution operation is then performed to output a vector of [320, 32, 32] dimensions.

S6) The vector output by the second decoding module and the vector output by the first encoding module are averaged and used as the query, and the first text feature information as the key and value are input into the third decoding module for calculation. The calculation process is the same as that of the first encoding module. After repeating the calculation twice, a vector with a vector dimension of [320, 64, 64] is output.

S7) The vector is calculated through a convolution layer (Conv_out) to obtain a [32, 64, 64]-dimensional vector, which is recorded as the image prediction noise matrix, that is, the image noise matrix predicted according to the first text. Finally, the image noise matrix is subtracted from the Latent to obtain the final image reconstruction matrix (Image Reconstruction Matrix, IRM), that is, the third image feature information mentioned above.

Step 201e: The electronic device inputs the second image feature information into the image decoder model to perform image decoding to obtain third image feature information, and outputs the second image based on the third image feature information.

For example, for the first deconvolution module, the electronic device can input the second image feature information with a vector dimension of [32, 64, 64] into the first deconvolution module, and calculate the feature matrix of [16, 128, 128] dimension through the first deconvolution layer, and then pass the feature matrix of [16, 128, 128] vector dimension through an activation function Relu, and still obtain the feature matrix of [8, 256, 256] dimension. Then, the feature matrix of [16, 128, 128] dimension is input into the second deconvolution layer and the third deconvolution layer in turn, and the calculation process is the same as the first deconvolution layer, but the deconvolution step size is changed to 1, and the output channel is changed to 8, so that the feature matrix of [16, 128, 128] dimension is finally obtained.

It should be noted that the [16, 128, 128]-dimensional feature matrix obtained by the output of the first deconvolution module is input into the second deconvolution module and the third deconvolution module in sequence. The calculation process is the same as that of the first deconvolution module, and finally the [3, 512, 512]-dimensional third image feature information is obtained.

Exemplarily, the electronic device converts the feature vector into the second image for output based on the feature vector corresponding to the feature information of the third image.

Step 201f: The electronic device inputs the first image feature information into a text decoder model to perform text prediction, obtains text prediction parameters, and outputs a second text based on the text prediction parameters.

Exemplarily, the first image feature information is input into a text decoder model, and text codebook information having a mapping relationship with the first image feature information is obtained, and prediction parameters are obtained based on the text codebook information.

Exemplarily, the prediction parameter is the probability that the text of the text content reflected by the first image feature information belongs to different preset texts.

Exemplarily, the above text codebook information is a text codebook provided by the system.

Exemplarily, the preset text is the text in the text codebook information.

Exemplarily, there are 12 text decoding modules in the above text decoder model, each of which includes: CrossAttention, Add&Norm, and FeedForward.

For example, the electronic device uses the first text codebook information in the text codebook information with a dimension of [1,768] as the Query value, and the first image feature information as the Key and Value values to input into the first decoding module, and performs cross-attention calculation to obtain the associated feature information between the text codebook information and the first image feature information, and then calculates the mean and standard deviation values corresponding to the associated feature information between the text codebook information and the first image feature information through Add&Norm, and performs feature information fusion processing on the mean and standard deviation values corresponding to the associated feature information between the text codebook information and the first image feature information through FeedForward. The fused vector is then input into the remaining 11 decoding modules for calculation in sequence, and the calculation process is the same, and finally the first text vector of [1,768] dimensions is obtained.

Then, the vocabulary vector matrix corresponding to the first text vector of dimension [1,768] is obtained through the vocabulary (TokenEmbedding), and then a layer of Softmax calculation is performed to select the text character with the highest probability as the first text character of the second text. After the text character is concatenated to the first image feature information, a vector of dimension [33,768] is obtained as the Key and Value value, and then the second text codebook information in the text codebook information of [1,768] is used as the Query value to be input into the first decoding module again for cross attention calculation, and then after Add, Norm, FeedForward operations, 12 decoding modules are calculated to finally obtain the second text vector of dimension [1,768]. Then, the second text character of the second text is obtained through the vocabulary and Softmax calculation. In this way, the above iterative word-by-word calculation is stopped until the set maximum text length or terminator is reached, and the final string is output as the second text.

Optionally, in an embodiment of the present application, after acquiring the first image feature information, the electronic device may map the first image feature information to the same dimension as the first text feature information, and then perform text prediction on the mapped input text decoder model to obtain text prediction parameters.

For example, the first image feature information is a vector of dimension [32, 64, 64]. The electronic device converts the vector of the first image feature information into a vector of dimension [32, 4096] through a reshaping function, and then maps the vector through a linear mapping layer Project to obtain a vector of dimension [32, 768] with the same vector dimension as the first text feature information, that is, the fourth image feature information.

It should be noted that the fourth image feature information can be input into the text decoder model as the first image feature information.

In this way, by using various network models in the image-text generation model, the text and image are fused and associated for training, which can improve the consistency between the generated text and image.

Optionally, in the embodiment of the present application, before the above step 202, the training method of the image-text generation model provided in the embodiment of the present application further includes steps 301 to 304:

Step 301: The electronic device constructs a first loss function based on first text feature information and first image feature information.

Exemplarily, the process of constructing the first loss function includes the following steps A1 to A3:

Step A1: The electronic device uses a linear mapping layer Project to obtain a [256,768]-dimensional text alignment vector TAV.

Step A2: The electronic device calculates the first image feature information through the convolution layer Conv to obtain an image alignment matrix IAM of [768, 16, 16] dimensions, and then reshapes it into an image alignment vector IAV of [256, 768] dimensions through a reshaping function Reshape.

Step A3: The electronic device performs Cosin similarity calculation on the text alignment vector TAV and the image alignment vector IAV to construct a text-image comparison loss function, which is recorded as C1=1-Cos(TAV,IAV).

Step 302: The electronic device constructs a second loss function based on the second image feature information and the Gaussian noise matrix.

Exemplarily, the second loss function construction process is as follows: the electronic device constructs an MSE loss function using a randomly sampled Gaussian noise matrix GNM and an image prediction noise matrix INM obtained based on feature information of the first text and the first image, which is recorded as C2. In this way, the image prediction noise matrix INM approximates the randomly sampled Gaussian noise matrix GNM, so that the image can be reconstructed as close to the original image as possible.

Step 303: The electronic device constructs a third loss function based on the text prediction parameters.

Exemplarily, the above text prediction parameter may be a probability value of the predicted text character being a certain text character.

Exemplarily, taking the i-th training sample pair as an example, the construction process of the third loss function is as follows: The embedding corresponding to the kth text character representing the first image feature information of the i-th sample (32 in total), The jth embedding representing the first image feature information of the i-th sample (the maximum length of the text is 128). Assume that after the second text character is predicted and calculated by softmax, the embedding corresponding to the text character (denoted as ) has the largest softmax value, that is, the probability value of pθ(c2i|p1i,…,p32i,c1i) is higher than the probability values of other characters in the vocabulary, so the third loss function is constructed as follows:

Step 303: The electronic device constructs an overall loss function based on the first loss function, the second loss function and the third loss function.

In the embodiment of the present application, the above overall loss function can be:

C = a × C1 + b × C2 + c × C3 (a + b + c = 1) Formula 1

Wherein, C1 represents the first loss function constructed based on the text encoder model and the image encoder model in the above-mentioned image-text generation model, C2 represents the second loss function constructed based on the diffusion model in the above-mentioned image-text generation model. C3 represents the third loss function constructed based on the text decoder model in the above-mentioned image-text generation model. a, b, and c represent the weights of the three loss functions respectively, and the sum of the three weights is 1.

It should be noted that the weights of the above three loss functions can be preset weights.

Optionally, in the embodiment of the present application, in combination with the above steps 301 to 304, after the above step 202, the training method of the image-text generation model provided in the embodiment of the present application further includes step 305:

Step 305: The electronic device calculates the image-text similarity of each training sample pair in the M training sample pairs using the overall loss function.

It should be noted that, as shown in FIG5 , the above-mentioned image-text generation model may include four modules: an input module, an inference module, an output module, and a training loss function module. During the training process, the electronic device will first input the training sample pair into the input module of the image-text generation model, and then go through the inference process of the inference module of the image-text generation model, and then output the final image and text through the output module of the image-text generation model. At the same time, an overall loss function will be created in the intermediate process of the inference module and the final output result for training and learning.

Optionally, in the embodiment of the present application, after the above step 204, the training method of the image-text generation model provided in the embodiment of the present application further includes the following step 401 or step 402:

Step 401: The electronic device inputs a fourth image into a target image-text generation model for image-text conversion, and outputs a fourth text.

Exemplarily, the fourth text is used to describe the image content of the fourth image.

Step 402: The electronic device inputs the fifth text into the target image-text generation model for image-text conversion, and outputs a fifth image.

Exemplarily, the fifth image is generated based on the fifth text.

It should be noted that the training method of the image-text generation model provided in the embodiment of the present application can be executed by a training device for the image-text generation model, or an electronic device, or a functional module or entity in the electronic device. In the embodiment of the present application, the training method of the image-text generation model executed by the training device for the image-text generation model is taken as an example to illustrate the training device for the image-text generation model provided in the embodiment of the present application.

Fig. 6 shows a possible structural diagram of a training device for a graph-text generation model involved in an embodiment of the present application. As shown in Fig. 6, the training device 700 for a graph-text generation model may include: a processing module 701, a generation module 702, a replacement module 703, and a training module 704;

The processing module 701 is used to input the first training sample pair in the first training sample pair set into the first image-text generation model, and output the second training sample pair, wherein the first image-text generation model is obtained by training based on the first training sample pair set, the first training sample pair includes a first image and a first text for describing the image content of the first image, the second training sample pair includes a second image and a second text, the second image is an image obtained by image-text conversion of the first text, and the second text is a text obtained by image-text conversion of the first image; the generation module 702 is used to generate a second training sample pair based on the first training sample pair and the second training sample pair. Sample pairs, generate M training sample pairs, the M training sample pairs include at least a first training sample pair and a second training sample pair, and M is an integer greater than 1; the above-mentioned replacement module 703 is used to replace the first training sample pair in the above-mentioned first training sample pair set with a third training sample pair to obtain a second training sample pair set, and the third training sample pair is the training sample pair with the highest image-text similarity among the M training sample pairs generated by the above-mentioned generation module 702; the above-mentioned training module 704 is used to train the first image-text generation model based on the second training sample pair set replaced by the above-mentioned replacement module 703 to obtain a target image-text generation model.

Optionally, in an embodiment of the present application, the above-mentioned image-text generation model includes a text encoder model, an image encoder model, a diffusion model, an image decoder model and a text decoder model;

The processing module 701 is specifically used for:

Input the first training sample pair in the first training sample pair set into the first image-text generation model; input the first text into the text encoder model for feature extraction to obtain first text feature information of the first text; input the first image into the image encoder model for feature extraction to obtain first image feature information of the first image; input the first text feature information and the first image feature information into the diffusion model for cross attention mechanism and convolution calculation to obtain second image feature information; input the second image feature information into the image decoder model for image decoding to obtain third image feature information, and output the second image based on the third image feature information; input the first image feature information into the text decoder model for text prediction to obtain text prediction parameters, and output the second text based on the text prediction parameters.

Optionally, in an embodiment of the present application, the above-mentioned processing module 701 is also used to construct a first loss function based on the first text feature information and the first image feature information before generating M training sample pairs based on the first training sample pair and the second training sample pair; the above-mentioned processing module 701 is also used to construct a second loss function based on the second image feature information and the Gaussian noise matrix; the above-mentioned processing module 701 is also used to construct a third loss function based on text prediction parameters; the above-mentioned processing module 701 is also used to construct an overall loss function based on the first loss function, the second loss function and the third loss function; the above-mentioned processing module 701 is also used to generate M training sample pairs based on the first training sample pair and the second training sample pair, and then use the overall loss function to calculate the image-text similarity of each training sample pair in the M training sample pairs.

Optionally, in an embodiment of the present application, the above-mentioned M training sample pairs include: a first training sample pair, a second training sample pair, a fourth training sample pair and a fifth training sample pair; wherein the fourth training sample pair includes a first image and a second text, and the fifth training sample pair includes a second image and a first text.

Optionally, in the embodiment of the present application, the training module 704 is specifically used for:

Based on the second training sample pair set, the first image-text generation model is trained to obtain the second image-text generation model; when the number of model training times does not reach the preset threshold, the sixth training sample pair in the second training sample pair set is input into the second image-text generation model, and the seventh training sample pair is output; based on the sixth training sample pair and the seventh training sample pair, N training sample pairs are generated, and the N training sample pairs include at least the sixth training sample pair and the seventh training sample pair, and N is an integer greater than 1; the sixth training sample pair in the second training sample pair set is replaced by the eighth training sample pair to obtain a third training sample pair set, and the eighth training sample pair is the training sample pair with the highest image-text similarity among the N training sample pairs; based on the third training sample pair set, the second image-text generation model is trained to obtain the third image-text generation model, and the above process is iterated until the number of model training times reaches the preset threshold, and the image-text generation model obtained by the last training is used as the target image-text generation model.

Optionally, in an embodiment of the present application, the above-mentioned processing module 701 is also used to train the first image-text generation model based on the second training sample pair set, and after obtaining the target image-text generation model, input the fourth image into the target image-text generation model for image-text conversion, and output a fourth text, where the fourth text is used to describe the image content of the fourth image; or, input the fifth text into the target image-text generation model for image-text conversion, and output a fifth image, where the fifth image is generated based on the fifth text.

In a training device for a picture-text generation model provided in an embodiment of the present application, the device inputs a first training sample pair in a first training sample pair set into a first picture-text generation model, and outputs a second training sample pair, wherein the first picture-text generation model is obtained by training based on the first training sample pair set, the first training sample pair includes a first image and a first text for describing the image content of the first image, the second training sample pair includes a second image and a second text, the second image is an image obtained by picture-text conversion of the first text, and the second text is a text obtained by picture-text conversion of the first image; based on the first training sample pair and the second training sample pair, M training sample pairs are generated, the M training sample pairs include at least the first training sample pair and the second training sample pair, and M is an integer greater than 1; the first training sample pair in the first training sample pair set is replaced by a third training sample pair to obtain a second training sample pair set, the third training sample pair is the training sample pair with the highest picture-text similarity among the M training sample pairs; the first picture-text generation model is trained based on the second training sample pair set to obtain a target picture-text generation model. In this way, one or more training sample pairs in the training sample set are input into the image-text generation model for image-text conversion to obtain new training sample pairs. Then, based on the image-text similarity between the text and the image in each training sample, the training sample pair set is continuously updated using training sample pairs with higher image-text similarity, so that the image-text generation model is trained based on the updated training sample pair set, thereby improving the model training accuracy of the image-text generation model and further improving the consistency of the image and text content generated by the image-text generation model.

The training device of the graphic generation model in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

The training device of the image-text generation model in the embodiment of the present application can be a device with an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

The training device for the image-text generation model provided in the embodiment of the present application can implement each process implemented in the method embodiments of Figures 1 to 5. To avoid repetition, they will not be described here.

Optionally, as shown in Figure 7, an embodiment of the present application also provides an electronic device 800, including a processor 801 and a memory 802, and the memory 802 stores a program or instruction that can be executed on the processor 801. When the program or instruction is executed by the processor 801, the various steps of the training method embodiment of the above-mentioned image and text generation model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

FIG8 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 110 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The electronic device structure shown in FIG8 does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently, which will not be described in detail here.

The processor 110 is used to input the first training sample pair in the first training sample pair set into the first image-text generation model, and output the second training sample pair, wherein the first image-text generation model is obtained by training based on the above-mentioned first training sample pair set, the first training sample pair includes a first image and a first text for describing the image content of the first image, and the second training sample pair includes a second image and a second text, the second image is an image obtained by image-text conversion of the above-mentioned first text, and the second text is a text obtained by image-text conversion of the above-mentioned first image; the above-mentioned processor 110 is also used to generate M training sample pairs based on the above-mentioned first training sample pair and the above-mentioned second training sample pair, and the M training sample pairs at least include the first training sample pair and the second training sample pair, and M is an integer greater than 1; the above-mentioned processor 110 is also used to replace the first training sample pair in the above-mentioned first training sample pair set with a third training sample pair to obtain a second training sample pair set, and the third training sample pair is the training sample pair with the highest image-text similarity among the M training sample pairs; the above-mentioned processor 110 is also used to train the first image-text generation model based on the second training sample pair set to obtain a target image-text generation model.

Optionally, in an embodiment of the present application, the above-mentioned image-text generation model includes a text encoder model, an image encoder model, a diffusion model, an image decoder model and a text decoder model;

The processor 110 is specifically configured to:

Input the first training sample pair in the first training sample pair set into the first image-text generation model; input the first text into the text encoder model for feature extraction to obtain first text feature information of the first text; input the first image into the image encoder model for feature extraction to obtain first image feature information of the first image; input the first text feature information and the first image feature information into the diffusion model for cross attention mechanism and convolution calculation to obtain second image feature information; input the second image feature information into the image decoder model for image decoding to obtain third image feature information, and output the second image based on the third image feature information; input the first image feature information into the text decoder model for text prediction to obtain text prediction parameters, and output the second text based on the text prediction parameters.

Optionally, in an embodiment of the present application, the processor 110 is further used to construct a first loss function based on the first text feature information and the first image feature information before generating M training sample pairs based on the first training sample pair and the second training sample pair; the processor 110 is further used to construct a second loss function based on the second image feature information and a Gaussian noise matrix; the processor 110 is further used to construct a third loss function based on text prediction parameters; the processor 110 is further used to construct an overall loss function based on the first loss function, the second loss function and the third loss function; the processor 110 is further used to calculate the image-text similarity of each training sample pair in the M training sample pairs after generating M training sample pairs based on the first training sample pair and the second training sample pair.

Optionally, in an embodiment of the present application, the above-mentioned M training sample pairs include: a first training sample pair, a second training sample pair, a fourth training sample pair and a fifth training sample pair; wherein the fourth training sample pair includes a first image and a second text, and the fifth training sample pair includes a second image and a first text.

Optionally, in the embodiment of the present application, the processor 110 is specifically configured to:

Based on the second training sample pair set, the first image-text generation model is trained to obtain the second image-text generation model; when the number of model training times does not reach the preset threshold, the sixth training sample pair in the second training sample pair set is input into the second image-text generation model, and the seventh training sample pair is output; based on the sixth training sample pair and the seventh training sample pair, N training sample pairs are generated, and the N training sample pairs include at least the sixth training sample pair and the seventh training sample pair, and N is an integer greater than 1; the sixth training sample pair in the second training sample pair set is replaced by the eighth training sample pair to obtain a third training sample pair set, and the eighth training sample pair is the training sample pair with the highest image-text similarity among the N training sample pairs; based on the third training sample pair set, the second image-text generation model is trained to obtain the third image-text generation model, and the above process is iterated until the number of model training times reaches the preset threshold, and the image-text generation model obtained by the last training is used as the target image-text generation model.

Optionally, in an embodiment of the present application, the processor 110 is further used to, after training a graph-text generation model based on a set of replaced training samples to obtain a target graph-text generation model, input the fourth image into the target graph-text generation model for graph-text conversion, and output a fourth text, where the fourth text is used to describe the image content of the fourth image; or, input the fifth text into the target graph-text generation model for graph-text conversion, and output a fifth image, where the fifth image is generated based on the fifth text.

In the electronic device provided in the embodiment of the present application, the electronic device inputs the first training sample pair in the first training sample pair set into the first image-text generation model, and outputs the second training sample pair, the first image-text generation model is obtained by training based on the first training sample pair set, the first training sample pair includes a first image and a first text for describing the image content of the first image, the second training sample pair includes a second image and a second text, the second image is an image obtained by image-text conversion of the first text, and the second text is a text obtained by image-text conversion of the first image; based on the first training sample pair and the second training sample pair, M training sample pairs are generated, the M training sample pairs include at least the first training sample pair and the second training sample pair, and M is an integer greater than 1; the first training sample pair in the first training sample pair set is replaced by the third training sample pair to obtain the second training sample pair set, the third training sample pair is the training sample pair with the highest image-text similarity among the M training sample pairs; the first image-text generation model is trained based on the second training sample pair set to obtain a target image-text generation model. In this way, one or more training sample pairs in the training sample set are input into the image-text generation model for image-text conversion to obtain new training sample pairs. Then, based on the image-text similarity between the text and the image in each training sample, the training sample pair set is continuously updated using training sample pairs with higher image-text similarity, so that the image-text generation model is trained based on the updated training sample pair set, thereby improving the model training accuracy of the image-text generation model and further improving the consistency of the image and text content generated by the image-text generation model.

It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the training method embodiment of the above-mentioned graphic generation model is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned training method embodiment of the graphic generation model, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the training method embodiment of the above-mentioned graphic generation model, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims (14)

1. The training method of the image-text generation model is characterized by comprising the following steps of:

inputting a first training sample pair in a first training sample pair set into a first image-text generation model, outputting a second training sample pair, wherein the first image-text generation model is obtained by training based on the first training sample pair set, the first training sample pair comprises a first image and a first text for describing the image content of the first image, the second training sample pair comprises a second image and a second text, the second image is an image obtained by converting the first text into the image, and the second text is a text obtained by converting the first image into the image;

generating M training sample pairs based on the first training sample pair and the second training sample pair, wherein the M training sample pairs at least comprise the first training sample pair and the second training sample pair, and M is an integer greater than 1;

replacing the first training sample pair in the first training sample pair set with a third training sample pair to obtain a second training sample pair set, wherein the third training sample pair is the training sample pair with highest image-text similarity in the M training sample pairs;

And training the first image-text generating model based on the second training sample pair set to obtain a target image-text generating model.

2. The method of claim 1, wherein the first teletext generation model comprises a text encoder model, an image encoder model, a diffusion model, an image decoder model, and a text decoder model;

inputting a first training sample pair in a first training sample pair set into the first image-text generating model, and outputting a second training sample pair, wherein the method comprises the following steps:

inputting the first training sample pair into the first image-text generating model;

inputting the first text into the text encoder model for feature extraction to obtain first text feature information of the first text;

inputting the first image into the image encoder model for feature extraction to obtain first image feature information of the first image;

inputting the first text feature information and the first image feature information into the diffusion model to perform a cross attention mechanism and convolution calculation to obtain second image feature information;

inputting the second image characteristic information into the image decoder model for image decoding to obtain third image characteristic information, and outputting the second image based on the third image characteristic information;

And inputting the first image characteristic information into the text decoder model for text prediction to obtain text prediction parameters, and outputting the second text based on the text prediction parameters.

3. The method of claim 2, wherein prior to generating M training sample pairs based on the first training sample pair and the second training sample pair, the method further comprises:

constructing a first loss function based on the first text feature information and the first image feature information;

constructing a second loss function based on the second image characteristic information and a Gaussian noise matrix;

constructing a third loss function based on the text prediction parameters;

constructing an overall loss function based on the first, second, and third loss functions;

after generating M training sample pairs based on the first training sample pair and the second training sample pair, the method further includes:

and respectively calculating the image-text similarity of each training sample pair of the M training sample pairs by adopting the integral loss function.

4. The method of claim 1, wherein the M training sample pairs comprise: the first training sample pair, the second training sample pair, the fourth training sample pair, and the fifth training sample pair; wherein the fourth training sample pair comprises the first image and the second text, and the fifth training sample pair comprises the second image and the first text.

5. The method according to claim 1, wherein training the first teletext generation model based on the second training sample pair set to obtain a target teletext generation model comprises:

training the first image-text generating model based on the second training sample pair set to obtain a second image-text generating model;

under the condition that the model training times do not reach a preset threshold value, inputting a sixth training sample pair in the second training sample pair set into the second image-text generating model, and outputting a seventh training sample pair;

generating N training sample pairs based on the sixth training sample pair and the seventh training sample pair, wherein the N training sample pairs at least comprise the sixth training sample pair and the seventh training sample pair, and N is an integer greater than 1;

replacing the sixth training sample pair in the second training sample pair set with an eighth training sample pair to obtain a third training sample pair set, wherein the eighth training sample pair is the training sample pair with highest image-text similarity in the N training sample pairs;

and training the second image-text generating model based on the third training sample pair set to obtain a third image-text generating model, iterating the process until the model training times reach the preset threshold value, and taking the image-text generating model obtained by the last training as the target image-text generating model.

6. The method according to claim 1, wherein after training the first teletext generation model based on the second training sample pair set to obtain a target teletext generation model, the method further comprises:

inputting a fourth image into the target image-text generation model for image-text conversion, and outputting a fourth text, wherein the fourth text is used for describing the image content of the fourth image;

or,

and inputting a fifth text into the target image-text generation model for image-text conversion, and outputting a fifth image, wherein the fifth image is generated based on the fifth text.

7. A training device for a pattern-text generation model, the device comprising: the device comprises a processing module, a generating module, a replacing module and a training module;

the processing module is used for inputting a first training sample pair in a first training sample pair set into a first image-text generating model and outputting a second training sample pair, the first image-text generating model is obtained by training based on the first training sample pair set, the first training sample pair comprises a first image and a first text for describing the image content of the first image, the second training sample pair comprises a second image and a second text, the second image is an image obtained by converting the first text into the image, and the second text is a text obtained by converting the first image into the image;

The generating module is configured to generate M training sample pairs based on the first training sample pair and the second training sample pair, where the M training sample pairs at least include the first training sample pair and the second training sample pair, and M is an integer greater than 1;

the replacing module is configured to replace the first training sample pair in the first training sample pair set with a third training sample pair to obtain a second training sample pair set, where the third training sample pair is a training sample pair with highest image-text similarity in the M training sample pairs generated by the generating module;

the training module is used for training the first image-text generating model based on the second training sample pair set replaced by the replacing module to obtain a target image-text generating model.

8. The apparatus of claim 7, wherein the teletext generation model comprises a text encoder model, an image encoder model, a diffusion model, an image decoder model, and a text decoder model;

the processing module is specifically configured to:

inputting a first training sample pair in the first training sample pair set to the first image-text generating model;

Inputting the first text into the text encoder model for feature extraction to obtain first text feature information of the first text;

inputting the first image into the image encoder model for feature extraction to obtain first image feature information of the first image;

inputting the first text feature information and the first image feature information into the diffusion model to perform a cross attention mechanism and convolution calculation to obtain second image feature information;

inputting the second image characteristic information into the image decoder model for image decoding to obtain third image characteristic information, and outputting the second image based on the third image characteristic information;

and inputting the first image characteristic information into the text decoder model for text prediction to obtain text prediction parameters, and outputting the second text based on the text prediction parameters.

9. The apparatus of claim 8, wherein the processing module is further configured to, prior to generating M training sample pairs based on the first training sample pair and the second training sample pair,

constructing a first loss function based on the first text feature information and the first image feature information;

The processing module is further used for constructing a second loss function based on the second image characteristic information and a Gaussian noise matrix;

the processing module is further used for constructing a third loss function based on the text prediction parameters;

the processing module is further configured to construct an overall loss function based on the first loss function, the second loss function, and the third loss function;

the processing module is further configured to calculate, according to the overall loss function, the image-text similarity of each of the M training sample pairs after generating the M training sample pairs based on the first training sample pair and the second training sample pair.

10. The apparatus of claim 7, wherein the M training sample pairs comprise: the first training sample pair, the second training sample pair, the fourth training sample pair, and the fifth training sample pair; wherein the fourth training sample pair comprises the first image and the second text, and the fifth training sample pair comprises the second image and the first text.

11. The device according to claim 7, wherein the training module is specifically configured to:

Training the first image-text generating model based on the second training sample pair set to obtain a second image-text generating model;

under the condition that the model training times do not reach a preset threshold value, inputting a sixth training sample pair in the second training sample pair set into the second image-text generating model, and outputting a seventh training sample pair;

generating N training sample pairs based on the sixth training sample pair and the seventh training sample pair, wherein the N training sample pairs at least comprise the sixth training sample pair and the seventh training sample pair, and N is an integer greater than 1;

replacing the sixth training sample pair in the second training sample pair set with an eighth training sample pair to obtain a third training sample pair set, wherein the eighth training sample pair is the training sample pair with highest image-text similarity in the N training sample pairs;

and training the second image-text generating model based on the third training sample pair set to obtain a third image-text generating model, iterating the process until the model training times reach the preset threshold value, and taking the image-text generating model obtained by the last training as the target image-text generating model.

12. The apparatus of claim 7, wherein the processing module is further configured to:

after the first image-text generating model is trained based on the second training sample pair set to obtain a target image-text generating model, inputting a fourth image into the target image-text generating model for image-text conversion, and outputting a fourth text, wherein the fourth text is used for describing the image content of the fourth image; or, inputting a fifth text into the target image-text generation model for image-text conversion, and outputting a fifth image, wherein the fifth image is generated based on the fifth text.

13. An electronic device comprising a processor, a memory and a program or instruction stored on the memory and executable on the processor, which when executed by the processor, implements the steps of the training method of the teletext generation model according to any one of claims 1 to 6.

14. A readable storage medium, wherein a program or instructions is stored on the readable storage medium, which when executed by a processor, implements the steps of the training method of the teletext generation model according to any one of claims 1 to 6.