CN113590800A - Training method and device of image generation model and image generation method and device - Google Patents

Abstract

The present application discloses a training method and device for an image generation model, and an image generation method and device, wherein the method includes: acquiring dialogue sample data, the dialogue sample data including dialogue text data, standard images, image description text and dialogue total rounds Based on the dialogue sample data, the interactive incremental image generation model is trained by using the dialogue sample data and the pre-trained heterogeneous recurrent neural network encoder by means of random replay training, so that the interactive The incremental incremental image model can generate interactive incremental images based on human-machine dialogue text and image description text; in which, using all dialogue text data obtained at the final moment of the dialogue and all dialogue text data obtained at the middle moment of the dialogue, the the training. Adopting the present application is beneficial to reasonably realize the task of dialogue-to-image generation.

Description

Training method and device for image generation model and image generation method and device

technical field

The present invention relates to artificial intelligence technology, in particular to an image generation model training method and device, and an image generation method and device.

Background technique

Dialogue is the most natural way for people to communicate, and it is also an ideal way to control the machine to generate images in the form of dialogue in the text-to-image generation task. Dialogue is a free interactive process: things that are difficult to say clearly in a sentence can be completed by adding dialogue rounds, and the subject is not limited.

For the task of dialogue-to-image generation, in order to improve the intelligence of human-machine dialogue, it is necessary for the machine to generate and display images after each round of dialogue as a timely feedback to people, not only after the entire dialogue is over. image.

In the process of realizing this application, the inventor found through research and analysis that: after each round of dialogue is over, directly using the existing text-to-image generation method to generate images cannot reasonably realize the dialogue-to-image generation task. The specific reasons are as follows:

During the dialogue process, the information input by the person will increase with the progress of the dialogue. Accordingly, in order to ensure the rationality and intelligence of dialogue-to-image generation, for each frame of image generated by the machine after each round of dialogue, it contains Information should also be incremental, a property known as the "incrementality" of the image. Thus, ideally, the dialogue-to-image generation process should start with a blank canvas, with information incrementally supplemented by the machine after each round of dialogue. Machines should not draw a complex image of a large number of objects at the beginning of a conversation to avoid giving false feedback to humans. The content of the image generated after each round of dialogue should "just cover all the information of the current dialogue process", no more or less. At the same time, the images generated later should not have huge changes in structure to maintain the coherence of the dialogue. Abstractly, the "incremental" requirement includes the following aspects:

Increase in the number of objects: The number of objects increases monotonically with the dialogue process, and is equal to the number of objects actually involved in the dialogue, and cannot be more or less than the number of objects involved in the dialogue.

Incremental properties and relationships: The properties and relationships of objects are gradually determined with the dialogue process, and can be memorized: in the images generated later in the dialogue, the properties and relationships established in the early stage of the dialogue cannot be lost.

Front-to-back coherence: Images generated in adjacent turns in a dialogue should be similar in structure. The dramatic change of the image during the dialogue is a kind of false feedback that affects the experience of the interlocutor.

Although the amount of information in the dialogue process is naturally increasing, the resulting image content is not necessarily so. Because, in the implementation of the text-to-image generation task, the image modality is not equal to the text modality, the information contained in the image is much larger than the text modality, and the text can only control a small part of the information in the image, while the other part Image information (ie, image-specific information) is randomly generated by machines, which brings uncertainty to image information. Due to the uncertainty of the specific information of the image, the image generated by the text with more information may contain less information than the image generated by the text with less information. In this way, after each round of dialogue is over, an image is generated based on the currently acquired dialogue text, and there is no guarantee that the information in the image will increase with the increase of dialogue rounds, so that the above-mentioned "incremental" requirement of the dialogue-to-image generation task cannot be achieved. , which in turn cannot reasonably achieve the task of dialogue-to-image generation.

SUMMARY OF THE INVENTION

In view of this, the main purpose of the present invention is to provide a training method for an encoder and an image generation model, and an image generation method and apparatus, which are beneficial to reasonably realize the task of generating dialogue to image.

In order to achieve the above purpose, the technical solutions proposed in the embodiments of the present invention are:

A training method for an interactive incremental image generation model, comprising:

Obtaining dialogue sample data, the dialogue sample data includes dialogue text data, standard images, image description texts and the total number of dialogue rounds;

The interactive incremental image generation model is trained by means of random replay training, using the dialogue sample data and the pre-trained heterogeneous recurrent neural network encoder, so that the interactive incremental image model can be based on human-machine dialogue Text and image description text generate images with interactive incrementality; wherein the training is performed using all dialogue text data obtained at the end of the dialogue and all dialogue text data obtained at the middle of the dialogue.

Preferably, the training includes:

Using random sampling, determine the current number of dialogue rounds t, where 2≤t≤T, and T is the total number of dialogues;

Inputting the image description text and the dialogue text data into the heterogeneous recurrent neural network encoder for encoding, and using the feature representation finally output from the encoding as a first text representation X′ _T ; using the first text representation Input to the image generator of the interactive incremental image generation model for image generation to obtain the first image Y′ _T ;

Based on the first text representation X' _T and the first image Y' _T , using the discriminator of the interactive incremental image generation model, a main adversarial loss is calculated; using the main adversarial loss, the interactive The cumulative gradient of the image generator and the discriminator of the incremental image generation model; the main adversarial loss includes the loss function value of the image generator and the discriminator;

Inputting the first t rounds of dialogue data in the image description text and the dialogue text data into the heterogeneous recurrent neural network encoder for encoding, and using the feature representation finally output from the encoding as the second text representation X′ _t ; The second text representation is input to the image generator for image generation to obtain a second image Y′ _t ;

Input the image description text and the first t-1 rounds of dialogue data in the dialogue text data into the heterogeneous recurrent neural network encoder for encoding, and take the feature representation finally output from the encoding as the third text representation _X't -1; input the third text representation to the image generator for image generation to obtain a third image Y′ _t -1;

constructing a first positive example based on the second text representation _X't and the second image _Y't ;

constructing a second positive example based on the third text representation _X't -1 and the third image _Y't -1;

Based on the first positive example, using the discriminator, calculate a first auxiliary adversarial loss; based on the first auxiliary adversarial loss, update the cumulative gradient of the image generator and the discriminator; the first auxiliary adversarial loss The adversarial loss includes the loss function values of the image generator and the discriminator;

Based on the second positive example, using the discriminator, calculate a second auxiliary adversarial loss; based on the second auxiliary adversarial loss, update the cumulative gradient of the image generator and the discriminator; the second auxiliary adversarial loss The adversarial loss includes the loss function values of the image generator and the discriminator;

Based on the current cumulative gradient of the image generator, the parameters of the image generator are updated; based on the current cumulative gradient of the discriminator, the parameters of the discriminator are updated.

Preferably, the training method further includes:

constructing a first negative example based on the third text representation _X't -1 and the second image _Y't ;

constructing a second negative example based on the second text representation _X't and the third image Y't _-1 ;

The calculating the first auxiliary adversarial loss includes:

Based on the first positive example and the first negative example, using the discriminator, calculate the first auxiliary adversarial loss;

The calculating the second auxiliary adversarial loss includes:

Using the discriminator, the second auxiliary adversarial loss is calculated based on the second positive example and the second negative example.

Preferably, the training of the heterogeneous RNN encoder includes:

Obtaining coding training sample data, the coding training sample data includes image description text and visual dialogue text of standard sample images;

The heterogeneous RNN encoder is trained by using the encoded training sample data, so that the heterogeneous RNN encoder can correlate the referential relationship in the visual dialogue text in the input data with the corresponding content in the image description text link.

Preferably, the heterogeneous RNN encoder is composed of a first RNN encoder, a second RNN encoder and a third RNN encoder;

The training of the heterogeneous recurrent neural network encoder using the encoded training sample data includes:

Using the first cyclic neural network encoder, the image description text is encoded with words as the basic coding unit, and the feature representation of each primary word obtained by encoding is output to the third cyclic neural network encoder. Utilize the second cyclic neural network encoder, take the sentence as the basic coding unit, encode the visual dialogue text, and express each primary sentence feature obtained by encoding, and output it to the third cyclic neural network encoding. device;

The third recurrent neural network encoder encodes based on the primary word feature representation and the primary sentence feature representation, and uses the final output feature representation as a global image associated with the visual dialogue text and the image description text. encoded representation;

Based on all the feature representations output by the third recurrent neural network encoder in the encoding process, using the DAMSM loss function of the deep attention multimodal similarity model, the first recurrent neural network encoder, the second recurrent neural network encoder and the second recurrent neural network encoder are used. The weight parameters of the neural network encoder and the third recurrent neural network encoder are adjusted.

Based on the above model training method, an embodiment of the present invention further discloses an interactive incremental image generation method, including:

In the process of visual dialogue, at the end of each round of human-machine dialogue, input all the generated human-machine dialogue texts and preset image description texts into the pre-trained interactive incremental image generation model for image generation and display. the generated image;

Wherein, the interactive incremental image generation model is obtained based on the above-mentioned training method.

The embodiment of the present invention also discloses a training device for an interactive incremental image generation model, comprising: a processor, where the processor is used for:

Obtaining dialogue sample data, the dialogue sample data includes dialogue text data, standard images, image description texts and the total number of dialogue rounds;

The interactive incremental image generation model is trained by means of random replay training, using the dialogue sample data and the pre-trained heterogeneous recurrent neural network encoder, so that the interactive incremental image model can be based on human-machine dialogue Text and image description text generate images with interactive incrementality; wherein the training is performed using all dialogue text data obtained at the end of the dialogue and all dialogue text data obtained at the middle of the dialogue.

The embodiment of the present invention also discloses a non-volatile computer-readable storage medium, where the non-volatile computer-readable storage medium stores instructions, and when executed by a processor, the instructions cause the processor to execute the above-mentioned steps for any of the training methods described.

The embodiment of the present invention also discloses an interactive incremental image generation device, comprising: a processor, and the processor is used for:

In the process of visual dialogue, at the end of each round of human-machine dialogue, input all the generated rounds of human-machine dialogue text and preset image description text into the pre-trained interactive incremental image generation model for image generation, and display the resulting image;

Wherein, the interactive incremental image generation model is obtained based on the above-mentioned training method.

The embodiment of the present invention also discloses a non-volatile computer-readable storage medium, where the non-volatile computer-readable storage medium stores instructions, and when executed by a processor, the instructions cause the processor to execute the above-mentioned Steps of the described interactive incremental image generation method.

To sum up, the above technical solutions proposed by the embodiments of the present invention train the interactive incremental image generation model by adopting random replay training, and during training, not only use all the data obtained at the final moment of the dialogue For dialogue text data, the training is also performed using all dialogue text data obtained in the middle of the dialogue. In this way, by adding the text at the intermediate moment during model training, the model can also perceive the dialogue text at the intermediate moment during the training process, thereby enhancing the interactive incrementality of the model. In addition, by introducing a heterogeneous recurrent neural network encoder during model training, the dialogue text data can be associated with the image description text, so that the encoded text feature vector can accurately represent the image features described by the user's language, which is conducive to random reconstruction. The release training method can accurately capture the incrementality in the interaction process, which in turn enables the trained model to generate images with interactive incrementality. Therefore, by adopting the technical solutions of the embodiments of the present invention, the task of generating dialogues to images can be reasonably realized.

Description of drawings

1 is a schematic flowchart of a method according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a heterogeneous cyclic neural network encoder according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

FIG. 1 is a schematic flowchart of a training method for an interactive incremental image generation model according to an embodiment of the present invention. As shown in FIG. 1 , the embodiment mainly includes:

Step 101: Acquire dialogue sample data, where the dialogue sample data includes dialogue text data, standard images, image description texts, and the total number of dialogue rounds.

Step 102, using the random replay training mode, using the dialogue sample data and the pre-trained heterogeneous recurrent neural network encoder to train the interactive incremental image generation model, so that the interactive incremental image model can be based on Human-machine dialogue text and image description text generate interactive incremental images; wherein the training is performed using all dialogue text data obtained at the end of the dialogue and all dialogue text data obtained in the middle of the dialogue.

In this step, in order to train the model to generate interactive incremental images, the random replay training method is used to train the model, and not only use all the dialogue text data obtained at the final moment of the dialogue to train the model, but also need to Use all the dialogue text data obtained in the middle of the dialogue to train the model, so that the model can also perceive the text at the middle moment during the training process, so that the ability of the model to realize the incrementality of images can be improved.

It should be noted here that in the existing model training algorithm based on dialogue generation images, all dialogue text data of the complete dialogue process and the images generated at the last moment of the dialogue are used as a pair of training samples to train the model. The goal of this training is to ensure the accuracy of images generated at the end of the conversation. Since the model observes all the dialogue text and the final image during training, and has not seen the dialogue and image samples at the intermediate moments, the images generated by the model based on the text at the intermediate moments may be incorrect. In this way, if such a model with the final dialogue image as the training target is applied to a task that requires generating images for each round of dialogue, it is necessary to generate not only images at the end of the dialogue, but also images at the middle of the dialogue. Since the training goal of this model is the accuracy of the final image generated at the last moment of the dialogue, rather than the accuracy of the image generated in the middle of the dialogue, that is, the training and application goals of the model are inconsistent, so the existing methods trained The image generation model cannot satisfy the requirement of incrementality of interaction (that is, in the process of dialogue, the image information generated each time can increase with the interaction round). In view of this problem, in this step, the random replay training method is used for model training. During training, not only all the dialogue texts and final images obtained at the last moment of the dialogue process are used to train the model, but also the middle moment of training is used to train the model. All the dialogue texts that have been acquired make the model also need to perceive the text in the middle during the training process, which can reduce the difference between model training and application, and strengthen the incrementality of image generation.

In addition, in step 102, a heterogeneous cyclic neural network encoder is introduced during model training, so as to use the heterogeneous cyclic neural network encoder to encode the dialogue text data and the image description text, so that the encoded text feature vector It can accurately represent the image features described by the user language, which is beneficial to the random replay training method. Based on the input image and text features, the incrementality in the interaction process can be accurately captured, so that the trained model can generate interactive incrementality. image.

In one embodiment, in step 102, the following method can be used to train the interactive incremental image generation model:

Step 1021: Determine the number of dialogue rounds t currently used by random sampling.

Among them, 2≤t≤T, T is the total number of dialogues in the dialogue sample data.

This step is used to determine the randomly sampled dialogue middle moment, so that the dialogue data at the randomly intercepted middle moment can be used for training subsequently.

Step 1022: Input the image description text and the dialogue text data into the heterogeneous recurrent neural network encoder for encoding, and use the feature representation finally output from the encoding as the first text representation X'_T; A text representation is input to the image generator of the interactive incremental image generation model for image generation to obtain the first image Y' _T .

Step 1023: Based on the first text representation X' _T and the first image Y' _T , use the discriminator of the interactive incremental image generation model to calculate the main adversarial loss; use the main adversarial loss to update the The cumulative gradient of the image generator and the discriminator of the interactive incremental image generation model; the main adversarial loss includes the loss function value of the image generator and the discriminator.

Preferably, in this step, the main adversarial loss of the image generator can be calculated according to the following formula 1:

表示从图像生成器G中得到的图像Y _T′，D( )表示判别器输出的概率值。

表示利用图像生成器G生成的图像Y′_T计算非条件对抗损失函数；

表示利用图像生成器G生成的图像Y′_T和相应文本表示X′_T计算条件对抗损失函数；in,

represents the image Y _T ′ obtained from the image generator G, and D( ) represents the probability value output by the discriminator.

Indicates that the unconditional adversarial loss function is calculated using the image Y′ _T generated by the image generator G;

Represents the image Y′ _T generated by the image generator G and the corresponding text representation X′ _T to calculate the conditional adversarial loss function;

In this step, the calculation of the loss function value of the discriminator can be implemented by the method adopted by the existing AttnGAN model, which will not be repeated here.

Step 1024: Input the image description text and the first t rounds of dialogue data in the dialogue text data into the heterogeneous recurrent neural network encoder for encoding, and use the feature representation finally output from the encoding as the second text representation X'_t; inputting the second text representation to the image generator for image generation to obtain a second image Y′ _t ; inputting the first t-1 rounds of dialogue data in the image description text and the dialogue text data The heterogeneous recurrent neural network encoder performs encoding, and uses the feature representation finally output by the encoding as a third text representation X′ _t-1 ; inputting the third text representation into the image generator for image generation, obtaining The third image Y' _t-1 .

Here, in step 1024, it is necessary to use the number of dialogue rounds t randomly sampled in step 1021 to randomly truncate the dialogue process to construct pseudo samples (that is, the dialogue data for the first t rounds and the dialogue data for the first t-1 rounds) to simulate the middle moment of the dialogue process. Enter information so that the model can capture the "incrementality" of the data.

Step 1025: Construct a first positive example based on the second text representation X' _t and the second image Y'_t; based on the third text representation X' _t-1 and the third image Y' _{t -1} , construct the second positive example.

Further, in order to improve the accuracy of training, in this step, the second text representation X′ _t , the second image Y′ _t , the third text representation X′ _t-1 and the For the third image Y′ _t-1 , a negative example is constructed, as follows:

constructing a first negative example based on the third text representation _X't-1 and the second image _Y't ;

A second negative example is constructed based on the second textual representation _X't and the third image Y't _-1 .

Step 1026: Based on the first positive example, use the discriminator to calculate a first auxiliary adversarial loss; based on the first auxiliary adversarial loss, update the cumulative gradient of the image generator and the discriminator; the The first auxiliary adversarial loss includes loss function values of the image generator and the discriminator; based on the second positive example, the discriminator is used to calculate the second auxiliary adversarial loss; based on the second auxiliary adversarial loss, the second auxiliary adversarial loss is updated. The accumulated gradients of the image generator and the discriminator; the second auxiliary adversarial loss includes the loss function values of the image generator and the discriminator.

计算第一辅对抗损失中图像生成器损失函数值

按照

计算第二辅对抗损失中图像生成器损失函数值

In this step, the specific can be according to the formula

Calculate the value of the image generator loss function in the first auxiliary adversarial loss

according to

Calculate the image generator loss function value in the second auxiliary adversarial loss

Further, if the first negative example and the second negative example are constructed in step 1025, the following methods may be used in step 1026 to calculate the first auxiliary adversarial loss and the second auxiliary adversarial loss:

Using the discriminator, the first auxiliary adversarial loss is calculated based on the first positive example and the first negative example.

具体可以按照下述公式计算得到：Among them, the loss function value of the image generator in the first auxiliary adversarial loss

Specifically, it can be calculated according to the following formula:

Using the discriminator, the second auxiliary adversarial loss is calculated based on the second positive example and the second negative example.

具体可以按照下述公式计算得到：Among them, the loss function value of the image generator in the first adversarial loss

Specifically, it can be calculated according to the following formula:

得到图像生成器的总损失函数；其中，w_RR为辅对抗损失函数的权重。In step 1026, after obtaining the first auxiliary adversarial loss and the second auxiliary adversarial loss, you can follow the formula

Obtain the total loss function of the image generator; where w _RR is the weight of the auxiliary adversarial loss function.

In this step, the calculation of the loss function value of the discriminator can be implemented by the method adopted by the existing AttnGAN model, which will not be repeated here.

Step 1027: Update the parameters of the image generator based on the current cumulative gradient of the image generator; update the parameters of the discriminator based on the current cumulative gradient of the discriminator.

The specific implementation of this step is mastered by those skilled in the art, and details are not repeated here.

In one embodiment, in order to enable the encoder to capture more comprehensive and complete image and text feature information, the following method can be used to train the heterogeneous recurrent neural network encoder in advance:

Step x1: Obtain coding training sample data, where the coding training sample data includes image description text and visual dialogue text of standard sample images.

Step x2, using the coding training sample data to train the heterogeneous RNN encoder, so that the heterogeneous RNN encoder can compare the reference relationship in the visual dialogue text in the input data with the reference relationship in the image description text. Corresponding content is associated.

Considering that in visual dialogue data, the semantics of text is also asymmetric. The image description sentence explains the main content of the image, and the dialogue text is the supplement of the image description sentence, which provides additional information beyond the main content. For this reason, when obtaining the feature representation of text description and dialogue text, they can be treated differently to improve the completeness and accuracy of text feature representation. Specifically, each word in the image description sentence is crucial, so the image description sentence should be modeled at the level of "words"; and the information in the dialogue text is relatively sparse and redundant, so it can be In a round of dialogue, the sentence corresponding to the question and answer and a word in the image description sentence are regarded as equivalent, and the modeling is carried out at the "sentence" level. At the same time, the image description is regarded as the beginning of the visual dialogue process, and the same encoder is used to model both kinds of data simultaneously. Based on this, the structure of the heterogeneous RNN encoder as shown in Figure 2 can be designed.

As shown in Figure 2, in one embodiment, the heterogeneous cyclic neural network encoder consists of three RNN layers, specifically: the first cyclic neural network encoder (that is, the text RNN encoder in the figure), The second RNN encoder (i.e. the dialog RNN encoder in the figure) and the third RNN encoder (i.e. the fused RNN encoder in the figure). Preferably, the three RNN layers are all bidirectional gated recurrent units (Bi-GRU) in model structure. For the sake of brevity, only the forward computation path of the bidirectional GRU is drawn in the figure.

Among them, the text RNN encoder accepts the word vector of each word described by the image as input, regards each word input as a moment, and outputs a feature representation at each moment, which can be regarded as the entire image description sentence. A simple fusion of word information, called primary word feature representation. Assuming that the number of words in the text description is M ₁ , the text RNN encoder can generate M ₁ primary word feature representations.

The dialogue RNN encoder accepts the word vector of each word in the dialogue as input (the question and answer sentences of the dialogue are spliced together as one sentence), but only the output of the last moment is retained when outputting. The feature representation output at the last moment encodes the information of the whole round of dialogue text, which is defined as the primary sentence feature representation of this round of dialogue. Each round of text in the dialogue process is input into the dialogue RNN encoder according to this processing method, which can be regarded as sharing the same encoder model weights throughout the dialogue process.

As shown in Figure 2, taking the forward calculation path of the bidirectional GRU as an example, the encoding process of the heterogeneous recurrent neural network encoder is as follows:

Assuming a total of _M2 rounds of dialogue, the dialogue RNN encoder can generate _M2 primary sentence feature representations. In the dialogue RNN encoder, each round of dialogue text is independent, that is, a randomly initialized hidden state is used when encoding each round of dialogue text. In this way, the model can be processed in parallel for all dialogue turns, speeding up training and testing. Both the text RNN encoder and the dialogue RNN encoder are regarded as the first layer of the whole heterogeneous RNN encoder, which can produce low-level feature representations for image descriptions as well as dialogue texts. The upper-layer fused RNN encoder further processes the output of the lower-layer encoder: at the first M ₁ moment, the output of the text RNN encoder is used as input to generate high-level word feature representation; at the later M ₂ moment, the dialogue RNN encodes The output of the generator is the input, which produces high-level sentence feature representations. The Bi-GRU structure of the RNN encoder is fused so that it can observe all the information of the current dialogue process at every moment. On the one hand, it establishes the association between the texts of each round of dialogue in the dialogue, which is conducive to dealing with cross-round problems such as referential resolution in the dialogue process, and offsets the oversimplification caused by the independent processing of each round of text in the lower dialogue RNN encoder. On the other hand, the association between dialogue and text description is established, which can better integrate the information of the two as needed, and generate more expressive text features.

Based on the above, in one embodiment, the following method can be specifically adopted to train the heterogeneous RNN encoder by using the encoded training sample data:

Step x21, use the first cyclic neural network encoder to encode the image description text with words as the basic coding unit, and output the feature representation of each primary word obtained by encoding to the third cyclic neural network. network encoder; using the second recurrent neural network encoder to encode the visual dialogue text with sentences as the basic coding unit, and output the feature representation of each primary sentence obtained by encoding to the third loop Neural Network Encoder.

Step x22, the third recurrent neural network encoder encodes based on the primary word feature representation and the primary sentence feature representation, and uses the final output feature representation as a correlation between the visual dialogue text and the image description text. Globally encoded representation of the link.

The global encoding representation obtained in this step is the final encoding result of the heterogeneous recurrent neural network encoder.

Step x23, based on all the feature representations output by the third recurrent neural network encoder in the encoding process, using a deep attention multimodal similarity model (DAMSM) loss function to encode the first recurrent neural network. The weight parameters of the encoder, the second RNN encoder and the third RNN encoder are adjusted.

It can be seen from the above encoder training method that the heterogeneous recurrent neural network encoder trained based on the above method is used to generate the text used to generate images after each round of dialogue. The dialogue information establishes the association between the dialogue texts in the completed dialogue rounds, which is conducive to dealing with cross-round problems such as referential resolution in the dialogue process, thereby helping to meet the incremental requirements of images. The relationship between the dialogue text and the image description text is better integrated, resulting in more expressive and accurate text features. Therefore, adopting the heterogeneous RNN encoder obtained by the above method is beneficial to reasonably implement the dialogue-to-image generation task.

It can be seen from the above-mentioned embodiment of the training method for the interactive incremental image generation model that in the above-mentioned embodiment, during model training, the dialogue text at the middle moment is used, combined with the heterogeneous recurrent neural network encoder, the trained model can be The model is able to generate images with interactive incrementality, which facilitates the rational implementation of dialogue-to-image generation tasks.

Based on the above model training method, an embodiment of the present invention also provides an interactive incremental image generation method, including:

In the process of visual dialogue, at the end of each round of human-machine dialogue, input all the generated human-machine dialogue texts and preset image description texts into the pre-trained interactive incremental image generation model for image generation and display. the generated image;

Wherein, the interactive incremental image generation model is obtained based on the above-mentioned training method.

Here, since the interactive incremental image generation model used is obtained based on the above model training method, and the model's ability to perceive the dialogue text at the middle moment is considered during model training, the interactive incrementality of the model is strengthened. Therefore, The plausibility of the images generated during the dialogue can be improved.

Based on the above model training method embodiment, the embodiment of the present invention further provides a training device for an interactive incremental image generation model, including: a processor, where the processor is configured to:

Obtaining dialogue sample data, the dialogue sample data includes dialogue text data, standard images, image description texts and the total number of dialogue rounds;

The interactive incremental image generation model is trained by means of random replay training, using the dialogue sample data and the pre-trained heterogeneous recurrent neural network encoder, so that the interactive incremental image model can be based on human-machine dialogue Text and image description text generate images with interactive incrementality; wherein the training is performed using all dialogue text data obtained at the end of the dialogue and all dialogue text data obtained at the middle of the dialogue.

Based on the above-mentioned embodiment of the interactive incremental image generation method, the embodiment of the present invention further discloses an interactive incremental image generation device, comprising: a processor, where the processor is configured to:

In the process of visual dialogue, at the end of each round of human-machine dialogue, input all the generated rounds of human-machine dialogue text and preset image description text into the pre-trained interactive incremental image generation model for image generation, and display the resulting image;

Wherein, the interactive incremental image generation model is obtained based on the above-mentioned training method.

Based on the embodiment of the training method for the interactive incremental image generation model, the embodiment of the present application also implements a training electronic device for the interactive incremental image generation model, including a processor and a memory; The application program executed by the processor is used to cause the processor to execute the training method of the interactive incremental image generation model as described above. Specifically, it is possible to provide a system or device equipped with a storage medium on which software program codes for realizing the functions of any one of the above-described embodiments are stored, and make the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium. In addition, a part or all of the actual operation can also be completed by an operating system or the like operating on the computer based on the instructions of the program code. The program code read from the storage medium can also be written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then the instructions based on the program code make the device installed in the computer. The CPU or the like on the expansion board or the expansion unit is used to perform part and all of the actual operations, so as to realize the function of any one of the above-mentioned embodiments of the training method for the interactive incremental image generation model.

Based on the embodiment of the interactive incremental image generation method, the embodiment of the present application further implements an interactive incremental image generation electronic device, including a processor and a memory; the memory stores an application program executable by the processor, for causing the processor to perform the interactive incremental image generation method as described above. Specifically, it is possible to provide a system or device equipped with a storage medium on which software program codes for realizing the functions of any one of the above-described embodiments are stored, and make the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium. In addition, a part or all of the actual operation can also be completed by an operating system or the like operating on the computer based on the instructions of the program code. The program code read from the storage medium can also be written into the memory provided in the expansion board inserted into the computer or into the memory provided in the expansion unit connected to the computer, and then the instructions based on the program code make the device installed in the computer. The CPU or the like on the expansion board or the expansion unit is used to perform part and all of the actual operations, so as to realize the functions of any one of the above-mentioned implementations of the interactive incremental image generation method.

The above-mentioned memory may be specifically implemented as various storage media such as Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash Memory (Flash memory), Programmable Program Read-Only Memory (PROM). The processor may be implemented to include one or more central processing units or one or more field programmable gate arrays, wherein the field programmable gate arrays integrate one or more central processing unit cores. Specifically, a central processing unit or central processing unit core may be implemented as a CPU or an MCU.

It should be noted that not all steps and modules in the above-mentioned processes and structural diagrams are necessary, and some steps or modules may be omitted according to actual needs. The execution order of each step is not fixed and can be adjusted as required. The division of each module is only to facilitate the description of the functional division used. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device. , or in a different device.

The hardware modules in various embodiments may be implemented mechanically or electronically. For example, a hardware module may include specially designed permanent circuits or logic devices (eg, special purpose processors, such as FPGAs or ASICs) for performing specific operations. Hardware modules may also include programmable logic devices or circuits (eg, including general-purpose processors or other programmable processors) temporarily configured by software for performing particular operations. As for the specific use of a mechanical method, or a dedicated permanent circuit, or a temporarily configured circuit (for example, configured by software) to realize the hardware module, it can be decided according to cost and time considerations.

As used herein, "schematic" means "serving as an example, instance, or illustration" and any illustration, embodiment described herein as "schematic" should not be construed as a preferred or advantageous one Technical solutions. In order to make the drawings concise, only the relevant parts of the present invention are schematically shown in each drawing, and do not represent the actual structure of the product. In addition, in order to make the drawings concise and easy to understand, in some drawings, only one of the components having the same structure or function is schematically shown, or only one of them is marked. As used herein, "one" does not mean to limit the number of relevant parts of the invention to "only one", and "one" does not mean to exclude "more than one" number of relevant parts of the invention. In this article, "upper", "lower", "front", "rear", "left", "right", "inner", "outer", etc. are only used to indicate the relative positional relationship between related parts, and The absolute positions of these relative parts are not limited.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A method for training an interactive incremental image generation model, comprising:

obtaining conversation sample data, wherein the conversation sample data comprises conversation text data, a standard image, an image description text and a total number of conversation rounds;

training an interactive incremental image generation model by using the dialogue sample data and a pre-trained heterogeneous cyclic neural network encoder in a random replay training mode so that the interactive incremental image generation model can generate an image with interactive incremental based on a man-machine dialogue text and an image description text; wherein the training is performed based on all dialog text data obtained at a final time of the dialog and all dialog text data obtained at an intermediate time of the dialog.

2. The method of claim 1, wherein the training comprises training

Determining the number T of currently adopted conversation rounds by adopting a random sampling mode, wherein T is more than or equal to 2 and less than or equal to T, and T is the total number of the conversation;

inputting the image description text and the dialogue text data into the heterogeneous cyclic neural network encoder for encoding, and taking the feature representation output at last of encoding as a first text representation X'_T(ii) a Inputting the first text representation to an image generator of an interactive incremental image generation model for image generation to obtain a first image Y'_T；

Representing X 'based on the first text'_TAnd the first image Y'_TComputing using said interactive incremental image generation model's discriminatorLoss of primary confrontation; updating the accumulated gradients of an image generator and a discriminator of the interactive incremental image generation model with the primary confrontation losses; the primary countermeasure loss includes a loss function value of an image generator and a discriminator;

inputting the image description text and the front t wheel in the dialogue text data into the heterogeneous cyclic neural network encoder for encoding, and using the feature representation output at last of encoding as a second text representation X'_t(ii) a Inputting the second text representation to the image generator for image generation to obtain a second image Y'_t；

Inputting the image description text and the first t-1 wheel in the dialogue text data into the heterogeneous recurrent neural network encoder for encoding, and using the feature representation output at last of encoding as a third text representation X'_t-1(ii) a Inputting the third text representation to the image generator for image generation to obtain a third image Y'_t-1；

Representing X 'based on the second text'_tAnd the second image Y'_tConstructing a first positive example;

representing X 'based on the third text'_t-1And the third image Y'_t-1, constructing a second positive example;

calculating a first secondary confrontation loss by using the discriminator based on the first positive example; updating a cumulative gradient of the image generator and the discriminator based on the first secondary confrontation loss; the first secondary countermeasure loss includes a loss function value of an image generator and a discriminator;

calculating a second secondary confrontation loss by using the discriminator based on the second positive example; updating the cumulative gradients of the image generator and the discriminator based on the second secondary confrontation loss; the second secondary countermeasure loss includes a loss function value of an image generator and a discriminator;

updating parameters of the image generator based on the current accumulated gradient of the image generator; and updating the parameters of the discriminator based on the current accumulated gradient of the discriminator.

3. The method of claim 2, wherein the training method further comprises:

representing X 'based on the third text'_t-1 and the second image Y'_tConstructing a first negative example;

representing X 'based on the second text'_tAnd the third image Y'_t-1, constructing a second negative example;

the calculating the first secondary confrontation loss comprises:

calculating the first secondary countermeasure loss using the discriminator based on the first positive example and the first negative example;

the calculating the second secondary confrontation loss comprises:

calculating, with the discriminator, the second secondary countermeasure loss based on the second positive example and the second negative example.

4. The method of claim 2, wherein the training of the heterogeneous cyclic neural network encoder comprises:

acquiring coding training sample data, wherein the coding training sample data comprises an image description text and a visual dialog text of a standard sample image;

and training a heterogeneous cyclic neural network encoder by using the encoding training sample data, so that the heterogeneous cyclic neural network encoder can associate the reference relationship in the visual dialog text in the input data with the corresponding content in the image description text.

5. The method of claim 4, wherein the heterogeneous recurrent neural network encoder consists of a first recurrent neural network encoder, a second recurrent neural network encoder, and a third recurrent neural network encoder;

the training a heterogeneous cyclic neural network encoder by using the encoded training sample data comprises:

encoding the image description text by using the first recurrent neural network encoder and an encoding unit taking words as basic, and outputting each primary word feature representation obtained by encoding to the third recurrent neural network encoder; coding the visual dialog text by using the second cyclic neural network coder and a sentence-based coding unit, and outputting each primary sentence feature representation obtained by coding to the third cyclic neural network coder;

the third recurrent neural network encoder encodes based on the primary word feature representation and the primary sentence feature representation and the last output feature representation as a global encoded representation in which the visual dialog text and the image description text are associated;

and adjusting the weight parameters of the first cyclic neural network encoder, the second cyclic neural network encoder and the third cyclic neural network encoder by adopting a deep attention multi-modal similarity model DAMSM loss function based on all feature representations output by the third cyclic neural network encoder in the encoding process.

6. An interactive incremental image generation method, comprising:

in the process of visual conversation, when each round of man-machine conversation is finished, inputting all the generated round of man-machine conversation texts and a preset image description text into a pre-trained interactive incremental image generation model for image generation, and displaying the generated image;

wherein the interactive incremental image generation model is obtained based on any one of the training methods of claims 1 to 5.

7. An interactive incremental image generation model training apparatus, comprising: a processor to:

obtaining conversation sample data, wherein the conversation sample data comprises conversation text data, a standard image, an image description text and a total number of conversation rounds;

training an interactive incremental image generation model by using the dialogue sample data and a pre-trained heterogeneous cyclic neural network encoder in a random replay training mode so that the interactive incremental image generation model can generate an image with interactive incremental based on a man-machine dialogue text and an image description text; wherein the training is performed using all dialog text data obtained at a final time of the dialog and all dialog text data obtained at an intermediate time of the dialog.

8. A non-transitory computer readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the steps of any of the training methods of claims 1 to 5.

9. An interactive incremental image generation apparatus, comprising: a processor to:

the interactive incremental image generation method comprises the steps of inputting all generated round man-machine conversation texts and preset image description texts into a pre-trained interactive incremental image generation model for image generation and displaying generated images when each round of man-machine conversation is finished in the visual conversation process;

wherein the interactive incremental image generation model is obtained based on any one of the training methods of claims 1 to 5.

10. A non-transitory computer readable storage medium storing instructions which, when executed by a processor, cause the processor to perform the steps of the interactive incremental image generation method of claim 6.

Images