CN116721186A - Drawing image generation method and device, electronic equipment and storage medium - Google Patents

Abstract

The invention provides a drawing image generation method, a device, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, wherein the method comprises the following steps: acquiring text information and a feature control image, wherein the text information is a text describing an image picture of a drawing image to be generated, and the drawing image to be generated has image features of the feature control image; based on the text information, obtaining a text code corresponding to the text information; based on text codes, intervening the image codes obtained by each round of reasoning in the diffusion model in the process of multiple rounds of iterative reasoning to obtain basic image codes of each round of reasoning; obtaining a feature control image code corresponding to the feature control image based on the feature control image; and based on the characteristic control image coding, intervening the basic image coding of the target round reasoning until the reasoning process is finished, so as to obtain the drawing image to be generated. The method reduces the running load of the video memory and the graphic processor and reduces the generation cost of the drawing image.

Description

Painting image generation method, device, electronic equipment and storage medium

Technical field

The present invention relates to the field of artificial intelligence technology, and in particular to a painting image generation method, device, electronic equipment and storage medium.

Background technique

It can be seen from related technologies that the industry uses the diffusion model (also known as the Diffusion model) as the main tool for painting image generation. Compared with other generative models, the Diffusion model has the characteristics of multiple rounds of iteration and gradual denoising, so the generated painting images have a higher level.

However, in terms of content control of the generated painting images, it is necessary to integrate the auxiliary neural network model into the Diffusion model. Since other auxiliary neural network models need to be introduced in the process of generating painting images, this will increase the operating load of the video memory and graphics processor, and increase the cost of generating painting images.

Contents of the invention

The present invention provides a painting image generation method, device, electronic equipment and storage medium, which enables the automatic acquisition of a painting image to be generated with the image characteristics of a feature control image without the need to introduce other auxiliary neural network models. This can reduce the running load of video memory and graphics processor, and reduce the cost of generating drawing images.

The present invention provides a painting image generation method. The method includes: obtaining text information and a feature control image, wherein the text information is text describing an image frame of the painting image to be generated, and the painting image to be generated has the The feature controls the image feature of the image; based on the text information, a text encoding corresponding to the text information is obtained; based on the text encoding, the image encoding obtained by the diffusion model in each round of iterative reasoning in the multi-round iterative reasoning process is performed Intervene to obtain the basic image coding of each round of reasoning; based on the feature control image, obtain the feature control image coding corresponding to the feature control image; based on the feature control image coding, the basis of the target round of reasoning Image coding intervenes until the end of the inference process to obtain the painting image to be generated.

According to a painting image generation method provided by the present invention, the image encoding is controlled based on the characteristics, and the basic image encoding of the target round reasoning is intervened until the end of the reasoning process, specifically including: in the target round reasoning In the case before the reference round of reasoning, the image coding is controlled based on the characteristics, and the basic image coding is fully replaced and intervened according to a first preset replacement rate until the end of the reasoning process, wherein the first preset The replacement rate is greater than or equal to the replacement rate threshold; the Gaussian noise of the basic image encoding obtained before the reference round of inference is greater than or equal to the noise threshold.

According to a painting image generation method provided by the present invention, the image encoding is controlled based on the characteristics, and the basic image encoding of the target round reasoning is intervened until the end of the reasoning process, specifically including: in the target round reasoning In the case after the reference round of inference, the image encoding is controlled based on the characteristics, and the base image encoding corresponding to the activation area is locally replaced and intervened according to the second preset replacement rate until the end of the inference process, where The second preset replacement rate is less than the replacement rate threshold; the activation area is an intervention area that requires intervention in the painting image to be generated determined according to the image characteristics of the feature control image; after the reference round inference The obtained Gaussian noise encoded by the basic image is less than the noise threshold.

According to a painting image generation method provided by the present invention, the feature control images include two or more, and the corresponding feature control image codes include two or more; the feature control image codes based on the , intervening in the basic image coding of the target round reasoning until the end of the reasoning process, specifically including: determining the replacement weight value of each feature control image coding; according to the replacement weight value, based on the feature control image The encoding replaces the base image encoding of the target round of reasoning and intervenes until the end of the reasoning process.

According to a painting image generation method provided by the present invention, the base image coding of target round reasoning is replaced and intervened until the end of the reasoning process based on the feature control image coding, specifically including: converting the feature control image into Encoding is performed for multiple rounds iteratively adding noise to the round corresponding to the target round of reasoning to obtain the feature-controlled image coding after adding noise; based on the feature-controlled image coding after adding noise, the basis for the target round reasoning is obtained The image encoding is replaced and intervenes until the end of the inference process.

According to a painting image generation method provided by the present invention, when the diffusion model is a diffusion model from depth information to an image, the image encoding is controlled based on the characteristics, and the basic image of the target round inference is Coding intervenes until the end of the inference process, specifically including: controlling image encoding based on the characteristics, intervening in the target channel of the basic image encoding of the target round inference until the end of the inference process, wherein the target channel is related to depth information For channels other than the corresponding channel, the depth information is image depth information corresponding to the basic image encoding.

According to a painting image generation method provided by the present invention, the feature control image includes a feature control image with preset depth image features, a feature control image with preset edge structure image features, and features with preset posture image features. Control one or more of the images.

The present invention also provides a painting image generation device. The device includes: an acquisition module for acquiring text information and a feature control image, wherein the text information is text describing the image frame of the painting image to be generated, and the text information to be generated is Generate a painting image with the image characteristics of the feature control image; a text encoding module for obtaining a text encoding corresponding to the text information based on the text information; an intervention module for applying the diffusion model based on the text encoding In the multi-round iterative reasoning process, the image coding obtained by each round of reasoning is intervened to obtain the basic image coding of each round of reasoning; the image coding module is used to control the image based on the feature to obtain the image corresponding to the feature control The characteristics control the image coding; the generation module is used to control the image coding based on the characteristics, and intervene in the basic image coding of the target round of reasoning until the end of the reasoning process to obtain the painting image to be generated.

The present invention also provides an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the painting image as described in any one of the above is realized. Generate method.

The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for generating a painting image as described above is implemented.

The present invention also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method for generating a painting image as described above is implemented.

The present invention provides a painting image generation method, device, electronic equipment and storage medium, obtains text information and feature control images, and based on the text encoding corresponding to the text information, infers the diffusion model in each round of the multi-round iterative reasoning process. The obtained image coding is intervened to obtain the basic image coding of each round of reasoning, and then based on the feature control image coding corresponding to the feature control image, the basic image coding of the target round of reasoning is intervened until the end of the reasoning process, so that it can be Without the need to introduce other auxiliary neural network models, the painting image to be generated with the image characteristics of the characteristic control image can be automatically obtained, thereby reducing the graphics memory and graphics processor running load, and reducing the cost of generating painting images.

Description of the drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are of the present invention. For some embodiments of the invention, those of ordinary skill in the art can also obtain other drawings based on these drawings without exerting creative efforts.

Figure 1 is a schematic flow chart of the painting image generation method provided by the present invention;

Figure 2 is a schematic flow chart of the feature-based control image coding provided by the present invention, which intervenes in the basic image coding of target round reasoning until the end of the reasoning process;

Figure 3 is a schematic flowchart of the feature-based control image coding provided by the present invention, which replaces and intervenes in the basic image coding of the target round of reasoning until the end of the reasoning process;

Figure 4 is a schematic diagram of the application scenario of the painting image generation method provided by the present invention;

Figure 5 is a schematic structural diagram of the painting image generating device provided by the present invention;

Figure 6 is a schematic structural diagram of the electronic device provided by the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the technical solutions in the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention. , not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present invention.

Figure 1 is a schematic flowchart of the painting image generation method provided by the present invention.

In order to further introduce the painting image generation method provided by the present invention, the following will be described in conjunction with the following embodiments.

In an exemplary embodiment of the present invention, as can be seen in conjunction with Figure 1, the painting image generation method may include steps 110 to 150. Each step will be introduced separately below.

In step 110, text information and feature control images are obtained.

Among them, the text information is the text describing the image picture of the painting image to be generated; the painting image to be generated has the image characteristics of the characteristic control image.

It can be understood that the text information is text information used to describe the overall portrait of the painting image to be generated. In other words, the painting image to be generated can be a form of expression in which the text information is presented in the form of an image.

In addition to the image described by the text information, the painting image to be generated also has corresponding image features, such as image structure features, image color features, pose features of each element in the image, or depth information features of the image. . During the application process, the image characteristics of the painting image can be intervened based on the image characteristics of the pre-provided feature control image. It can be understood that the image in which intervention is performed based on the feature control image may also have the image features of the feature control image.

In yet another exemplary embodiment of the present invention, the feature control image may include a feature control image with preset depth image features, a feature control image with preset edge structure image features, and a feature control image with preset pose image features. one or more of them.

In this embodiment, the specific content of the feature control image is not limited, and it can be determined according to the actual needs of the user.

In step 120, based on the text information, a text code corresponding to the text information is obtained.

In one embodiment, the text information can be encoded and represented through a language model, such as a clip model, a T5 model, etc., to obtain a text encoding corresponding to the text information. It is understandable that in the subsequent multi-round iterative reasoning process based on the diffusion model, text encoding will serve as a factor that interferes with the reasoning process.

In step 130, based on the text coding, the image coding obtained by the diffusion model in each round of inference in the multi-round iterative reasoning process is intervened to obtain the basic image coding of each round of inference.

It should be noted that during the multi-round iterative reasoning process of the diffusion model (Diffusion model), image coding will be obtained in each round of reasoning. It can be understood that the image coding is iteratively obtained by the Diffusion model from noise. During the application process, image coding can be intervened based on text coding to obtain the basic image coding for each round of reasoning. Since the basic image coding is obtained based on text coding intervention, each basic image coding obtained can be considered as an intermediate state in which the painting image to be generated is obtained based on diffusion model reasoning.

In another embodiment, the cross attention model can be used to fuse text coding and image coding to realize the intervention processing of text coding on image coding, and then obtain basic image coding.

In step 140, based on the feature control image, a feature control image code corresponding to the feature control image is obtained.

In step 150, the image coding is controlled based on the features, and the basic image coding of the target round of reasoning is intervened until the end of the reasoning process to obtain the painting image to be generated.

In one embodiment, the feature control image can be extracted and processed to obtain a feature control image code corresponding to the feature control image. Furthermore, the image coding can be controlled by combining features to intervene in the basic image coding obtained during each round of reasoning, and the reasoning can be iterated until the end of the reasoning process.

In yet another example, image encoding can also be controlled based on features, and intervention processing begins with the basic image encoding obtained from the target round of inference, and continues to intervene in the iterative inference until the end of the inference process. Among them, the target round of reasoning can be determined according to the actual situation. In an example, taking the iterative reasoning process of the Diffusion model as 50 rounds as an example, the target round of reasoning can be any one of the 50 rounds of reasoning. . When the target round is the first round of reasoning process, the feature-based control image coding will be intervened starting from the basic image coding of the first round of reasoning, and will continue to intervene in the iterative reasoning until the end of the 50th round of reasoning process, so that it can Obtain the painting image to be generated with the image characteristics of the characteristic control image. Through this embodiment, the painting image to be generated with the image characteristics of the characteristic control image can be automatically obtained without the need to introduce other auxiliary neural network models, thereby reducing the running load of the video memory and the graphics processor, and reducing the cost of painting. Image generation cost.

It should be noted that in the current process of introducing an auxiliary neural network model to obtain the painting image to be generated with the image characteristics of the characteristic control image, it is also necessary to additionally train an independent auxiliary neural network model, which is time-consuming and labor-intensive; in addition, in the insertion and auxiliary In the process of plug-in the auxiliary module corresponding to the neural network model, there will be situations where it is not suitable for the Diffusion model. However, the painting image generation method provided by the present invention intervenes in the intermediate state, such as basic image coding, obtained during the iterative reasoning process of the Diffusion model until the end of the reasoning process, thereby eliminating the need to introduce other auxiliary neural network models. In this case, the painting image to be generated with the image characteristics of the feature control image can be automatically obtained, thereby avoiding the time-consuming and labor-intensive problem of additional training of an independent auxiliary neural network model, as well as the auxiliary module plug-ins and corresponding auxiliary neural network models. The problem of Diffusion model mismatch can decouple the auxiliary module plug-in from the Diffusion model. Among them, the auxiliary module can be understood as a processing module that can intervene in the painting image to be generated into image features with characteristic control images.

Among them, the feature control image can be adjusted according to the user's needs. In addition, the target round can also be determined according to the user's needs, so that the process of generating the painting image can be automatically controlled and the feature-based control image can be personalized according to the user's needs. The intervention process of image coding to obtain the painting image to be generated that meets the user's needs.

The painting image generation method provided by the present invention obtains text information and feature control images, and based on the text coding corresponding to the text information, intervenes in the image coding obtained by the diffusion model in each round of iterative reasoning in the multi-round iterative reasoning process to obtain The basic image coding of each round of reasoning, and then based on the feature control image coding corresponding to the feature control image, intervenes in the basic image coding of the target round of reasoning until the end of the reasoning process, so that there is no need to introduce other auxiliary neural network models. In this case, the painting image to be generated with the image characteristics of the characteristic control image can be automatically obtained, thereby reducing the running load of the video memory and the graphics processor, and reducing the cost of generating the painting image.

Figure 2 is a schematic flowchart of the feature-based control image coding provided by the present invention, intervening in the basic image coding of the target round of reasoning until the end of the reasoning process.

In order to further introduce the painting image generation method provided by the present invention, the following will be described in conjunction with Figure 2.

In an exemplary embodiment of the present invention, it can be seen from Figure 2 that controlling image coding based on features and intervening in the basic image coding of target round reasoning until the end of the reasoning process may include steps 210 and 220. Each step will be introduced below. .

In step 210, when the target round of inference is before the reference round of inference, the image encoding is controlled based on the features, and the basic image encoding is fully replaced and intervened according to the first preset replacement rate until the end of the inference process.

In step 220, when the target round inference is after the reference round inference, the image encoding is controlled based on the features, and the basic image encoding corresponding to the activation area is locally replaced and intervened according to the second preset replacement rate until inference. The process ends.

Among them, the first preset replacement rate is greater than or equal to the replacement rate threshold; the second preset replacement rate is less than the replacement rate threshold; the activation area is the intervention area in the painting image to be generated that needs intervention determined according to the image characteristics of the feature control image; The Gaussian noise of the base image encoding obtained before the reference round of inference is greater than or equal to the noise threshold; the Gaussian noise of the base image encoding obtained after the reference round of inference is less than the noise threshold.

In one embodiment, the DDIM algorithm can be used as an optimization method to speed up inference and reduce iteration rounds. Let’s continue to take the iterative reasoning process of the Diffusion model as 50 rounds as an example. In the 50 rounds of iterative reasoning, the Gaussian noise in the first 25 rounds of iterative reasoning was obvious, and the actual content was almost invisible; the content gradually became clear in the last 25 rounds of iterative reasoning. Therefore, deep intermediate state intervention is carried out in the first 25 rounds, and each round is set to a comprehensive replacement rate of 0.7 in weight. In the last 25 rounds, shallow intervention is performed, only the replacement of the active area is updated, and the replacement rate is set to 0.4. Therefore, based on the painting image generation method provided by the present invention, there is no need to additionally train an independent auxiliary network to control the generation of new auxiliary modules.

In yet another embodiment, the reference round may be understood as the aforementioned 25th round. Among them, the Gaussian noise of the basic image encoding obtained before the reference round inference is greater than or equal to the noise threshold. Therefore, in the case where the target round inference is before the reference round inference, the basic image encoding of the target round inference is performed. During the intervention process, the basic image coding can be completely replaced and intervened according to the first preset replacement rate until the iterative reasoning process ends. The first preset replacement rate may correspond to the aforementioned 0.7. In this embodiment, the first preset replacement rate is not specifically limited. It should be noted that when the total number of rounds of iterative reasoning is 50, the first preset replacement rate can be set above 0.5, and the second preset replacement rate can be set below 0.5.

In yet another embodiment, the Gaussian noise of the base image encoding obtained after the reference round of inference is less than the noise threshold. Therefore, in the case where the target round of inference is after the reference round of inference, the basis of the target round of inference is During the intervention process of image coding, the basic image coding corresponding to the activation area can be partially replaced and intervened according to the second preset replacement rate until the iterative reasoning process ends. The second preset replacement rate may correspond to the aforementioned 0.4. In this embodiment, the second preset replacement rate is not specifically limited.

The activation area can be understood as: if it is determined based on the image characteristics of the feature control image that intervention is required on the background of the painting image to be generated, then the activation area can immediately be the area corresponding to the background of the painting image to be generated.

Figure 3 is a schematic flowchart of the feature-based control image coding provided by the present invention, which replaces and intervenes in the basic image coding of the target round of reasoning until the end of the reasoning process.

The following will be described with reference to the embodiment shown in FIG. 3 .

In an exemplary embodiment of the present invention, the feature control images may include two or more, and the corresponding feature control image codes may also include two or more. It can be seen from Figure 3 that controlling image coding based on features and intervening in the basic image coding of target round reasoning until the end of the reasoning process may include steps 310 and 320. Each step will be introduced below.

In step 310, the replacement weight value of each feature control image encoding is determined respectively;

In step 320, according to the replacement weight value, the basic image coding of the target round inference is replaced based on the feature control image coding and intervenes until the end of the reasoning process.

In one embodiment, when the feature control image codes include two or more, direct addition and replacement at the corresponding positions of the images corresponding to the basic image codes will result in too high values. During the application process, you can The replacement weight value of each feature control image coding is determined respectively; and according to the replacement weight value, the basic image coding of the target round of reasoning is replaced and intervened until the end of the reasoning process based on the feature control image coding.

In yet another embodiment, the basic image coding of the target round reasoning can be replaced and intervened until the end of the reasoning process based on the feature control image coding according to the replacement weight value, where the basic image coding can be understood as corresponding to the activation area. basic image coding.

In yet another exemplary embodiment of the present invention, image coding is controlled based on features, and the basic image coding of target round reasoning is replaced and intervened until the end of the reasoning process. This can also be implemented in the following ways:

Perform multiple rounds of iterative noise on the feature control image coding to the round corresponding to the target round of reasoning, and obtain the feature control image coding after noise addition;

The image coding is controlled based on the features after adding noise, and the basic image coding of the target round reasoning is replaced and intervened until the end of the reasoning process.

In one embodiment, the feature control image encoding corresponding to the feature control image can be iteratively denoised for multiple rounds to the round corresponding to the target round of inference, which can be understood as a process of reverse reasoning to obtain the noise addition signal. Features control image encoding. Furthermore, the image coding can be controlled based on the noise-added features, and the basic image coding of the target round of reasoning can be replaced and intervened until the end of the iterative reasoning process. In this embodiment, the feature control image coding is denoised before the basic image coding is replaced and intervened. This can ensure the correspondence between the feature control image coding and the basic image coding in the inference rounds during the iterative reasoning process, thus In the intervention process of each round after the target round of reasoning, the feature control image coding and the basic image coding are round-corresponding, so that the replacement and intervention of the basic image coding can be better realized.

Figure 4 is a schematic diagram of an application scenario of the painting image generation method provided by the present invention.

In order to further introduce the aforementioned feature-based control image coding, the process of replacing the basic image coding of the target round reasoning and intervening until the end of the reasoning process will be explained below with reference to Figure 4.

In one embodiment, it can be seen in conjunction with Figure 4 that from X _T to X ₀ can be understood as 50 rounds of reasoning in the iterative reasoning process based on the Diffusion model. Among them, X _T is the image encoding corresponding to the random noise when the Diffusion model starts inference. X ₀ can be understood as the feature control image encoding corresponding to the feature control image. When the target round of reasoning is round t corresponding to Figure 4, the feature control image encoding can be iteratively denoised for multiple rounds to the round corresponding to the target round of reasoning (corresponding to round t), and the result after adding noise is Feature-controlled image coding, in which the feature-controlled image coding after adding noise can correspond to q (x _t | x _t-1 ) in Figure 4, and the basic image coding can correspond to p (x _t-1 | x _t ) in Figure 4 . During the application process, when the feature control image encoding is intervened in round t, the feature control image coding must also be processed to round t through reverse noise addition. Further, in rounds T to 0, that is, rounds t to t-1 of each intermediate state process, different replacement weight values can be set for different feature control image encodings, and intervention can be carried out according to the replacement weight values. replace.

In an example, when the feature control image coding is the feature control image coding corresponding to canny, and the feature control image coding corresponding to pose, the replacement weight value of the feature control image coding corresponding to canny can be set to 0.6 , set the replacement weight value of the feature control image encoding corresponding to pose to 0.4.

In another exemplary embodiment of the present invention, in the case where the diffusion model is a diffusion model from depth information to image (corresponding to the depth to Image generation method), the description will continue to take the embodiment described in Figure 1 as an example. Based on Features control image coding, and intervention in the basic image coding of target round reasoning until the end of the reasoning process (corresponding to step 150) can be implemented in the following ways:

Control image coding based on features, and intervene in the target channel of the basic image coding of the target round inference until the end of the inference process. The target channel is a channel other than the channel corresponding to the depth information, and the depth information is a channel corresponding to the basic image coding. Image depth information.

In one embodiment, in the case of the depth to image generation method, under the unet framework, only the first four channels can be subjected to intervention inference processing, and the depth information of the fifth channel remains unchanged, thus achieving a double-sided control effect. . Among them, the first four channels can correspond to the target channels mentioned above; the fifth channel can correspond to the depth information corresponding channels mentioned above.

The painting image generation method provided by the present invention, in order to control the generation cost of generating the painting image to be generated, proposes to use the sampling iteration (corresponding to iterative reasoning) of the Diffusion model to insert an auxiliary module into the intermediate state (corresponding to the basic image encoding). The main effects of intervention control methods are:

1. No additional training network: No need to train additional networks avoids the increase of video memory during the inference phase, and also solves the problem of increasing video memory for multi-module expansion, saving deployment costs;

2. Support the expansion of new auxiliary modules: The way of inserting auxiliary modules for intervention during sampling iteration not only avoids the need for independent network training of auxiliary modules, but also facilitates the expansion of new auxiliary modules;

3. Support multi-auxiliary module combination control: use different weights to control the intervention of different auxiliary modules, and intervene in image generation in the corresponding activation area;

4. The auxiliary module is decoupled from the Diffusion basic model. It is beneficial to apply auxiliary modules to various different Diffusion basic models. At the same time, the combination of the intermediate-state interference control method without training and various basic models further provides a convenient way for the expansion of various vertical categories.

According to the above description, the present invention provides a painting image generation method, which obtains text information and feature control images, and based on the text encoding corresponding to the text information, performs the multi-round iterative reasoning process on the images obtained by each round of the diffusion model. Coding is intervened to obtain the basic image coding of each round of reasoning, and then based on the feature control image coding corresponding to the feature control image, the basic image coding of the target round of reasoning is intervened until the end of the reasoning process, so that no additional introduction is needed In the case of other auxiliary neural network models, the painting image to be generated with the image characteristics of the characteristic control image can be automatically obtained, thereby reducing the graphics memory and graphics processor running load, and reducing the cost of generating painting images.

Based on the same concept, the present invention also provides a painting image generating device.

The painting image generating device provided by the present invention will be described below. The painting image generating device described below and the painting image generating method described above may be referenced correspondingly.

Figure 5 is a schematic structural diagram of the painting image generating device provided by the present invention.

In an exemplary embodiment of the present invention, as can be seen in conjunction with Figure 5, the painting image generation device may include an acquisition module 510, a text encoding module 520, an intervention module 530, an image encoding module 540 and a generation module 550. Each module will be introduced separately below.

The acquisition module 510 may be configured to acquire text information and a feature control image, where the text information is text describing the image frame of the painting image to be generated, and the painting image to be generated has the image features of the feature control image;

The text encoding module 520 may be configured to obtain text encoding corresponding to the text information based on the text information;

The intervention module 530 may be configured to intervene, based on text coding, on the image coding obtained by the diffusion model in each round of inference in the multi-round iterative reasoning process to obtain the basic image coding of each round of inference;

The image encoding module 540 may be configured to obtain a characteristic control image encoding corresponding to the characteristic control image based on the characteristic control image;

The generation module 550 may be configured to control image encoding based on features, and intervene in the basic image encoding of target round inference until the end of the inference process to obtain the painting image to be generated.

In an exemplary embodiment of the present invention, the generation module 550 can implement feature-based control image coding in the following manner, and intervene in the basic image coding of the target round of reasoning until the end of the reasoning process:

When the target round of reasoning is before the reference round of reasoning, the image encoding is controlled based on the features, and the basic image encoding is fully replaced and intervened according to the first preset replacement rate until the end of the reasoning process, where the first preset replacement The rate is greater than or equal to the replacement rate threshold; the Gaussian noise of the base image encoding obtained before the reference round of inference is greater than or equal to the noise threshold.

In an exemplary embodiment of the present invention, the generation module 550 can implement feature-based control image coding in the following manner, and intervene in the basic image coding of the target round of reasoning until the end of the reasoning process:

In the case where the target round of reasoning is after the reference round of reasoning, the image encoding is controlled based on the features, and the basic image encoding corresponding to the activation area is locally replaced and intervened according to the second preset replacement rate until the end of the inference process, where, The second preset replacement rate is less than the replacement rate threshold; the activation area is the intervention area that needs to be intervened in the painting image to be generated determined according to the image characteristics of the feature control image; the Gaussian noise of the basic image encoding obtained after the reference round of inference is less than Noise threshold.

In an exemplary embodiment of the present invention, the feature control image may include two or more, and the corresponding feature control image encoding may include two or more;

The generation module 550 can implement feature-based control of image coding in the following manner, and intervene in the basic image coding of the target round of reasoning until the end of the reasoning process:

Determine the replacement weight value of each feature control image encoding separately;

According to the replacement weight value, the basic image coding of the target round reasoning is replaced and intervened until the end of the reasoning process based on the feature control image coding.

In an exemplary embodiment of the present invention, the generation module 550 can use the following method to replace the basic image coding of the target round of reasoning based on feature control image coding and intervene until the end of the reasoning process:

Perform multiple rounds of iterative noise on the feature control image coding to the round corresponding to the target round of reasoning, and obtain the feature control image coding after noise addition;

The image coding is controlled based on the features after adding noise, and the basic image coding of the target round reasoning is replaced and intervened until the end of the reasoning process.

In an exemplary embodiment of the present invention, when the diffusion model is a diffusion model from depth information to images, the generation module 550 can implement feature-based control image coding in the following manner, and perform basic image coding for target round reasoning. Intervene until the end of the reasoning process:

Control image coding based on features, and intervene in the target channel of the basic image coding of the target round inference until the end of the inference process. The target channel is a channel other than the channel corresponding to the depth information, and the depth information is a channel corresponding to the basic image coding. Image depth information.

In an exemplary embodiment of the present invention, the feature control image may include a feature control image with preset depth image features, a feature control image with preset edge structure image features, and a feature control image with preset pose image features. one or several kinds.

Figure 6 illustrates a schematic diagram of the physical structure of an electronic device. As shown in Figure 6, the electronic device may include: a processor (processor) 610, a communications interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640. Among them, the processor 610, the communication interface 620, and the memory 630 complete communication with each other through the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a painting image generation method, which method includes: obtaining text information and a feature control image, where the text information is text describing an image frame of the painting image to be generated, so The painting image to be generated has the image characteristics of the characteristic control image; based on the text information, a text code corresponding to the text information is obtained; based on the text code, each diffusion model is used in a multi-round iterative reasoning process. Intervene with the image coding obtained in each round of reasoning to obtain the basic image coding of each round of reasoning; based on the feature control image, obtain the feature control image coding corresponding to the feature control image; based on the feature control image coding, The basic image encoding of target round reasoning intervenes until the end of the reasoning process to obtain the painting image to be generated.

In addition, the above-mentioned logical instructions in the memory 630 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program code. .

On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can Executing the painting image generation method provided by the above methods, the method includes: obtaining text information and a feature control image, wherein the text information is text describing the image frame of the painting image to be generated, and the painting image to be generated has the The above characteristics control the image characteristics of the image; based on the text information, a text encoding corresponding to the text information is obtained; based on the text encoding, the image encoding obtained by the diffusion model in each round of iterative reasoning process is coded Intervene to obtain the basic image coding of each round of reasoning; based on the feature control image, obtain the feature control image coding corresponding to the feature control image; based on the feature control image coding, the target round of reasoning is Basic image coding intervenes until the end of the inference process to obtain the painting image to be generated.

In another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. The computer program is implemented when executed by a processor to execute the painting image generation method provided by each of the above methods. The method includes : Obtain text information and a feature control image, wherein the text information is text describing the image frame of the painting image to be generated, and the painting image to be generated has the image features of the feature control image; based on the text information, obtain Text coding corresponding to the text information; based on the text coding, intervene in the image coding obtained by the diffusion model in each round of inference in the multi-round iterative reasoning process to obtain the basic image coding of each round of reasoning; based on the The feature control image is used to obtain the feature control image coding corresponding to the feature control image; based on the feature control image coding, intervention is performed on the basic image coding of the target round reasoning until the end of the reasoning process to obtain the to-be-determined Generate painting images.

The device embodiments described above are only illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in One location, or it can be distributed across multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Persons of ordinary skill in the art can understand and implement the method without any creative effort.

Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the part of the above technical solution that essentially contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, magnetic disc, optical disk, etc., including a number of instructions to cause a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments.

It should be further understood that although the operations are described in a specific order in the drawings in the embodiments of the present invention, this should not be understood as requiring that these operations be performed in the specific order shown or in a serial order, or that it is required that the operations be performed in a specific order. Perform all operations shown to obtain the desired results. In certain circumstances, multitasking and parallel processing may be advantageous.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be used Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent substitutions are made to some of the technical features; however, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A painting image generation method, characterized in that the method includes: Obtain text information and a feature control image, wherein the text information is text describing the image frame of the painting image to be generated, and the painting image to be generated has the image features of the feature control image; Based on the text information, obtain a text encoding corresponding to the text information; Based on the text coding, intervene in the image coding obtained by each round of reasoning in the multi-round iterative reasoning process of the diffusion model to obtain the basic image coding of each round of reasoning; Based on the characteristic control image, obtain a characteristic control image code corresponding to the characteristic control image; The image encoding is controlled based on the characteristics, and the basic image encoding of the target round inference is intervened until the end of the inference process to obtain the painting image to be generated. 2. The painting image generation method according to claim 1, characterized in that the control of image coding based on the characteristics intervenes in the basic image coding of target round reasoning until the end of the reasoning process, specifically including: When the target round of reasoning is before the reference round of reasoning, the image coding is controlled based on the characteristics, and the basic image coding is fully replaced and intervened according to the first preset replacement rate until the end of the reasoning process, where , the first preset replacement rate is greater than or equal to the replacement rate threshold; the Gaussian noise of the basic image encoding obtained before the reference round of inference is greater than or equal to the noise threshold. 3. The painting image generation method according to claim 1, characterized in that the control image encoding based on the characteristics intervenes in the basic image encoding of the target round inference until the end of the inference process, specifically including: In the case where the target round of reasoning is after the reference round of reasoning, the image coding is controlled based on the characteristics, and the basic image coding corresponding to the activation area is locally replaced and intervened according to a second preset replacement rate until The reasoning process ends, wherein the second preset replacement rate is less than the replacement rate threshold; the activation area is an intervention area that requires intervention in the painting image to be generated determined according to the image characteristics of the feature control image; in The Gaussian noise of the base image encoding obtained after the reference round of inference is less than the noise threshold. 4. The painting image generation method according to any one of claims 1 to 3, characterized in that the characteristic control images include two or more, and the corresponding characteristic control image codes include two or more more than The method of controlling image coding based on the characteristics and intervening in the basic image coding of target round reasoning until the end of the reasoning process specifically includes: Determine the replacement weight value of each of the feature control image encodings respectively; According to the replacement weight value, the basic image coding of the target round of reasoning is replaced and intervened until the end of the reasoning process based on the feature control image coding. 5. The painting image generation method according to claim 4, characterized in that the control image coding based on the characteristics replaces and intervenes in the basic image coding of target round reasoning until the end of the reasoning process, specifically including: Perform multiple rounds of iterative noise on the feature control image coding to the round corresponding to the target round of reasoning to obtain the noise-added feature control image coding; The image encoding is controlled based on the noise-added features, and the basic image encoding of the target round inference is replaced and intervened until the end of the inference process. 6. The painting image generation method according to claim 1, characterized in that, when the diffusion model is a diffusion model from depth information to an image, the image encoding is controlled based on the characteristics, and the target round is The basic image encoding of reasoning intervenes until the end of the reasoning process, specifically including: The image encoding is controlled based on the characteristics, and the target channel of the basic image encoding of the target round inference is intervened until the end of the inference process, wherein the target channel is a channel other than the channel corresponding to the depth information, and the depth The information is image depth information corresponding to the basic image encoding. 7. The painting image generation method according to claim 1, characterized in that the feature control image includes a feature control image with preset depth image features, a feature control image with preset edge structure image features, and a feature control image with preset edge structure image features. The characteristics of pose image features control one or more types in the image. 8. A painting image generating device, characterized in that the device includes: An acquisition module, configured to acquire text information and a feature control image, where the text information is text describing the image frame of the painting image to be generated, and the painting image to be generated has the image features of the feature control image; A text encoding module, configured to obtain a text encoding corresponding to the text information based on the text information; An intervention module, configured to intervene in the image coding obtained by each round of reasoning in the multi-round iterative reasoning process of the diffusion model based on the text coding, and obtain the basic image coding of each round of reasoning; An image coding module, configured to obtain a feature control image coding corresponding to the feature control image based on the feature control image; A generation module, configured to control image coding based on the characteristics, and intervene in the basic image coding of the target round of reasoning until the end of the reasoning process to obtain the painting image to be generated. 9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that when the processor executes the program, it implements claim 1 The painting image generation method described in any one of to 7. 10. A non-transitory computer-readable storage medium with a computer program stored thereon, characterized in that when the computer program is executed by a processor, the painting image generation method as claimed in any one of claims 1 to 7 is implemented. .