CN118691923A - Method, device, computer equipment and storage medium for generating image of target subject - Google Patents

Abstract

The present application relates to a method, device, computer equipment, storage medium and computer program product for generating an image of a target theme. The method comprises: obtaining a theme image generation model obtained by performing secondary training on a pre-trained model through sample images; the pre-trained model is used to generate images according to text; the sample image contains at least one theme element that matches the target theme; based on the theme element description text for describing the theme element carried by each of the sample images, a text set containing the theme element description text of the same type is obtained; theme element description texts are respectively selected from at least a part of the text set, and a target description text containing the selected theme element description text is obtained by combining; through the theme image generation model, image generation processing is performed according to the target description text to obtain a target image matching the target theme. The present method can generate a high-quality image matching the target theme.

Description

Method, device, computer equipment and storage medium for generating image of target subject

Technical Field

The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, storage medium and computer program product for generating an image of a target subject.

Background Art

With the development of artificial intelligence technology, intelligent image generation technology has emerged, which can generate images based on the text description input by the user, and the generated images have relatively fine details.

The technical method of generating images based on text can switch to a specified theme style when generating images. However, for images with specified theme grids, a lot of parameter adjustment and trial work is required. The image generation process has the problem of unstable quality, and it is difficult to generate high-quality images that meet the theme style.

Summary of the invention

Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for generating an image of a target theme that can generate high-quality images that conform to the theme style in order to address the above technical problems.

In a first aspect, the present application provides a method for generating an image of a target subject. The method comprises:

Obtaining a subject image generation model obtained by secondary training a pre-trained model through sample images; the pre-trained model is used to generate images according to text; the sample image contains at least one subject element that conforms to the target subject;

Based on the subject element description text for describing the subject element carried by each of the sample images, obtaining a text set containing the subject element description text of the same type;

Selecting subject element description texts from at least a portion of the text set, respectively, and combining them to obtain a target description text containing the selected subject element description texts;

The subject image generation model is used to perform image generation processing according to the target description text to obtain a target image that matches the target subject.

In a second aspect, the present application also provides a device for generating an image of a target subject. The device comprises:

A model acquisition module, used to acquire a theme image generation model obtained by retraining a pre-trained model through sample images; the pre-trained model is used to generate images according to text; the sample image contains at least one theme element that conforms to the target theme;

A text set determination module, configured to obtain a text set containing the same type of subject element description texts based on the subject element description texts carried by each of the sample images and used to describe the subject element;

A description text determination module, used to select subject element description texts from at least a part of the text set, and combine them to obtain a target description text containing the selected subject element description texts;

The target image generation module is used to perform image generation processing according to the target description text through the subject image generation model to obtain a target image matching the target subject.

In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned method for generating an image of a target subject when executing the computer program.

In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned method for generating an image of a target subject when executed by a processor.

In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the image generation method of the target subject are implemented.

The above-mentioned target theme image generation method, device, computer equipment, storage medium and computer program product obtain a model that can generate images that meet the target theme according to text by acquiring a theme image generation model obtained by secondary training of a pre-trained model. In the process of determining the text of the input model, the sample images of the training theme image generation model are used to contain at least one theme element that meets the target theme, and the theme element description text for describing the theme element carried by each sample image. A text collection containing the same type of theme element description text can be used as a text component of the input model. By selecting theme element description texts from at least a part of the text collection, target description texts containing the selected theme element description texts are obtained by combining them. This enables the theme image generation model to perform image generation processing according to the target description text to obtain a high-quality target image that is different from the sample image but can highly match the target theme.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a diagram showing an application environment of a method for generating an image of a target subject in one embodiment;

FIG2 is a schematic diagram of a flow chart of a method for generating an image of a target subject in one embodiment;

FIG3 is a schematic diagram of image screening by comparing a target image with different candidate images in one embodiment;

FIG4 is a schematic diagram of a process of acquiring sample images based on a target subject video in one embodiment;

FIG5 is a schematic diagram of the structure of a subject image generation model in one embodiment;

FIG6 is a schematic diagram of an image encoding and decoding process in a subject image generation model in one embodiment;

FIG7 is a schematic diagram of a denoising process in a subject image generation model in one embodiment;

FIG8 is a schematic diagram of the structure of a Unet model in a subject image generation model in one embodiment;

FIG9 is a schematic diagram of the training process and application process of a subject image generation model in one embodiment;

FIG10 is a schematic diagram of a layout diagram of segmented semantic objects in one embodiment;

FIG11 is a schematic diagram of an image generated according to a layout diagram of a semantic object in one embodiment;

FIG12 is a schematic diagram of an image generated based on the influence of a depth map in one embodiment;

FIG13 is a schematic diagram of different sample images in one embodiment;

FIG14 is a schematic diagram of different generated images in one embodiment;

FIG15 is a block diagram of a device for generating an image of a target subject in one embodiment;

FIG16 is a diagram showing the internal structure of a computer device in one embodiment;

FIG. 17 is a diagram showing the internal structure of a computer device in one embodiment.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

The technical terms involved in this application are defined below:

IP (Intellectual Property): In the content field of the Internet, it is defined as a cultural brand that can be developed in multiple dimensions, with characteristics such as subject value recognition (which can be a character image, a specific story, a story scene), and multiple carriers (which can be novels, comics, animations, TV series, movies, real people, virtual images, etc.).

Input text description (Prompt): Based on the text description, the model can generate the corresponding image. Prompt contains several elements, namely the object of the painting (the main body of the painting, usually as the foreground), the details of the painting object (clothing, expression, color, action, facial and body details, etc.), the background of the painting, the shooting angle and distance (angle includes front, side, back, etc.; distance includes close-up, half-body, full-body, etc.).

The method for generating an image of a target theme provided in an embodiment of the present application can be applied in an application environment as shown in FIG. 1 . Among them, the terminal 102 communicates with the server 104 via a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on a cloud or other server. In the data storage system, a pre-trained model for generating images by text is stored. The server 104 can receive the image generation request time for the target theme sent by the terminal 102, and the server 104 obtains a sample image containing at least one theme element that meets the target theme, and obtains a theme image generation model by performing secondary training on the pre-trained model through the sample image. It can be understood that in other embodiments, the server 104 can pre-train the pre-trained models through sample images of multiple different themes to obtain multiple theme image generation models with different themes. After receiving the request from the terminal 102, the server 104 can directly select a theme image generation model that matches the target theme from the trained theme image generation models of multiple different themes.

After determining the subject image generation model that matches the target subject, server 104 can obtain a text collection containing subject element description texts of the same type based on the subject element description texts carried by each sample image for describing the subject elements, so as to select subject element description texts from at least a part of the text collection respectively, thereby combining to obtain a target description text containing the selected subject element description texts.

In one embodiment, the target description text can be randomly selected and combined by the server 104 based on the text set, or the server 104 can feed back the text set to the terminal 102. The terminal user can select the subject element description text from the text set to combine and obtain the target description text containing the selected subject element description text and feed it back to the server 104. The server 104 uses the subject image generation model to perform image generation processing according to the target description text, obtains the target image matching the target subject and feeds it back to the terminal 102.

In another embodiment, the server 104 may also feed back the text set and the subject image generation model to the terminal 102, so that the terminal 102 randomly selects and combines the target description text based on the text set, or determines the target description text in response to a description text selection operation triggered by the terminal user for the text set, and then uses the subject image generation model to perform image generation processing according to the target description text to obtain a target image matching the target subject.

The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster or cloud server consisting of multiple servers.

In one embodiment, as shown in FIG2 , a method for generating an image of a target subject is provided, and the method can be applied to a computer device. The computer device can be a terminal or a server. For ease of description, the following embodiments are described by taking the method applied to a computer device as an example, and the method for generating an image of a target subject specifically includes the following steps:

Step 202, obtaining a subject image generation model obtained by performing secondary training on the pre-trained model through sample images.

Among them, the pre-trained model is a technical method for generating images based on text. The model for generating images according to text can generate images according to the input text description, and the generated images have relatively detailed details. Among them, the technical method for generating images from text includes GAN (Generative Adversarial Network) framework, autoregressive framework, and diffusion framework (Diffusion). Among them, the idea of the GAN framework is to train the image encoder, image decoder and GAN discriminator uniformly, use CNN (Convolutional Neural Networks) structure to capture local features, and VIT structure to capture global features. The autoregressive framework encodes the image into a discrete sequence, so that both the image and the text are represented by a sequence, and then uses the autoregressive language model of the NLP (Natural Language Processing) model to generate text to image and image to text. The idea of the Diffusion diffusion framework is to encode the image into the hidden variable domain of the two-dimensional structure, and perform corresponding diffusion process, noise addition process, and text control training in the hidden variable domain. In this embodiment, the pre-trained model can be a model obtained by training any one of the above various frameworks.

Secondary training is a fine-tuning training for a pre-trained pre-trained model so that the pre-trained model can have better data processing capabilities. In some embodiments, different secondary trainings can be performed on the same pre-trained model based on different purposes so that each model obtained after secondary training can achieve different functions. For example, for different IP themes, sample images of different IP themes can be used to train the same pre-trained model that can generate images according to text, so that different models obtained by training can generate images of different IP themes.

The theme image generation model is a model that is obtained by retraining the pre-trained model with sample images that match the target theme, and can generate images that match the target theme based on input text. It should be noted that in order to ensure that the images generated by the theme image generation model have a high degree of fit with the target theme, the input text of the theme image generation model can be text that matches the target theme.

The sample image is an image that contains at least one subject element that matches the target subject and carries a subject element description text for describing the subject element. It can be understood that the number of sample images used for model training is multiple. Each sample image can have the same image parameters, such as the same size, the same resolution, etc., so that the same amount of information is obtained when performing vector conversion based on the sample image during the secondary training of the model, and the computing resources of the model can be balanced during the processing, thereby improving data processing efficiency.

Among them, each sample image may have the same data source or different data sources. The data source may include obtaining from an image web page search, or obtaining from the same video by extracting image frames. When the acquired images have different image parameters, the image parameters may be aligned first so that the acquired images have the same image parameters. For example, when the sample image is an image of a specified IP theme, the specified IP theme is a TV series, and some image frames containing at least one theme element that meets the target theme can be selected from the TV series video, and the theme elements of each image frame are annotated to obtain a theme element description text that describes the theme elements and matches the image frame, thereby obtaining a sample image of the specified IP theme.

The subject elements that match the target theme refer to the specific elements of the target theme in the image. Specifically, they can be the characters, specific stories, story scenes, etc. of the target theme. For example, they can be the main characters and specific scenes in a TV series. Taking a certain costume TV series as an example, the main characters can include the male protagonist A, the female protagonist B, etc.; the specific scenes can be representative scenes in the TV series, such as famous scenes that are widely circulated.

The subject element description text is a text that describes the content of the subject element in the sample image. For example, the subject element description text for the sample image of a TV series may include at least one of the following contents: the main character, the character's appearance, the character's expression, the background in which the character is located, and the angle of view of the character in the picture. In some specific embodiments, the description of the character's appearance can be divided into the description of clothing and the description of hairstyle. For example, the description of the protagonist A's clothing includes: a light yellow-green collared brocade robe, a dark purple silk waistcoat, a white cotton and linen suit, etc. The description of the protagonist A's hairstyle includes: half-loose hair and tied hair, etc.; the character's expression includes: calm, angry, doubtful, etc.; the description of the character's background includes: a carriage, a square, etc.; the description of the angle of view of the character in the picture includes: a close-up of the head, a half-body mid-shot, a full-body long shot, etc.

In some specific embodiments, a computer device may respond to a training request for a subject image generation model for a target subject, obtain sample images that match the target subject based on a pre-trained model for generating images from text, and perform secondary training of the model in real time to obtain a subject image generation model.

In other specific embodiments, the computer device may also pre-train theme image generation models for different themes. In response to a request to obtain a theme image generation model for a target theme, the computer device searches for a theme image generation model that matches the target theme from multiple trained theme image generation models.

Step 204 : based on the subject element description text for describing the subject element carried by each sample image, a text set containing the subject element description text of the same type is obtained.

Among them, each sample image carries a subject element description text for describing the subject element. The subject element of each sample image is described from multiple angles. For each description angle, there are many sample images with corresponding subject element description texts, and each subject element description text of the description angle can constitute a text set. For example, for the subject element of each sample image, it is described from six angles: main character, character clothing, character hairstyle, character expression, character background, and appearance perspective. Accordingly, six different types of text sets can be constructed: main character, character clothing, character hairstyle, character expression, character background, and appearance perspective. The composition of the description text in the text set can be obtained by deduplicating the subject element description text of the angle in each sample image, and the result is used as the description text contained in the text set of the angle.

It should be noted that, in other embodiments, description texts of other external elements besides the target theme may be added to the text collection. For example, by adding external characters to the text collection of the main characters, the external characters may be integrated with the core scene of the target IP to embed the target IP scene. The added external elements may be elements that can obtain specific image information based on a specific path or elements that can extract corresponding image information based on images uploaded by users. For example, in the text collection of the main characters, the names of famous people widely circulated on the Internet or the names of characters in uploaded photos of the characters may be added.

Step 206, selecting subject element description texts from at least a portion of the text set, respectively, and combining them to obtain a target description text containing the selected subject element description texts.

Wherein, the text set used to select the subject element description text can be a text set of each type constructed, or a part of the text set in the constructed text set. For example, in the construction of the text set, a total of 6 text sets are constructed, then the text set used to select the description text can be 6 text sets, or 5 text sets or 4 text sets, etc. Wherein, the target description text can include multiple subject element description texts selected from the text set, or can include the subject element description text selected from the text set, and the object element description text of the external object. Wherein, the object element description text of the external object can be pre-added to the text set corresponding to the object element obtained in step 204, so as to be directly selected from the text set corresponding to the object element. It can be understood that in other embodiments, the object element description text of the external object can also be selected from the text set that only contains the object element description text of the external object, and the specific acquisition method of the object element description text of the external object is not limited here.

In some specific implementations, the computer device may randomly select subject element description texts from at least a portion of the text collections, and combine them to obtain a target description text containing the selected subject element description texts. The computer device may also display each text collection on a display interface, and in response to a user's selection operation of a subject element description text in a portion of the text collection, combine multiple description texts including the subject element description text selected by the user to obtain a target description text.

In some specific applications, taking the example that the subject elements include main characters, when the number of the main characters is one, at most one object element description text is selected from each text set to avoid the situation where the description content conflicts, thereby affecting the quality of the generated image. When the number of the main characters is two or more, the number of object element description texts selected from each text set can be one, or a number less than or equal to the number of the main characters.

Accordingly, when the number of main characters is two or more and the number of object element description texts selected from each text set is 1, the two or two main characters in the generated image can have the same and similar expressions at the angle corresponding to the description text. When the number of main characters is N and the number of object element description texts selected from each text set is N, the N characters in the generated image can have different expressions at the angle corresponding to the description text. Step 208, through the subject image generation model, image generation processing is performed according to the target description text to obtain a target image that matches the target subject.

Specifically, the target description text is the input data of the subject image generation model. Since the subject image generation model is a model obtained by retraining the pre-trained model with sample images carrying subject element description texts, and the target description text is a combination result selected from the text set containing the subject element description texts of the sample images, the computer device can perform image generation processing according to the target description text through the subject image generation model to obtain a high-quality target image matching the target subject.

In some embodiments, the subject image generation model is a pre-trained model based on the Diffusion diffusion framework, and the model is obtained by fine-tuning the pre-trained model of the framework based on the sample image of the target subject. The data processing process of the pre-trained model based on the Diffusion diffusion framework includes a diffusion process and a denoising process, so the subject image generation model obtained by fine-tuning training also has a diffusion process and a denoising process. The diffusion process refers to the process of gradually converting a data sample (a hidden variable obtained by encoding the image) into random noise by adding noise. The denoising process is to learn the noise in each step of the denoising process through a complex Unet model, so that the difference between the estimated noise and the actual noise is as small as possible. The input of the denoising process is a white noise. Through the noise estimation of multiple steps of noise, the noise of the image is made smaller and smaller, and finally a high-definition image is sampled.

Specifically, in addition to obtaining the target description file, the computer device can also obtain a random seed, and input the target description file and the random seed into the theme image generation model. The random seed uses a random number generator to generate a random latent variable, and the target description file is encoded by a text encoder to generate a text vector. Based on the text vector and the random latent variable, the image latent variable can be reconstructed, and then the random latent variable is decoded into an image, so that a high-quality target image matching the target theme can be obtained.

The above-mentioned method for generating an image of a target theme obtains a theme image generation model obtained by performing secondary training on a pre-trained model, thereby obtaining a model that can generate an image that conforms to the target theme according to text. In the process of determining the text of the input model, the sample images of the trained theme image generation model are used to contain at least one theme element that conforms to the target theme, and the theme element description text for describing the theme element carried by each sample image. A text set containing theme element description texts of the same type can be used as a text component of the input model. By selecting theme element description texts from at least a part of the text set, target description texts containing the selected theme element description texts are obtained by combining them. This enables the theme image generation model to perform image generation processing according to the target description text, thereby obtaining a high-quality target image that is different from the sample image but can be highly matched with the target theme.

Next, the following embodiments are used to illustrate the method of obtaining sample images for model training.

In some embodiments, sample images can be obtained from a video, and the specific process includes: obtaining a target video that matches the target theme, performing frame extraction processing on the target video, and obtaining multiple candidate image frames with different scenes; performing theme element identification on each candidate image frame, and determining a preferred image frame that contains at least one theme element; based on the theme elements, annotating the preferred image frame with text describing the theme elements to obtain a sample image.

The target video that meets the target theme may be a video for a specific IP, such as a TV series, movie, cartoon, animation, etc. The target video is a video that is presented around the target theme and has at least one of a specific character image, a specific story, and a story scene. The specific character image, the specific story, and the story scene may serve as a theme element. For example, if the target video has a specific character image, the theme element of the target video may be the protagonist of the target video.

The target video is composed of multiple video frames. In order to achieve effective training of the model, the target video can be processed by frame extraction, and a part of the video frames with better effects can be extracted to be used as sample images for model training. In a specific embodiment, the frame rate of the video file is 30frame/s, and a video episode is about 40min-120min. On average, 70k-210k images can be extracted from each video file. However, since the image content changes very little between consecutive frames of the video file, only image frames with relatively large scene changes need to be extracted. Through this method, for a 40min video episode, generally only 2k-4k images need to be extracted, which can effectively realize the screening of video frames in the target video, improve the effectiveness of sample images, and further improve the model training effect, shorten the training time required for the model training process to achieve the desired effect, and reduce the occupation of data processing resources of the computer equipment used to train the model.

The scene is used to characterize the main content in the image frame. The image frames with different scenes can be multiple image frames with one or more differences among different backgrounds, different characters, different perspectives, different character shapes, etc. By extracting multiple candidate image frames with different scenes, the content of the image frames used as sample images can be greatly different, thereby improving the training effect of the sample images in model training.

Theme element recognition is a process for determining whether an image frame contains a theme element. In a specific application, an image frame of a target video may have one theme element, two theme elements, or no theme element. In the process of screening image frames, image frames that do not contain theme elements are eliminated so that the image frames obtained after filtering all have at least one theme element, such as the preferred image frame obtained by eliminating image frames that do not contain theme elements from candidate image frames in this embodiment.

Thematic element description text annotation refers to the process of using text to describe the thematic elements in the image frame. Thematic element description text is the text that describes the elements appearing in the image frame. Thematic element description text can include the main object (the main body of the image, generally the foreground), image background, shooting angle and distance, and some detailed descriptions (clothing, expression, color, action, facial and body details, etc.). Among them, the main object can be a character in the IP video, which can be described by the name of the character. For example, Zhang Xiaoming, the protagonist in "TV Series A". The image background can be a scene in the IP video, which can be a building, a natural landscape, or a storyline. For example, the ancient pavilions, ancient temples, and famous scenes of holding large flags in "TV Series A". Detailed description-clothing category: unique clothing in IP videos. For example, the clothes of the male and female protagonists in "TV Series A" are described in natural text, light yellow and green collar brocade robes, blue cotton and linen suits, blue and white checkered cotton and linen suits, dark purple silk waistcoats, and white cotton and linen suits. Detailed description-expression category: the expression of the character in the IP video, the classification of expression includes calm, serious, happy, angry, confused, etc. Based on the above text description, for each image frame, one or more text descriptions will be generated, which form paired training data with the image frame, such as the sample image carrying the text describing the subject elements in the above embodiment.

In some specific embodiments, when the subject element is a specific person, each subject element in the image frame can be described from the perspectives of the main person, the person's clothing, the person's hairstyle, the person's expression, the person's background, the viewing angle, etc. When the subject element is a specific static object, each subject element in the image frame can be described from the perspectives of the object name, the object background, the viewing angle, etc.

In this embodiment, the computer device obtains a target video that matches the target theme, performs frame extraction processing on the target video, obtains multiple candidate image frames with different scenes, implements preliminary filtering of similar image frames, and then performs theme element recognition on each candidate image frame, determines the preferred image frame containing at least one theme element, removes invalid image frames that do not contain theme elements, and finally annotates the preferred image frame with a description of the theme element based on the theme element, so as to obtain high-quality sample images conveniently and efficiently. It can be understood that in other embodiments, the execution order of each step can be reorganized according to actual needs.

The frame extraction process is described in detail below. It is understood that in other embodiments, the frame extraction process can also be implemented in other ways, which are not limited here. In some embodiments, the target video is subjected to frame extraction processing to obtain multiple candidate image frames with different scenes, including:

The target video is frame-processed to obtain multiple image frames with the same pixel arrangement; for two video frames in the multiple image frames, the sum of the absolute values of the pixel granularity differences at the same pixel arrangement position is determined; when the ratio between the sum of the absolute values and the total value of the pixel granularity of the image frame is greater than a scene change threshold, the two image frames are determined to be candidate image frames with different scenes.

Among them, pixel arrangement refers to the relative position of each pixel that constitutes the image frame. Since the image frame is obtained by dividing the frame from the target video, for the same target video, the video frame obtained by extracting the frame through the same processing method has the same image attributes, and the same image attributes include the same pixel arrangement. Pixel granularity is a parameter used to describe the specific content of each pixel point in the image frame, which can be one of the parameters such as RGB value and grayscale value. For two image frames that are compared with each other, the computer device can obtain the pixel granularity difference of each pixel arrangement position by performing difference processing on the specific parameter values of the pixel granularity of the same pixel arrangement position of the two image frames, wherein the pixel granularity difference can be a positive value or a negative value. In order to avoid interference caused by the numerical offset between positive and negative values, the computer device accumulates the absolute value of the pixel granularity difference of each pixel arrangement position to obtain the sum of the absolute values to characterize the difference between the two image frames that are compared with each other.

Whether there is a scene difference between two image frames can be specifically reflected by comparing the ratio between the sum of absolute values and the total value of pixel granularity of the image frame with the numerical value of the scene change threshold. In a specific embodiment, for the convenience of explanation, the image frame is simplified into a single-channel 3x3 image, as shown in FIG3, which are the target image, candidate image 1 and candidate image 2. For example, by comparing the target image with candidate image 1, by calculating the pixel granularity difference of each pixel point in the target image and candidate image 1, it can be obtained that the sum of the absolute values of the pixel granularity difference is 20, the total value of the pixel granularity of the target image is 44, and the ratio is 20/44=0.45. The threshold of scene change is set to 0.3, that is, the ratio of scene change is required to exceed 0.3, then candidate image 1 meets the condition, and the target image and candidate image 1 can be determined as two images with scene differences.

Similarly, as shown in Figure 3, comparing the target image with candidate image 2, by calculating the pixel granularity difference of each pixel in the target image and candidate image 2, the sum of the absolute values of the pixel granularity difference can be obtained to be 4, the total pixel granularity of the target image is 44, and the ratio is 4/44 = 0.09. The threshold of scene change is set to 0.3, that is, the ratio of scene change is required to exceed 0.3, then candidate image 2 does not meet the condition.

Furthermore, when the comparison result between the target image and the candidate image is that the candidate image does not meet the conditions, the candidate image can be discarded and the next image of the candidate image can be compared with the target image. When a candidate image with a comparison result that meets the conditions appears with the target image, the candidate image can be used as a new target image and compared with the next image. By judging the scene changes of image frames arranged in sequence, efficient and accurate filtering of image frames can be achieved while effectively reducing the number of comparisons, thereby improving the quality of sample images.

In some embodiments, subject element identification is performed on each candidate image frame to determine a preferred image frame containing at least one subject element, including: obtaining attribute information and content information of each candidate image frame; discarding candidate image frames whose attribute information and content information do not satisfy at least one subject element identification condition, to obtain a preferred image frame containing at least one subject element.

Among them, the attribute information of the image frame includes the attributes of the image frame in the target video, such as picture clarity, whether it is an invalid image frame such as a title image frame, an end image frame, an advertisement image frame, a transition image frame, etc. that is not suitable for use as a sample image. The attribute information of whether it is an invalid image frame can be determined by comparing it with a pre-configured attribute identifier, or by identifying the start and end time nodes of the title, end, advertisement, and transition in the target video, and comparing the time information of the image frame with the above-mentioned start time nodes, thereby determining the attribute information of the image frame. In other embodiments, the attributes of the image frame can also be determined by using image recognition tools, including title and end recognition tools and advertisement recognition tools. Content information refers to the content contained in the image frame, which can specifically be whether the character in the picture is the protagonist, whether the scene is recognizable, whether the character in the picture is completely captured, etc.

For the image frame used for the sample image, the attribute information and content information need to meet the subject element identification conditions at the same time. The subject element identification conditions may specifically include attribute information screening conditions and content information screening conditions. Only when the attribute information of the sample image meets the attribute information screening conditions and the content information meets the content information screening conditions can it be ensured that the preferred image can contain at least one subject element, and the image frame can be determined as a preferred image frame that meets the subject element identification conditions.

In a specific application, IP video may include some opening and ending credits, advertisements, and transition images. In addition, the plot may include some supporting characters, unrecognizable scenes, low image clarity, and incomplete images of characters, all of which are not suitable as training corpus. The computer device obtains the attribute information and content information of each candidate image frame; the attribute information and content information are screened separately, and the candidate image frames whose attribute information and content information do not meet the subject element recognition conditions are discarded to ensure that the preferred image frame contains at least one subject element.

In some embodiments, the subject elements include object elements that conform to the target subject; based on the subject elements, the preferred image frame is annotated with a text description of the subject elements to obtain a sample image, including:

For each preferred image frame, the center point of the object element in the preferred image frame is located to obtain the located position; according to the size conditions of the sample image, the preferred image frame is cropped with the located position as the center to obtain a cropped preferred image frame; the cropped preferred image frame is annotated with a text description of the subject element to obtain a sample image.

Among them, the object element refers to the theme element that exists in the form of a core object. For example, the protagonist in a TV series, the main virtual image in a cartoon, etc. Center point positioning refers to the process of determining the center position of the object element in the preferred image frame for positioning. The size condition of the sample image refers to the size that the sample image of the input model needs to meet. By cropping the preferred image frame with the positioning position as the center, the cropped preferred image frame is obtained, which can achieve effective display of the object element.

In some embodiments, the image of the IP video frame extraction is generally a widescreen image of 1920x1080 in horizontal screen, while the input requirement of the training image is relatively small (such as 512x512 or 768x768, etc.). If the image is directly scaled, it will lead to the loss of image clarity and insufficient detail resolution of the main elements. The above embodiment can improve the quality of the sample image while ensuring that the sample images have the same size by locating the object element and taking the object element as the center to perform image interception of the target size.

In a specific embodiment, as shown in FIG4 , the specific process of obtaining the sample image may include: obtaining a video file that meets the target theme, calculating the absolute value sum of the difference between the pixel granularity of two consecutive frames of images, and comparing the ratio of the absolute value sum to the pixel granularity value of the target image with a set threshold value to realize frame extraction processing of the video file, and then filtering out some opening and ending images, advertisements, and transition images by identifying and judging the quality of the theme elements. The plot may include some supporting character images, unrecognizable scene images, images with low clarity, and incomplete character images. These images are not suitable as image frames for training corpus and therefore need to be filtered, and then locating the theme elements in the image frame, and taking the theme elements as the center, performing image capture of the target size, and finally performing text annotation on each processed image. The text annotation uses words to describe the theme elements appearing in the image, including the main object (the main body of the image, generally the foreground), the image background, the shooting angle and distance, and some detailed descriptions (clothing, expression, color, action, facial and body details, etc.).

Next, a possible implementation method of determining a target text combination based on a text set is introduced through the following embodiment.

In some embodiments, the text set includes an open text set and a closed text set. Among them, the open text set includes object element description text for describing object elements in the subject elements, and object element description text for describing external object elements. External object elements refer to object elements that do not conform to the target theme. For example, taking the object element as the main character as an example, the open text set is used to describe the main character. In addition to the main character in the IP field, it can also be a famous person in the outside world, or a person whose amateur photos are uploaded by the user. Taking "TV Series A" as an example, the characters in the IP field include protagonist A, protagonist B, protagonist C, etc. Famous people in the outside world can be any celebrity in the outside world, such as famous movie actors, famous singers, etc., and can also be amateur photos uploaded by users, generally non-famous people in the outside world, such as people corresponding to photos containing character portraits taken by users with cameras.

A closed text set refers to a text set constructed based on the subject element description text of the subject element. In some embodiments, the element type corresponding to the closed text set and the open text set may be different. For example, when the element type corresponding to the open text set is an object element, the element type corresponding to the closed text set is other subject elements except the object element. For example, the closed text set may specifically include a clothing text set: [light yellow-green collar brocade robe, blue cotton and linen suit, blue and white checkered cotton and linen suit, dark purple silk waistcoat, white cotton and linen suit], a hairstyle text set: [half-open hair, tied hair], an expression text set: [calm, serious, surprised, happy, angry, confused], and a perspective text set: [close-up of the head, mid-length shot of the body, and long shot of the whole body].

Furthermore, subject element description texts are selected from at least a portion of the text set and combined to obtain a target description text containing the selected subject element description texts, including: selecting object element description text from the open text set; combining the object element description text with subject element description texts selected from at least a portion of the closed text set to obtain a target text combination.

Among them, selecting object element description text from the open text set can determine the object in the target image to be generated. Since the open text set includes not only object element description text describing the object element in the subject element, but also object element description text describing the external object element, it is convenient to embed external famous figures or ordinary people as the subject of the target image in the form of text-generated image, thereby integrating the external figures with the core scene of the target theme and realizing the embedding of the target theme scene.

In this embodiment, when the selected object element description text is an external object element, the theme image generation model can obtain the image of the external object element as the initial feature of the latent variable for processing, thereby achieving effective fusion of the external object element with the content of other target topics, thereby expanding the application scenarios of the theme image generation model.

Next, the topic image generation model will be introduced through the following embodiments. The topic image generation model is the result of fine training of the pre-trained model based on the training corpus of a specific IP topic. The topic image generation model has the same model structure as the pre-trained model.

In one embodiment, the model structure of the theme image generation model, i.e., the pre-trained model, is first introduced. As shown in FIG5 , the model includes a control domain, a latent variable domain, and an image domain. Among them, the image domain includes an image encoder and an image decoder, and the image encoder and the image decoder are used to establish a connection between the image domain and the latent variable domain. Among them, the image encoder is used to encode and compress the image vector with the dimension [Channel, Height, Width] in the image domain to obtain a latent variable with the dimension [LatentChannel, LatentHeight, LatentWidth]. Usually, the image is RGB encoded, Channel is 3, Height and Width represent the height and width of the original image; LatentChannel is the number of channels in the latent variable domain, for example, 4 is selected, and LatentHeight and LatentWidth will be reduced by a fixed ratio (scale_factor, an exponent of 2, such as 4, 8, 16, etc.) based on the original image. LatentHeight＝Height/scale_factor, LatentWidth＝Width/scale_factor. To ensure that the height and width of the latent variable domain are integers, the input image is generally aligned, for example, by image resizing and padding operations, to ensure that the width and height of the image can be divided by a fixed ratio scale_factor.

The processing of the image decoder is the inverse of the processing of the image encoder, and is responsible for restoring the latent variables in the latent variable domain to the image vector in the image domain. For example, the image decoder can receive a latent variable with an input dimension of [LatentChannel,LatentHeight,LatentWidth] in the latent variable domain, restore the latent variable to an image vector of [Channel,Heigth,Width], and then generate the corresponding image.

In one embodiment, image encoding and image decoding can be implemented using a VAE framework (Variational Auto-Encoder), and the model framework is shown in Figure 6, where the image encoder (Encoder) is used to encode the image vector of the input image into a latent variable, specifically to estimate the mean E(z) and variance V(z) of the Gaussian distribution of each dimension in the latent variable space, and then sample the Gaussian distribution based on the mean and variance of each dimension to obtain the latent variable. The image decoder (Decoder) is used to decode the latent variable into an image vector.

The control domain includes a text encoder for converting natural text into a text vector of fixed dimension. Specifically, the text encoder can adopt a transformer structure, including the following steps: natural text is converted into an ID sequence through word segmentation and vocabulary lookup by a word segmenter, and the ID sequence is encoded into a sequence vector matrix of [Seq_Len, HiddenSize] through a multi-layer transformer structure; wherein Seq_Len is the length of the sequence and HiddenSize is the dimension of the hidden vector. In some specific embodiments, the text encoder can be obtained using an open source pre-trained model, including CLIPTextEncoder, BertTextEncoder, T5TextEncoder, etc.

The data processing process in the latent variable domain includes diffusion process and denoising process. Diffusion process refers to the process of gradually converting a data sample (the latent variable obtained by VAE encoding) into random noise by adding noise. The specific formula is as follows

X _t ＝α _t X _t-1 +β _t ∈ _t ,∈ _t ～N(0,1)

Among them, Xt _-1 is the hidden variable of the previous step, αt _,t is the pre-set noise weight, and _∈t is the randomly sampled Gaussian noise. For the initial hidden variable, a diffusion noise adding process of more than 1000 steps will be performed. The noise weight will be smaller in the early stage and larger in the later stage. Specifically, it can follow a fixed weight distribution.

As shown in Figure 7, the denoising process uses a complex Unet model to learn the noise in each step of the denoising process, so that the difference between the estimated X _t-1 ' and the actual X _t-1 is as small as possible. The input of the denoising process is a white noise. Through multi-step noise estimation, the noise of the image is reduced and finally a high-definition image is sampled. The loss function of the Unet model training is

By minimizing the difference between the estimated noise at time t and the actual noise, the difference between the estimated X _t-1 ' and the actual X _t-1 can be minimized. After t steps of denoising, the original image of X at time t = 0 is finally restored. The model structure of the denoising process is a Unet structure, as shown in Figure 8. The Unet structure includes a group of downsamplers, a group of upsamplers, and a middle layer of dimensionality-preserving samplers. The downsampler can be a CNN+transformer structure, the upsampler CNN+transformer structure, and the middle layer of samplers has the same structure as the upsamplers and downsamplers, except that the dimensionality is not scaled.

In the process of fine-tuning the pre-trained model based on the training corpus of a specific IP topic, the training framework mainly trains the text encoder of the control domain and the Unet module of the denoising process. For the other two parts of the overall framework, the image encoder and the image decoder, the pre-trained VAE is used. The image encoder and the image decoder are only used for inference and do not participate in training. The diffusion process is a fixed multi-step process of adding Gaussian noise and does not participate in training.

Next, we will introduce the application process of the theme image generation model to generate images. It should be noted that the framework in the application process of the theme image generation model is different from the framework in the training process. Since there is no input image in the framework in the application process, it does not include the encoding process and diffusion process based on the input image in the training framework, but uses random seeds to directly sample random latent variables in the latent variable domain.

In one embodiment, the subject image generation model is used to perform image generation processing according to the target description text to obtain a target image matching the target subject, including:

Based on the target description text input from the control domain, the target description text is converted into a text vector; based on the white noise corresponding to the random latent variables in the latent variable domain, denoising processing is performed under the guidance of the text vector to reconstruct the image latent variables; the image latent variables are decoded in the image domain to obtain a target image matching the target subject.

Specifically, as shown in FIG9 , the computer device inputs the target description text through the control domain of the subject image generation model, and the text encoder based on the control domain converts the target description text into a text vector in the latent variable domain, and generates an image latent variable Z_0 corresponding to the text vector; the random seed randomly generates random Gaussian white noise as a random latent variable in the latent variable domain by using a random number generator, and then based on the random latent variable, a multi-step denoising process is used to reconstruct the image latent variable Z_0 under the guidance of the text vector to obtain Z'_0, and then an image decoder in the image domain is used to obtain the image vector to achieve the generation of the target image. In this embodiment, the subject image generation model can effectively achieve the reconstruction of the image latent variable and achieve fast and high-quality generation of the image by generating random latent variables and performing denoising under the guidance of the text vector.

In the application process of the theme image generation model, by inputting text related to the theme IP, the theme image generation model can generate images that conform to the style of the theme IP, are consistent with the text description, and have great diversity, that is, the generated images are not exactly the same as the training data. In order to further improve the quality of the images generated by the theme image generation model, the following embodiments will introduce the methods of improving image quality in this application from different aspects. It can be understood that the generation of images can be achieved through each embodiment alone, or in combination with other embodiments, which is not limited here.

In some embodiments, the computer device can improve the quality of the generated image by weighting the text description. Specifically, in some embodiments, based on the target description text input from the control domain, the target description text is converted into a text vector, including:

For the target description text input from the control domain, the weight data of the key description text in the target description text is obtained; based on the weight data of the key description text, the key description text is vector-weighted during the vector conversion process to obtain the text vector corresponding to the target description text.

Specifically, the subject image generation model obtained by fine training using the standard training process and model framework has the problem of weak control over some features, which is specifically manifested in that the generated image is inconsistent with the text description. In order to solve this technical problem, in this embodiment, by weighting the key features in the text vector domain, it is ensured that the key features have a higher weight in the image generation process than other texts. The specific implementation method is to merge the weight with the text during the text construction process, and perform corresponding parsing and weighting during the text encoding process.

For example, in a specific application, the target description text is "Character: Protagonist A, Perspective: (half-length mid-shot: 1.1), Hairstyle: tied hair, Clothing: (blue cotton and linen suit: 1.2)". Through the method of (subject element description text: weight), the text part to be weighted can be determined, and different weights can be applied to different texts; among them, the text without brackets can be weighted as 1.0 by default.

In this embodiment, during the application process of the theme image generation model, for the target description text input from the control domain, the weight data of the key description text in the target description text is determined according to the text format of the target description text, and then based on the weight data of the key description text, during the vector conversion process performed by the text encoder, the key description text is vector-weighted, so as to obtain the text vector corresponding to the target description text, which can effectively improve the effective expression of key features in the generated image, thereby improving the image quality.

In some embodiments, the computer device can improve the quality of the generated image by fusing vectors of different layers in the text encoder. Specifically, in some embodiments, the control domain includes a text encoder that converts text into a vector; the text encoder includes at least two network layers. For example, the text encoder can be a transformer model with multiple layers.

Furthermore, based on the target description text input from the control domain, the target description text is converted into a text vector, including: performing vector conversion processing on the input target description text in turn according to each network layer in the text encoder to obtain output features of each network layer; performing feature fusion on the output features of the target network layer of the text encoder and the output features of the last network layer to obtain a text vector corresponding to the target description text; the target network layer is a network layer in the text encoder used to improve text control capabilities.

Specifically, we found in multiple experiments and repeated studies that the semantic control ability of different layers of the text encoder and the clarity of the final image will be different. By determining the target network layer in the text encoder for improving the text control ability, the input features of the target network layer are fused with the output features of the last layer, where the feature fusion method includes weighted interpolation fusion, so that the obtained text vector has better semantic control ability in the image generation process.

Taking the transformer model as an example, the penultimate layer of the transformer can improve the control ability of the text. The text vectors of the penultimate layer and the last layer of the transformer model are interpolated and fused. The specific fusion method is as follows:

text_emb＝alpha*text_emb(L-1)+(1-alpha)*text_emb(L)

Among them, text_emb(L-1) is the text vector of the second-to-last layer, and its dimension is [Seq_Len, HiddenSize]; text_emb(L) is the text vector of the last layer, and its dimension is consistent with text_emb(L-1), so linear interpolation fusion can be performed; alpha is the weight of linear interpolation, and its value is a decimal between [0,1]. The higher the value, the higher the weight of the second-to-last layer.

In this embodiment, by performing feature fusion on features of different layers of the file encoder, a vector with better semantic control ability can be fused to the output layer of the file encoder, so that the vector encoded by the file encoder has better semantic control ability in the image generation process, which can effectively improve the quality of the generated image.

In some embodiments, the computer device can improve the quality of the generated image by adding negative descriptions and weighting the negative descriptions. Specifically, in some of the embodiments, the input text of the control domain also includes negative description text. The negative description text is a text description of the content that is not desired to appear in the image. For example, in order to improve the quality of image generation, the description words of low-quality images such as "low quality, low resolution" can be added to the negative description text.

The noise latent variable predicted by the denoising process of the topic image generation model can be fused with the estimated noise of the input text and the estimated noise of the negative description text using a guidancefree weighted method. In the model training, the negative description text is unconditional without input text. Furthermore, the estimated noise of the input text and the estimated noise of the negative description text can be fused specifically by the following formula:

noise_predict=noise_uncond+scale*(noise_text-noise_uncond)

Among them, noise_predict is the fusion result, noise_uncond is the estimated noise of the negative description text, noise_text is the estimated noise of the input text, and scale is used to control the amplification rate of the difference between noise_text and noise_uncond. The higher the value, the more the generated image will be biased towards the input text and the further away from the text of noise_uncond, thereby achieving the effect of controlling the negative description from appearing.

Furthermore, based on the white noise corresponding to the random latent variables in the latent variable domain, denoising is performed under the guidance of the text vector to reconstruct the image latent variables, including:

The estimated noise of the negative description text is taken as the fusion object in the denoising process and fused with the estimated noise of the target description text to obtain the fused estimated noise; the white noise corresponding to the random latent variable in the latent variable domain is denoised according to the fused estimated noise under the guidance of the text vector to reconstruct the image latent variable.

Specifically, when the computer device performs the denoising process through the subject image generation model, it first identifies whether the input text of the control domain includes negative description text, wherein the target description text and the negative description text can be marked by different identifiers for identification and distinction. After identifying the negative description text, the computer device obtains the estimated noise of the negative description text, and fuses the estimated noise of the negative description text with the estimated noise of the target description text to obtain the fused estimated noise, so that the content of the negative description text can be better eliminated by noise, so as to reduce the probability that the content represented by the negative description text appears in the generated image, and improve the quality of the generated image.

In some embodiments, the computer device can control the way in which different texts take effect at different stages of the denoising process to improve the quality of the generated image. Specifically, in some of the embodiments, the denoising process includes a first stage and a second stage that occur in sequence. The first stage can specifically be the early stage of denoising, and the second stage can specifically be the later stage of denoising. The dividing point between the first stage and the second stage can be set based on actual needs and is not limited here. The target description text includes a main description text and a detailed description text, wherein the main description text can be text related to the theme character and layout, and the detailed description text can be text related to hairstyle, clothing, etc.

In many experiments and repeated studies on the denoising process, it was found that in the early stage of denoising, the main work is to draw the layout of the painting and the main elements, and in the later stage of denoising, the local details are drawn. Based on this discovery, the image latent variables are further decoded in the image domain to obtain the target image that matches the target theme, including:

In the first stage, the mask covers the detail description text, and the subject image latent variables of the subject description text are decoded to obtain a subject image matching the target subject; in the second stage, the mask covers the subject description text, and the detail image latent variables of the detail description text are decoded to obtain a detail image; the detail image is rendered to the corresponding area in the subject image to obtain a target image matching the target subject.

Among them, mask coverage refers to temporarily shielding the detail description text or the main description text so that the subject image generation model will not pay attention to the detail description text or the main description text covered by the mask during the denoising process.

Specifically, the computer device controls the subject image generation model in the early image generation stage corresponding to the denoising process, so that only the main description text related to the subject character and layout is effective, and the detail description text is invalidated by masking, ensuring that more attention is paid to the main description text in the early stage of the image. In the later generation stage, the opposite process is performed to make it easier for the detail description text to control the generation of the image, and the text that controls a large area such as the subject character and layout is covered by a mask, ensuring that more attention is paid to the detail description text in the later stage of the image. The two stages work together to further enhance the control of the generated text and the image refinement of the detail text.

In some embodiments, the computer device can improve the quality of the generated image by controlling the generated objects in different regions through semantic segmentation. Specifically, in some of the embodiments, the method for generating an image of a target subject also includes: semantic segmentation of the target description text, and determining a layout diagram of each semantic object obtained by segmentation in the image based on the semantic object layout configuration parameters; and determining the influence weight of the attention influence mechanism of each layout area of each semantic object in the layout diagram.

Semantic segmentation refers to the process of identifying entity objects in text and segmenting the recognition results to obtain semantic objects. Through semantic segmentation, the objects to be depicted in different areas of the image can be pre-set before image generation. During the generation process, the goal of generating different objects in different areas can be achieved by influencing the weights of the attention influence mechanism of different text fragments in different areas. The semantic object layout configuration parameters are used to determine the layout of the target image for the semantic object and obtain a semantic object layout diagram. Different semantic objects have different influence weights in the semantic object layout diagram. The specific value of the influence weight can be obtained through the semantic object layout configuration parameters.

Furthermore, the image latent variables are decoded in the image domain to obtain a target image matching the target subject, including: decoding the image latent variables obtained based on the influence weights in the image domain to obtain a target image in which each semantic object is laid out according to the layout diagram.

In a specific embodiment, as shown in FIG10 , the input text is: “1 girl in a white skirt with long hair, next to a boy in a red scarf, tree, sky, grass, highly detailed, best quality”, and the following layout diagram of the semantic objects obtained by semantic cutting can be generated through the configuration file. Among them, the white area is “1 girl in a white skirt with long hair”, the black area is “1 boy in a red scarf”, the green area is “tree”, the blue area is “sky”, and the brown area is “grass”. Among them, the influence weight of “1 girl in a white skirt with long hair” on the white area is 1.0, the influence weight of “1 boy in a red scarf” on the black area is 1.4, the influence weight of “tree” on the green area is 1.2, the influence weight of “sky” on the blue area is 0.2, and the influence weight of “grass” on the brown area is 0.2. Further, based on the semantic object layout configuration parameters and the input target description text, the generated image is shown in FIG11 to ensure that different semantic objects appear in different areas cut by semantic objects in the layout diagram.

In this embodiment, by controlling the generated objects in different areas through semantic segmentation, the accuracy of the generated positions of the semantic objects can be effectively improved, and the reasonable layout of the non-semantic objects in the generated image can be achieved, thereby effectively improving the quality of the generated image.

In some embodiments, the sample image also includes a depth map. The computer device can improve the quality of the generated image by a control method based on depth cutting. Specifically, the depth cutting is to introduce the depth map of the training image during the training process so that the black and white channels of the depth map can affect the position of the final image generation.

In the application process of the subject image generation model, the method also includes: obtaining a single-channel grayscale image of the same size as the target image to be generated, and the black and white channels of the single-channel grayscale image are used to locate the foreground and background parts of the target image respectively. As shown in the left figure of Figure 12, in the process of generating an image by the subject image generation model, in addition to the input target description text, a pre-selected alpha image is also input. Among them, the alpha image is a single-channel grayscale image, and the size of the grayscale image is the same as the size of the target generated image. Among them, the part with an alpha value of 1 (corresponding to a pixel value of 255) is the foreground, the part with an alpha value of 0 (corresponding to a pixel value of 0) is the background, and the area between 0-1 (corresponding to a pixel value of 0-255) is the semi-transparent part.

Correspondingly, in the image generation process, denoising is performed on the white noise corresponding to the random latent variables in the latent variable domain under the guidance of the text vector to reconstruct the image latent variables, including: constructing initial latent variables based on the white noise corresponding to the random latent variables in the latent variable domain and the black and white channels of the single-channel grayscale image; and denoising is performed based on the initial latent variables under the guidance of the text vector to reconstruct the image latent variables.

Among them, the role of the black and white channels of the single-channel grayscale image is to affect the position of the subject elements in the target description text in the image as the foreground part or the background part. In a specific application, the initial latent variables based on the alpha image channel and random noise constitute the latent variables, so that the initial latent variables and the input target description text work together to generate an image that conforms to the target theme. Specifically, as shown in the right figure of Figure 12, from the generated image based on the input target text "pineapple standing in front of the podium", the generated target object "pineapple" can be well circled in the foreground part of the depth map, thereby effectively improving the quality of the generated image.

The present application also provides an application scenario, which applies the above-mentioned method for generating an image of a target subject. Specifically, the application of the method for generating an image of a target subject in the application scenario is as follows:

Specifically, this solution is based on the pre-trained model of the Diffusion framework, and fine-tunes the framework based on the data of a specific IP to solve the problem that the Diffusion framework cannot stably generate images in a specific IP style. In several key steps such as training corpus construction, model training and tuning, and text generation and reasoning, technologies that adapt to the style of a specific IP are incorporated, so that the output model can stably and reliably generate high-quality images that match the content style of the IP based on the input text.

Among them, the core scenes and core features of the IP are learned through the model, and external celebrities and ordinary people are used as the subjects of the painting, and embedded in the way of text-generated images, so as to integrate the external characters with the core scenes of the IP and realize the embedding of the IP scenes. Users can also upload an image of their own, and use the hidden variable encoding of the image as the initial condition to complete the IP style transfer of the input image. In addition, corresponding training can be carried out based on the limited materials of the IP, and the core elements of the IP can be learned. According to the combination of features, a large amount of related materials can be generated and released on social media to assist the promotion of the IP.

The model is mainly composed of three parts, namely the control domain, the latent variable domain and the image domain. Among them, the image domain includes an image encoder and an image decoder, which are used to establish a connection between the image domain and the latent variable domain. Among them, the image encoder is used to encode and compress the image vector with the dimension [Channel, Height, Width] in the image domain to obtain a latent variable with the dimension [LatentChannel, LatentHeight, LatentWidth]. Usually, the image is RGB encoded, Channel is 3, Height and Width represent the height and width of the original image; LatentChannel is the number of channels in the latent variable domain, for example, if it is selected as 4, LatentHeight, LatentWidth will be reduced by a fixed ratio (scale_factor, an exponent of 2, such as 4, 8, 16, etc.) based on the original image. LatentHeight = Height/scale_factor, LatentWidth = Width/scale_factor. To ensure that the height and width of the latent variable domain are integers, the input image is generally aligned, for example, by image resizing and padding operations, to ensure that the width and height of the image can be divided by a fixed ratio scale_factor.

The processing of the image decoder is the inverse of the processing of the image encoder, and is responsible for restoring the latent variables in the latent variable domain to the image vector in the image domain. For example, the image decoder can receive a latent variable with an input dimension of [LatentChannel,LatentHeight,LatentWidth] in the latent variable domain, restore the latent variable to an image vector of [Channel,Heigth,Width], and then generate the corresponding image.

In one embodiment, image encoding and image decoding can be implemented using a VAE framework, which includes an image encoder and an image decoder, wherein the image encoder is used to encode the image vector of the input image into a latent variable, specifically to estimate the mean E(z) and variance V(z) of the Gaussian distribution of each dimension in the latent variable space, and then sample the Gaussian distribution based on the mean and variance of each dimension to obtain the latent variable. The image decoder is used to decode the latent variable into an image vector.

The control domain includes a text encoder, which is used to convert natural text into a text vector of fixed dimension. Specifically, the text encoder can adopt a transformer structure, including the following steps: through word segmentation by a tokenizer and word table lookup, the natural text is converted into an ID sequence, and through a multi-layer transformer structure, the ID sequence is encoded into a sequence vector matrix of [Seq_Len, HiddenSize]; where Seq_Len is the length of the sequence and HiddenSize is the dimension of the hidden vector.

The data processing process in the latent variable domain includes diffusion process and denoising process. The diffusion process refers to the process of gradually converting a data sample (the latent variable obtained by VAE encoding) into random noise by adding noise. The specific formula is as follows

X _t ＝α _t X _t-1 +β _t ∈ _t ,∈ _t ～N(0,1)

Among them, Xt _-1 is the hidden variable of the previous step, _αt , _αt are the pre-set noise weights, and _∈t is the randomly sampled Gaussian noise. For the initial hidden variable, a diffusion noise adding process of more than 1000 steps will be performed. The noise weight will be smaller in the early stage and larger in the later stage. Specifically, it can follow a fixed weight distribution.

The denoising process uses a complex Unet model to learn the noise in each step of the denoising process, so that the difference between the estimated X _t-1 ' and the actual X _t-1 is as small as possible. The input of the denoising process is a white noise. Through multi-step noise estimation, the noise of the image is reduced and finally a high-definition image is sampled. The loss function of the Unet model training is

By minimizing the difference between the estimated noise at time t and the actual noise, the difference between the estimated X _t-1 ' and the actual X _t-1 can be minimized. After t steps of denoising, the original image of X at time t = 0 is finally restored. The model structure of the denoising process is a Unet structure, which includes a group of downsamplers, a group of upsamplers, and a middle layer of dimensionality-preserving samplers. The downsampler can be a CNN+transformer structure, the upsampler CNN+transformer structure, and the middle layer of samplers has the same structure as the upsamplers and downsamplers, except that the dimensionality is not scaled.

In the process of fine-tuning the pre-trained model based on the training corpus of a specific IP topic, the training framework mainly trains the text encoder of the control domain and the Unet module of the denoising process. For the other two parts of the overall framework, the image encoder and the image decoder, the pre-trained VAE is used. The image encoder and the image decoder are only used for inference and do not participate in training. The diffusion process is a fixed multi-step process of adding Gaussian noise and does not participate in training.

In the process of model application, the goal of the model application is to randomly generate an image consistent with the description of the prompt based on the input prompt. The computer device inputs the target description text through the control domain of the theme image generation model, and the text encoder based on the control domain converts the target description text into a text vector in the latent variable domain, and generates an image latent variable Z_0 corresponding to the text vector; the random seed randomly generates random Gaussian white noise as a random latent variable in the latent variable domain by using a random number generator, and then based on the random latent variable, a multi-step denoising process is used to reconstruct the image latent variable Z_0 under the guidance of the text vector to obtain Z'_0, and then the image decoder in the image domain is used to obtain the image vector to achieve the generation of the target image. In this embodiment, the theme image generation model can effectively achieve the reconstruction of the image latent variables and achieve fast and high-quality generation of images by generating random latent variables and performing denoising under the guidance of the text vector.

During the model training process, the training corpus constructed is images related to the subject IP. The specific acquisition process includes:

The video file is framed, the frame rate of the video file is 30frame/s, and one video is about 40min-120min. On average, 70k-210k images can be extracted from each file. However, since the image changes very little between consecutive frames of the video file, only image frames with relatively large scene changes need to be extracted. Through this method, 2k-4k images are extracted for a 40min video. Specifically, the image frames can be compared one by one. For two image frames that are compared with each other, the computer device can obtain the pixel granularity difference of each pixel arrangement position by performing difference processing on the specific parameter values of the pixel granularity of the same pixel arrangement position of the two image frames. The computer device accumulates the absolute values of the pixel granularity differences of each pixel arrangement position to obtain the sum of the absolute values to characterize the difference between the two image frames that are compared with each other.

In addition, IP videos may include some opening and ending credits, advertisements, and transition images. In addition, the plot may include some supporting characters, unrecognizable scenes, low-definition images, and incomplete images of characters. These are not suitable as training materials and can be regarded as invalid images. Image recognition tools, including opening and ending credits recognition, advertisement recognition, and image clarity models, can be used to filter out invalid images based on these tools.

In addition, the images extracted from IP video frames are generally widescreen images with a horizontal screen size of 1920x1080, while the input requirements for training images are relatively small. If the image is directly scaled, it will lead to a loss of image clarity and insufficient detail resolution of the main elements. Therefore, it is necessary to locate the subject element and capture the image of the target size with the subject element as the center.

Each captured image is annotated with text. The text annotation uses words to describe the theme elements in the image, including the main object (the main body of the image, generally the foreground), the image background, the shooting angle and distance, and some detailed descriptions (clothing, expression, color, action, facial and body details, etc.). Among them, the main object can be a character in the IP video, which can be described by the name of the character. For example, Zhang Xiaoming, the protagonist in "TV Series A". The image background can be a scene in the IP video, which can be a building, a natural landscape, or a storyline. For example, the ancient pavilions, ancient temples, and famous scenes of holding large flags in "TV Series A". Detailed description-clothing category: unique clothing in IP videos. For example, the clothes of the male and female protagonists in "TV Series A" are described in natural text, light yellow and green collar brocade robe, blue cotton and linen suit, blue and white checkered cotton and linen suit, dark purple silk waistcoat, and white cotton and linen suit. Detailed description-expression category: the expression of the character in the IP video, and the classification of expression includes calm, serious, happy, angry, confused, etc. Based on the above text descriptions, one or more text descriptions are generated for each image frame, forming paired training data with the image frame.

As shown in FIG13 , taking TV Series A as an example, the following training data may be generated. For the left picture in FIG13 , the annotated text is: [Character: Protagonist A, Perspective: Close-up of head, Clothing: Blue cotton and linen suit, Hairstyle: Hair tied up, Action: Sitting in the car looking out, Expression: Calm, Background: Carriage]. For the right picture in FIG13 , the annotated text is: [Character: Protagonist A, Perspective: Full-body long-distance view, Clothing: Blue cotton and linen suit, Hairstyle: Half-loose hair, Action: Standing and looking far away, Expression: Serious, Background: Open space in the square].

Based on the text description of the IP, for each attribute feature, a closed text set can be constructed as a feature attribute word list. Taking the TV series A above as an example, the feature attribute word list can be constructed as follows:

For example, the clothing text set: [light yellow and green collar brocade robe, blue cotton and linen suit, blue and white checkered cotton and linen suit, dark purple silk waistcoat, white cotton and linen suit], the hairstyle text set: [half-length hair, tied hair], the expression text set: [calm, serious, surprised, happy, angry, confused], the perspective text set: [close-up of the head, mid-length shot, long shot of the whole body].

During the application of the model, the text description part of the input model includes content that can be obtained from open text descriptions and closed attribute word lists. Among them, open text descriptions are generally used to describe the main characters. In addition to the characters in the IP field, they can also be famous characters in the outside world, or characters represented by amateur photos uploaded by users. Taking "TV Series A" as an example, the characters in the IP field include protagonist A, protagonist B, protagonist C, etc. Famous people in the outside world can be any celebrities in the outside world, such as famous movie actors, famous singers, etc., and can also be amateur photos uploaded by users, generally non-famous people in the outside world, such as people corresponding to photos containing character portraits taken by users with cameras.

Based on the above text input elements, a free input text can be constructed and used as the input for model reasoning. For example, the image effect generated by "Character: Protagonist A, Perspective: Half-length mid-shot, Hairstyle: Half-down hair, Clothing: Blue cotton and linen suit" is shown in Figure 14.

During the model application process, in order to further improve the quality of image generation and ensure the controllability of text, the solution of this application also added many text vector encoding and fine-tuning modules, and made corresponding adjustments to the denoising process.

The theme image generation model obtained by fine training using the standard training process and model framework has the problem that some features are not well controlled, which is specifically reflected in that the generated image is inconsistent with the text description. In order to solve this technical problem, in this embodiment, by performing vector weighting on key features in the text vector domain, it is ensured that the key features have a higher weight in the image generation process relative to other texts. The specific implementation method is to merge the weight with the text during the text construction process, and perform corresponding parsing and weighting during the text encoding process. In a specific application, the target description text is "Character: Protagonist A, Perspective: (half-length mid-shot: 1.1), Hairstyle: Bunch, Clothing: (blue cotton and linen suit: 1.2)". By means of (theme element description text: weight), the text part to be weighted can be determined, and different weights can be applied to different texts; among them, the text without brackets can be weighted by default as 1.0. Based on the weight data of the key description text, the key description text can be vector weighted during the vector conversion process performed by the text encoder, so as to obtain the text vector corresponding to the target description text, which can effectively improve the effective expression of the key features in the generated image, thereby improving the image quality.

The semantic control ability of different layers of the text encoder and the clarity of the final image will be different. Taking the file encoder as a transformer model as an example, the penultimate layer of the transformer can improve the control ability of the text. The text vectors of the penultimate layer and the last layer of the transformer model are interpolated and fused. The specific fusion method is as follows:

text_emb＝alpha*text_emb(L-1)+(1-alpha)*text_emb(L)

Among them, text_emb(L-1) is the text vector of the second-to-last layer, and its dimension is [Seq_Len, HiddenSize]; text_emb(L) is the text vector of the last layer, and its dimension is the same as text_emb(L-1), so it can be linearly interpolated and fused; alpha is the weight of linear interpolation, and its value is a decimal between [0,1]. The higher the value, the higher the weight of the second-to-last layer. By fusing the features of different layers of the file encoder, the vector with better semantic control ability can be fused to the output layer of the file encoder, so that the text vector encoded by the file encoder has better semantic control ability in the image generation process, which can effectively improve the quality of the generated image.

In the early stage of denoising, the main work is to draw the layout of the painting and the main elements, and in the later stage of denoising, the drawing of local details is completed. The computer device controls the subject image generation model in the early stage of the image generation corresponding to the denoising process, so that only the main description text related to the subject character and layout is effective, and the detail description text is invalidated by masking, ensuring that more attention is paid to the main description text in the early stage of the image. In the later generation stage, the opposite processing is performed, making it easier for the detail description text to control the generation of the image, and the text that controls a large area such as the subject character and layout is covered by a mask, ensuring that more attention is paid to the detail description text in the later stage of the image. The two stages work together to further improve the control of the generated text and the image refinement of the detail text.

Computer equipment can improve the quality of generated images by controlling the generated objects in different regions through semantic cutting. Through semantic cutting, the objects to be depicted in different regions of the image can be pre-set before the image is generated. During the generation process, the goal of generating different objects in different regions can be achieved by affecting the weights of the attention influence mechanism of different text fragments in different regions. The semantic object layout configuration parameters are used to determine the layout of the target image for the semantic object and obtain the semantic object layout map. Among them, different semantic objects have different influence weights in the semantic object layout map. For example, if the input text is: "1 girl with long hair in a white skirt, 1 boy with a red scarf next to her, tree, sky, grass, highly detailed, best quality", the following semantic object layout map obtained by semantic cutting can be generated through the configuration file. Among them, the white area is "1 girl with long hair in a white skirt", the black area is "1 boy with a red scarf", the green area is "tree", the blue area is "sky", and the brown area is "grass". Among them, the influence weight of "a girl in a white skirt and long hair" on the white area is 1.0, the influence weight of "a boy in a red scarf" on the black area is 1.4, the influence weight of "tree" on the green area is 1.2, the influence weight of "sky" on the blue area is 0.2, and the influence weight of "grass" on the brown area is 0.2. Further, based on the semantic object layout configuration parameters and the input target description text, the generated image is shown in Figure 11 to ensure that different semantic objects appear in different areas cut by semantic objects in the layout diagram.

In addition, in the process of generating images by the theme image generation model, in addition to the input target description text, a pre-selected alpha image can also be input. Among them, the alpha image is a single-channel grayscale image, and the size of the grayscale image is the same as the size of the target generated image. Based on the alpha image channel and random noise, the initial latent variable of the latent variable is formed together, so that the initial latent variable and the input target description text work together to generate an image that conforms to the target theme. From the generated image based on the input target text "pineapple standing in front of the podium", the generated target object "pineapple" can be well circled in the foreground part of the depth map, thereby effectively improving the quality of the generated image. This solution is customized for the content style and scene of a specific IP, such as a video IP, so that the model after finetune training can stably and controllably generate images that conform to the IP content style according to user needs. In addition, the model trained and implemented in accordance with this solution can also be used as the basic model of other technical solutions, such as basic models such as image generation or personalized customization.

It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

Based on the same inventive concept, the embodiment of the present application also provides a target subject image generation device for implementing the target subject image generation method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations of the one or more target subject image generation device embodiments provided below can refer to the limitations of the target subject image generation method above, and will not be repeated here.

In one embodiment, as shown in FIG. 15 , a device for generating an image of a target subject is provided, comprising: a model acquisition module 1502, a text set determination module 1504, a description text determination module 1506, and a target image generation module 1508, wherein:

The model acquisition module 1502 is used to obtain a theme image generation model obtained by secondary training a pre-trained model through sample images; the pre-trained model is used to generate images according to text; and the sample image contains at least one theme element that conforms to the target theme.

The text set determining module 1504 is configured to obtain a text set containing the same type of subject element description text based on the subject element description text carried by each of the sample images and used to describe the subject element.

The description text determination module 1506 is used to select subject element description texts from at least a part of the text set, and combine them to obtain a target description text containing the selected subject element description texts.

The target image generation module 1508 is used to perform image generation processing according to the target description text through the subject image generation model to obtain a target image matching the target subject.

In some embodiments, the image generation device of the target theme also includes a sample image acquisition module, which is used to obtain a target video that conforms to the target theme, perform frame extraction processing on the target video, and obtain multiple candidate image frames with different scenes; perform theme element recognition on each of the candidate image frames to determine a preferred image frame containing at least one theme element; based on the theme elements, perform theme element description text annotation on the preferred image frame to obtain a sample image.

In some embodiments, the sample image acquisition module is also used to perform frame processing on the target video to obtain multiple image frames with the same pixel arrangement; for two video frames among the multiple image frames, based on the pixel granularity difference at the same pixel arrangement position, the sum of the absolute values of the pixel granularity differences at each of the pixel arrangement positions is determined; when the ratio between the sum of the absolute values and the total pixel granularity value of the image frame is greater than a scene change threshold, the two image frames are determined to be candidate image frames with scene differences.

In some embodiments, the sample image acquisition module is further used to respectively acquire the attribute information and content information of each of the candidate image frames; discard the candidate image frames whose at least one item of the attribute information and the content information does not meet the subject element recognition condition, and obtain the preferred image frames containing at least one subject element.

In some embodiments, the sample image acquisition module is also used to locate the center point of the object element in each of the preferred image frames to obtain the located position; according to the size conditions of the sample image, the preferred image frame is cropped with the located position as the center to obtain a cropped preferred image frame; and the cropped preferred image frame is annotated with text describing the subject elements to obtain a sample image.

In some embodiments, the text set includes an open text set and a closed text set; the open text set includes object element description texts of object elements in the subject elements and object element description texts of external object elements;

The description text determination module is also used to select object element description text from the open text set; combine the object element description text with the subject element description text selected from at least a part of the closed text set to obtain a target text combination.

In some embodiments, the subject image generation model includes a control domain, a latent variable domain, and an image domain;

The target image generation module is also used to convert the target description text into a text vector based on the target description text input from the control domain; perform denoising processing under the guidance of the text vector based on the white noise corresponding to the random latent variables in the latent variable domain to reconstruct the image latent variables; and decode the image latent variables in the image domain to obtain a target image matching the target subject.

In some embodiments, the target image generation module is also used to obtain weight data of key description text in the target description text input from the control domain; based on the weight data of the key description text, the key description text is vector-weighted during the vector conversion process to obtain a text vector corresponding to the target description text.

In some embodiments, the control domain includes a text encoder that converts text into a vector; the text encoder includes at least two network layers; the target image generation module is also used to perform vector conversion processing on the input target description text in turn according to each network layer in the text encoder to obtain the output features of each network layer; the output features of the target network layer of the text encoder and the output features of the last network layer are feature fused to obtain the text vector corresponding to the target description text; the target network layer is a network layer in the text encoder used to improve text control capabilities.

In some embodiments, the input text of the control domain also includes negative description text; the target image generation module is also used to use the estimated noise of the negative description text as a fusion object in the denoising process, and fuse it with the estimated noise of the target description text to obtain a fused estimated noise; based on the white noise corresponding to the random latent variables in the latent variable domain, denoising is performed according to the fused estimated noise under the guidance of the text vector to reconstruct the image latent variables.

In some embodiments, the denoising process includes a first stage and a second stage that occur sequentially; the target description text includes a main description text and a detail description text; the target image generation module is also used to, in the first stage, mask the detail description text, decode the main image latent variables of the main description text, and obtain a main image matching the target subject; in the second stage, mask the main description text, decode the detail image latent variables of the detail description text, and obtain a detail image; render the detail image to a corresponding area in the main image to obtain a target image matching the target subject.

In some embodiments, the target image generation module is also used to perform semantic segmentation on the target description text, determine a layout diagram of each semantic object obtained by segmentation in the image based on the semantic object layout configuration parameters; determine the influence weight of the attention influence mechanism of each layout area of each semantic object in the layout diagram; and decode the image latent variables obtained based on the influence weights in the image domain to obtain a target image in which each semantic object is laid out according to the layout diagram.

In some embodiments, the sample image also includes a depth map; the target image generation module is also used to obtain a single-channel grayscale image of the same size as the target image to be generated, and the black and white channels of the single-channel grayscale image are used to locate the foreground and background parts of the target image respectively; based on the white noise corresponding to the random latent variables in the latent variable domain and the black and white channels of the single-channel grayscale image, an initial latent variable is constructed; and denoising is performed based on the initial latent variables under the guidance of the text vector to reconstruct the image latent variables.

The above-mentioned target theme image generation device obtains a model that can generate images that meet the target theme according to text by acquiring a theme image generation model obtained by secondary training of a pre-trained model. In the process of determining the text of the input model, the sample images of the training theme image generation model are used to contain at least one theme element that meets the target theme, and the theme element description text for describing the theme element carried by each sample image. A text set containing the same type of theme element description text can be used as a text component of the input model. By selecting theme element description texts from at least a part of the text set, target description texts containing the selected theme element description texts are obtained by combining them. The theme image generation model can perform image generation processing according to the target description text to obtain a high-quality target image that is different from the sample image but can highly match the target theme.

Each module in the image generation device of the above target subject can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG16. The computer device includes a processor, a memory, an input/output interface (Input/Output, referred to as I/O) and a communication interface. The processor, the memory and the input/output interface are connected via a system bus, and the communication interface is connected to the system bus via the input/output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pre-trained models and sample images. The input/output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for generating an image of a target subject is implemented.

In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG17. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input/output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input/output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input/output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for generating an image of a target subject is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

Those skilled in the art will understand that the structures shown in Figures 16 and 17 are merely block diagrams of partial structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims (17)

1. A method for generating an image of a target subject, characterized in that the method comprises: Obtaining a subject image generation model obtained by secondary training a pre-trained model through sample images; the pre-trained model is used to generate images according to text; the sample image contains at least one subject element that conforms to the target subject; Based on the subject element description text for describing the subject element carried by each of the sample images, obtaining a text set containing the subject element description text of the same type; Selecting subject element description texts from at least a portion of the text set, respectively, and combining them to obtain a target description text containing the selected subject element description texts; The subject image generation model is used to perform image generation processing according to the target description text to obtain a target image that matches the target subject. 2. The method according to claim 1, characterized in that the method further comprises: Acquire a target video that matches the target theme, perform frame extraction on the target video, and obtain a plurality of candidate image frames with different scenes; Performing subject element recognition on each of the candidate image frames to determine a preferred image frame containing at least one subject element; Based on the subject elements, the preferred image frame is annotated with subject element description text to obtain a sample image. 3. The method according to claim 2, wherein the step of extracting frames from the target video to obtain a plurality of candidate image frames having different scenes comprises: Performing frame processing on the target video to obtain multiple image frames with the same pixel arrangement; For two video frames in the plurality of image frames, based on the pixel granularity difference at the same pixel arrangement position, determine the sum of the absolute values of the pixel granularity difference at each of the pixel arrangement positions; When the ratio of the sum of the absolute values to the total value of the pixel granularity of the image frames is greater than a scene change threshold, the two image frames are determined to be candidate image frames with scene differences. 4. The method according to claim 2, wherein the step of performing subject element recognition on each candidate image frame to determine a preferred image frame containing at least one subject element comprises: Respectively obtaining attribute information and content information of each of the candidate image frames; The candidate image frames whose at least one item of the attribute information and the content information does not satisfy the subject element recognition condition are discarded to obtain a preferred image frame containing at least one subject element. 5. The method according to claim 4, characterized in that the subject elements include object elements that conform to the target subject; and based on the subject elements, the preferred image frames are annotated with subject element description text to obtain sample images, comprising: For each of the preferred image frames, positioning the center point of the object element in the preferred image frame to obtain a positioning position; According to the size condition of the sample image, the preferred image frame is cropped with the positioning position as the center to obtain a cropped preferred image frame; The cropped preferred image frame is annotated with a textual description of subject elements to obtain a sample image. 6. The method according to claim 1, characterized in that the text set includes an open text set and a closed text set; the open text set includes object element description texts of object elements in the subject elements and object element description texts of external object elements; The step of selecting subject element description texts from at least a portion of the text set and combining them to obtain target description texts containing the selected subject element description texts comprises: Selecting object element description text from the open text set; The object element description text is combined with the subject element description text selected from at least a part of the closed text set to obtain a target text combination. 7. The method according to any one of claims 1 to 6, characterized in that the subject image generation model comprises a control domain, a latent variable domain and an image domain; The subject image generation model is used to perform image generation processing according to the target description text to obtain a target image matching the target subject, including: Based on the target description text input from the control domain, converting the target description text into a text vector; Based on the white noise corresponding to the random latent variable in the latent variable domain, denoising is performed under the guidance of the text vector to reconstruct the image latent variable; The image latent variables are decoded in the image domain to obtain a target image matching the target subject. 8. The method according to claim 7, characterized in that the step of converting the target description text into a text vector based on the target description text input from the control domain comprises: For the target description text input from the control domain, obtaining weight data of key description texts in the target description text; Based on the weight data of the key description text, the key description text is subjected to vector weighting processing during the vector conversion process to obtain a text vector corresponding to the target description text. 9. The method according to claim 7, characterized in that the control domain includes a text encoder that converts text into a vector; the text encoder includes at least two network layers; The step of converting the target description text into a text vector based on the target description text input from the control domain comprises: According to each network layer in the text encoder, the input target description text is sequentially subjected to vector conversion processing to obtain output features of each network layer; The output features of the target network layer of the text encoder and the output features of the last network layer are feature fused to obtain a text vector corresponding to the target description text; the target network layer is the network layer in the text encoder used to improve text control capabilities. 10. The method according to claim 7, characterized in that the input text of the control field also includes negative description text; The white noise corresponding to the random latent variable based on the latent variable domain is subjected to denoising processing under the guidance of the text vector to reconstruct the image latent variable, including: The estimated noise of the negative description text is used as a fusion object in the denoising process, and is fused with the estimated noise of the target description text to obtain a fused estimated noise; Based on the white noise corresponding to the random latent variable in the latent variable domain, denoising processing is performed according to the fused estimated noise under the guidance of the text vector to reconstruct the image latent variable. 11. The method according to claim 7, characterized in that the denoising process comprises a first stage and a second stage occurring in sequence; the target description text comprises a main description text and a detail description text; The decoding of the image latent variable in the image domain to obtain a target image matching the target subject includes: In the first stage, the mask covers the detail description text, and the subject image latent variables of the subject description text are decoded to obtain a subject image matching the target subject; In the second stage, the main description text is covered with a mask, and the detail image latent variables of the detail description text are decoded to obtain a detail image; The detail image is rendered to a corresponding area in the main image to obtain a target image matching the target subject. 12. The method according to claim 7, characterized in that the method further comprises: Performing semantic segmentation on the target description text, and determining a layout diagram of each semantic object obtained by segmentation in the image based on semantic object layout configuration parameters; Determine the influence weight of each of the semantic objects on the attention influence mechanism of each layout area in the layout diagram; The decoding of the image latent variable in the image domain to obtain a target image matching the target subject includes: The image latent variables obtained based on the influence weights are decoded in the image domain to obtain a target image in which each of the semantic objects is laid out according to the layout diagram. 13. The method according to claim 7, wherein the sample image further comprises a depth map; the method further comprises: Obtain a single-channel grayscale image of the same size as the target image to be generated, wherein the black and white channels of the single-channel grayscale image are used to locate the foreground and background of the target image respectively; The white noise corresponding to the random latent variable based on the latent variable domain is subjected to denoising processing under the guidance of the text vector to reconstruct the image latent variable, including: Constructing an initial latent variable based on the white noise corresponding to the random latent variable in the latent variable domain and the black and white channels of the single-channel grayscale image; A denoising process is performed based on the initial latent variables under the guidance of the text vector to reconstruct the image latent variables. 14. A device for generating an image of a target subject, characterized in that the device comprises: A model acquisition module, used to acquire a theme image generation model obtained by retraining a pre-trained model through sample images; the pre-trained model is used to generate images according to text; the sample image contains at least one theme element that conforms to the target theme; A text set determination module, configured to obtain a text set containing the same type of subject element description texts based on the subject element description texts carried by each of the sample images and used to describe the subject element; A description text determination module, used to select subject element description texts from at least a part of the text set, and combine them to obtain a target description text containing the selected subject element description texts; The target image generation module is used to perform image generation processing according to the target description text through the subject image generation model to obtain a target image matching the target subject. 15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the processor implements the steps of the method according to any one of claims 1 to 13 when executing the computer program. 16. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented. 17. A computer program product, comprising a computer program, characterized in that when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.