WO2025256530A1 - 图像生成方法、装置、电子设备及存储介质 - Google Patents

图像生成方法、装置、电子设备及存储介质

Info

Publication number
WO2025256530A1
WO2025256530A1 PCT/CN2025/100222 CN2025100222W WO2025256530A1 WO 2025256530 A1 WO2025256530 A1 WO 2025256530A1 CN 2025100222 W CN2025100222 W CN 2025100222W WO 2025256530 A1 WO2025256530 A1 WO 2025256530A1
Authority
WO
WIPO (PCT)
Prior art keywords
character
model
preset
image
control conditions
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/100222
Other languages
English (en)
French (fr)
Inventor
潘瑞东
孟泉
王雨舟
康宏伟
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Zitiao Network Technology Co Ltd
Original Assignee
Beijing Zitiao Network Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Zitiao Network Technology Co Ltd filed Critical Beijing Zitiao Network Technology Co Ltd
Publication of WO2025256530A1 publication Critical patent/WO2025256530A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces

Definitions

  • This disclosure relates to an image generation method, apparatus, electronic device, and storage medium.
  • This disclosure provides an image generation method, apparatus, electronic device, and storage medium that can ensure the consistency of the same characters in an image.
  • embodiments of this disclosure provide an image generation method, including:
  • prompt words are generated based on the descriptive text, and character tags are determined based on the prompt words;
  • control conditions are determined based on at least one of the prompt words, the character tags, and the preset character models;
  • the character image is generated based on the preset generation model and the control conditions.
  • embodiments of this disclosure also provide an image generation apparatus, comprising:
  • a text receiving module is used to receive content text and determine the description text of the characters in the content text
  • the text understanding module is used to generate prompt words based on the description text using a preset language model, and to determine the role label based on the prompt words;
  • a matching module is used to determine the degree of matching between the character tags and preset character models
  • a control condition determination module is used to determine control conditions based on at least one of the prompt words, the character tags, and the preset character models in response to the matching degree.
  • the image generation module is used to generate a character image of the character according to the control conditions based on a preset generation model.
  • embodiments of this disclosure also provide an electronic device, the electronic device comprising:
  • One or more processors are One or more processors;
  • Storage device for storing one or more programs.
  • the one or more processors When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method as described in any of the embodiments of this disclosure.
  • embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the image generation method as described in any of the embodiments of this disclosure.
  • Figure 1 is a schematic flowchart of an image generation method provided in an embodiment of this disclosure
  • FIG. 2 is a schematic flowchart of an image generation method provided in an embodiment of this disclosure
  • FIG. 3 is a schematic flowchart of an image generation method provided in an embodiment of this disclosure.
  • Figure 4 is a data flow diagram of constructing a character model in an image generation method provided in an embodiment of this disclosure
  • Figure 5 is a data flow diagram of image stylization in an image generation method provided in an embodiment of this disclosure.
  • Figure 6 is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of this disclosure.
  • Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.
  • Figure 1 is a schematic flowchart of an image generation method provided in an embodiment of this disclosure. This embodiment is applicable to situations involving the generation of character images, such as generating character images corresponding to long text passages.
  • the method can be executed by an image generation device, which can be implemented in software and/or hardware and can be configured in an electronic device, such as a computer.
  • the image generation method provided in this embodiment may include:
  • S110 Receive content text and determine the description text of the role in the content text.
  • the content text can include, but is not limited to, the content text of works in the form of novels, essays, poems, and dramas.
  • the content text When the content text is short, it can be received directly.
  • Pre-processing may include segmenting and storyboarding to obtain at least two parts of the content text. Then, each part of the content text can be received sequentially according to its order of appearance within the overall content text.
  • the character description text can include text representing at least one descriptive dimension, such as the character's name, appearance, physique, clothing, actions, expressions, and language.
  • text representing at least one descriptive dimension such as the character's name, appearance, physique, clothing, actions, expressions, and language.
  • word matching can be used to determine if the content text contains character description text; these words can include pre-defined common words representing at least one of the descriptive dimensions, such as common words representing the expression dimension, which may include, but are not limited to, sexual, indifferent, shy, and arrogant.
  • an existing language model can be used to identify whether the content text contains text representing at least one of the descriptive dimensions.
  • the image generation process can be stopped; if the content text contains descriptive text for at least one character, the descriptive text for at least one character can be extracted in the order of the text.
  • the prompt words can be considered as keywords in the descriptive text containing descriptive dimensions.
  • At least one keyword for a descriptive dimension can be pre-set as a character tag; for example, a character tag may include keywords related to dimensions such as appearance and expression from the prompt words.
  • the preset language model can include a language model with natural language understanding capabilities.
  • This preset language model can be constructed based on sample description text, as well as corresponding sample prompt words and sample role labels. Therefore, based on the constructed preset language model, it is possible to perform tasks such as extracting prompt words from the description text and determining role labels from the prompt words.
  • the character model includes a low-rank adaptive model for the face.
  • the low-rank adaptive model (LoRA) can be understood as a plugin for the text-to-image model. By loading the low-rank adaptive model into the text-to-image model, low-rank weight updates can be introduced, enabling fine-tuning of the model's parameters and performance improvement, thus controlling the generation of stable, customized images.
  • the character model includes a LoRA model that controls the stable generation of the face.
  • the correspondence between characters and preset character models can be stored in advance.
  • the correspondence between characters and preset character models can be stored in Jason format; wherein, the character's name can be used as the key, and the character's preset character model can be used as the value.
  • other methods can be used to store this correspondence, such as storing the correspondence in a list, etc., which will not be exhaustively listed here. Accordingly, based on the pre-stored correspondence, it can be determined whether a character appearing in the content text has been associated with a character model.
  • each character model can be built based on predefined tags; that is, each character model has corresponding tags.
  • predefined tags that is, each character model has corresponding tags.
  • the degree of matching between the character's tags and the predefined character model's tags can be determined by judging the tag similarity.
  • existing text similarity methods can be used to determine the tag similarity between the character's tags and the predefined character model's tags.
  • the relationship between tag similarity and matching degree can be predefined.
  • the matching degree can be defined as three levels: first matching degree, second matching degree, and third matching degree.
  • the first matching degree indicates that the tag similarity is greater than or equal to a first threshold;
  • the second matching degree indicates that the tag similarity is greater than or equal to a second threshold but less than the first threshold;
  • the third matching degree indicates that the tag similarity is less than the second threshold.
  • the matching degree can be divided into multiple levels according to specific business needs, and the thresholds used to divide different levels can be set according to actual applications. This is only an example illustrating the different matching degrees at each level.
  • the matching degree between the character tag and the preset character model can be determined according to the tag similarity between the character tag and the preset character model's tag.
  • control conditions In response to the degree of matching, determine control conditions based on at least one of the prompt words, role tags, and preset role models.
  • the matching degree when the matching degree is determined to be high, the prompt words and the preset character model can be directly used as control conditions.
  • the matching degree is determined to be medium, adjustment information can be determined based on the prompt words, and the prompt words, adjustment information, and the preset character model can be used as control conditions.
  • the matching degree is low, the character model can be re-associated based on the character tag, and the prompt words and the re-associated character model can be used as control conditions.
  • the preset generation model may include a diffusion model capable of generating images based on text.
  • the preset generation model can be constructed based on sample control conditions and sample character images. Therefore, by constructing the preset generation model, the diffusion process from random noise images to character images can be controlled according to the control conditions, so that the generated character images match the control conditions.
  • control conditions can be determined according to different matching degrees between the descriptive text and the preset character model, the character image generated according to the control conditions can ensure consistency when the character's appearance does not change significantly, and also ensure the correct presentation when the character's appearance changes significantly.
  • the descriptive dimensions in the descriptive text may not include dimensions such as appearance; that is, the prompt words may not contain character tags.
  • the descriptive text may only contain "Zhang San's” name and language, without further describing Zhang San's appearance. In this case, it can be assumed that Zhang San's appearance has not changed significantly, and there is no need to judge the matching degree between the character tags and the associated character model.
  • the control conditions can be determined directly based on the prompt words and the preset character model to ensure the consistency of the character.
  • pre-storing the correspondence between characters and associated character models it is also possible to pre-storing the correspondence between characters and cue words for certain descriptive dimensions.
  • characters can be pre-stored along with cue words such as pupil color and hair color, serving as supplementary information to enhance the details of the character model.
  • control conditions can be determined simultaneously based on the supplementary information to further ensure the consistency of the characters.
  • character images can be randomly generated based solely on prompts using a preset generation model. This avoids designing character models for arbitrary characters and minimizes resource waste.
  • the technical solution of this disclosure embodiment receives content text and determines the description text of the character in the content text; generates prompt words based on the description text using a preset language model, and determines the character tag based on the prompt words; determines the matching degree between the character tag and the preset character model; in response to the matching degree, determines control conditions based on at least one of the prompt words, character tags, and the preset character model; and generates the character image of the character based on the control conditions using a preset generation model.
  • the current descriptive text of a character can be understood to obtain prompts for generating character images.
  • character tags containing facial descriptions can be identified from these prompts, and control conditions for generating character images can be determined based on the degree of matching between the character tags and the character's preset character model.
  • the matching degree is high
  • the control conditions are determined using the prompts and the preset character model to ensure consistency for the same character in the generated images.
  • a new character model can be determined based on the character tags, and the control conditions are determined using the prompts and the new character model to ensure correct representation of the character even when its appearance changes.
  • control conditions can be directly determined based on the prompt words and the preset character model.
  • an auxiliary model can be determined based on the prompt words, and control conditions are determined based on the prompt words, the preset character model, and the auxiliary model.
  • the prompt words and auxiliary model can be adjusted to adapt to certain changes in the character while ensuring character consistency.
  • the character model can be re-determined based on the character tag, and control conditions can be determined based on the prompt words and the re-determined character model. Therefore, the correctness of the character's presentation in the image can be ensured when the character's image undergoes significant changes.
  • Figure 2 is a schematic flowchart of an image generation method provided in an embodiment of this disclosure.
  • the image generation method provided in this embodiment may include:
  • S210 Receive content text and determine the description text of the role in the content text.
  • the character model includes a low-rank adaptive model of the face.
  • the first matching degree can characterize the basic match between the role label and the associated model.
  • the current prompt word and the preset role model can be directly used as control conditions.
  • the pre-stored auxiliary information can also be updated according to the current prompt word.
  • the second matching degree can characterize the difference between the character label and the preset model. For example, when a character's age changes within a certain range, the character's appearance undergoes subtle and complex changes due to age (e.g., fine lines appear on the face); or, for example, the character wears specific clothing on a special occasion, such as a wedding dress.
  • auxiliary labels can be extracted from the prompt words; wherein, at least one keyword describing a dimension can be preset as auxiliary labels, for example, auxiliary labels can include keywords related to dimensions such as clothing and age in the prompt words.
  • the auxiliary model includes a clothing and/or age adjustment model, and the adjustment model may also include a low-rank adaptive model.
  • a preset auxiliary model library may contain multiple pre-built auxiliary models, and each auxiliary model can be built based on set tags during the construction process.
  • candidate auxiliary models matching each auxiliary tag can be selected from the preset auxiliary model library.
  • auxiliary models already associated with other roles can be removed from the candidate auxiliary models, and an auxiliary model matching the auxiliary tag can be randomly selected from the removed candidate auxiliary models.
  • the auxiliary model can also be included in the auxiliary information, which can store the correspondence between roles and auxiliary models to associate roles and auxiliary models.
  • the preset auxiliary model can be updated based on the currently matched auxiliary model.
  • prompt words preset role models, and currently matched auxiliary models can be used as control conditions.
  • the third matching degree can characterize that the character tag is fundamentally different from the preset character model. For example, the character's age has changed significantly (e.g., from a teenager to an old person), resulting in a large change in the character's appearance. In this case, a new character model that matches the character tag can be determined from the preset character library.
  • the preset character library can be generated based on pre-built character models, and it can store tags for each character model. Candidate character models matching the tags can be selected from the preset character library. Then, character models already associated with other characters can be removed from the candidate models, and a new character model can be randomly selected from the removed candidate models. After determining the new character model that matches the character, the preset character model can be updated based on the new character model, thus storing the correspondence between the character and the new character model.
  • prompt words and new character models can be used as control conditions.
  • the currently generated character image can be determined, and its similarity can be determined by comparing it with a preset number of historical character images corresponding to that character.
  • the currently generated character image can be determined, and its similarity can be determined by comparing it with the previous historical character image corresponding to that character.
  • the method for determining character similarity may include, for example, extracting a first facial feature from the currently generated character image and a second facial feature from historical character images; determining the feature similarity between the first and second facial features based on existing feature similarity determination methods; and determining the character similarity based on the feature similarity.
  • it may include inputting the currently generated character image, historical character images corresponding to that character, and prompts for character similarity determination (e.g., "Please output the similarity between characters in the image") into a language model with multimodal understanding capabilities; and outputting the character similarity through the multimodal language model.
  • the consistency of the characters in the images is high; when the character images of the same character are generated based on different character models, the characters in the images still need to have similarities in details such as face shape and eye color.
  • a preset threshold can be set according to business needs. If the character similarity is less than the preset threshold, the similarity requirement for the same character is considered not met. In this case, the diffusion process of new random noise images to the character image can be controlled according to control conditions to regenerate the character image. The repeated generation process of the character image can be stopped until the character similarity is greater than or equal to the preset threshold, thus obtaining a character image that meets the similarity requirement.
  • control conditions can be directly determined based on the prompt words and a preset character model.
  • an auxiliary model can be determined based on the prompt words, the preset character model, and the auxiliary model to determine the control conditions.
  • adjustments to the prompt words and the setting of the auxiliary model can adapt to changes in the character to a certain extent, while ensuring character consistency.
  • the character model can be re-determined based on the character tag, and control conditions can be determined based on the prompt words and the re-determined character model. Therefore, the correctness of the character's presentation in the image can be ensured even when the character's image undergoes significant changes.
  • the image generation method provided in this embodiment belongs to the same concept as the image generation method provided in the above embodiments.
  • Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.
  • the image generation method provided in this embodiment describes in detail the determination of control conditions when a preset character model is not determined. It can achieve the initial association of a character model when a preset character model is not determined, and can determine the control conditions based on prompt words and the character model.
  • FIG 3 is a flowchart illustrating an image generation method provided in an embodiment of this disclosure. As shown in Figure 3, the image generation method provided in this embodiment may include:
  • S301 Receive content text and determine the description text of the role in the content text.
  • the current character has been assigned a preset character model, i.e. whether it is associated with a preset character model, based on the pre-stored correspondence between characters and preset character models.
  • a character model matching the character tag can be determined from a preset character library, and the character and the character model can be associated.
  • the process of determining the matching character model and associating the character with the character model can refer to the process of determining the matching new character model and associating the character with the new character model described above, and will not be repeated here.
  • auxiliary tags can be determined based on the prompts in the clothing description dimension; and auxiliary models that match the auxiliary tags can be determined from the preset auxiliary library, and the character and auxiliary models can be associated.
  • auxiliary model and other auxiliary information related to the role After determining the role model, auxiliary model and other auxiliary information related to the role, the correspondence between the role and the role model, auxiliary model and other auxiliary information can be stored to associate the role with the role model, auxiliary model and other auxiliary information.
  • control conditions In response to the degree of matching, determine control conditions based on at least one of the prompt words, role tags, and preset role models.
  • FIG. 4 is a data flow diagram of constructing a role model in an image generation method provided by an embodiment of this disclosure. Referring to Figure 4, in some optional implementations, when the determination of the role model fails, the following may also be included:
  • an image set is generated based on the character label; facial features of each image in the image set are extracted, and the images are clustered based on each facial feature to obtain the most similar cluster; based on the images in the most similar cluster, a character model matching the character label is constructed; the constructed character model is loaded into the preset generation model to generate a new image set; in response to the most similar cluster satisfying the preset conditions, the constructed character model is used as the preset character model.
  • the process of constructing a character model can be achieved by using character tags as control conditions.
  • a preset generation model can be used to control the diffusion process from different noisy images to character images, generating an image set containing a sufficient number of images.
  • facial features can be extracted from each image in the image set using existing facial feature extraction methods.
  • images can be clustered based on their facial features using existing clustering algorithms (such as K-means), and the most cohesive cluster can be selected as the most similar cluster. Cohesion can be determined based on the average distance between the facial features and the centroid of each image in the cluster, and cohesion is inversely correlated with the average distance; the centroid can be determined based on the facial features of each image.
  • character models can be constructed by extracting identities from the images within the most similar cluster.
  • the character model constructed in the current round can also be used as a control condition and loaded into the preset generation model to generate new image sets based on different morning images.
  • the steps of extracting facial features from the images in the image set, clustering images based on facial features, constructing a character model based on the most similar cluster, and loading the newly constructed character model into the preset generation model to generate a new image set can be repeated until the most similar cluster meets the preset conditions (e.g., the cohesion meets the preset value).
  • the character model can be considered to have converged, and the constructed character model is obtained.
  • the character model construction process can achieve unsupervised or semi-supervised character model construction.
  • a pre-built character library can also be constructed based on this process.
  • new character models can be generated through weighted combinations of different character models.
  • multiple corresponding character models can also be trained to enrich the pre-built character library.
  • character images matching the character tags can be obtained through various channels, and character models can be directly constructed based on these images. It is understood that the character images involved in this technical solution (including but not limited to the character images themselves, their acquisition, or use) should comply with relevant laws, regulations, and provisions.
  • the above operations can be performed to construct a new character model, and the constructed new character model can also be stored in the preset character library.
  • the determination of an auxiliary model fails, the above operations can be performed to construct an auxiliary model, and the constructed auxiliary model can also be stored in the preset auxiliary model library.
  • a preset auxiliary model library can be pre-constructed based on the above process.
  • the technical solution of this disclosure provides a detailed description of determining control conditions when a preset character model is not determined. This enables the initial association of a character model even when no preset model is available, and the control conditions can be determined based on prompts and the character model. Furthermore, the image generation method provided in this disclosure belongs to the same concept as the image generation methods provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.
  • the image generation method provided in this embodiment details the stylization and video generation of the generated character image.
  • stylizing the character image according to a stylization model with a small number of parameters By stylizing the character image according to a stylization model with a small number of parameters, low-cost, high-quality style transfer can be achieved while maintaining character consistency.
  • Generating a video from the character image further enhances the user's visual experience.
  • the image generation method after generating the character image, it may further include: extracting features from the character image in at least one preset dimension to obtain image features; and generating a stylized image based on the image features using a preset stylization model.
  • the stylization model can include a diffusion model with image-generating capabilities. This stylization model can be constructed based on sample character images and sample stylized images. Compared to the preset generation model, the stylization module can be considered a small model with fewer parameters but stronger stylization.
  • Figure 5 is a data flow diagram of image stylization in an image generation method provided in this embodiment.
  • feature extraction of a character image in at least one preset dimension can include extracting features of the character image in at least one preset dimension, such as expression, posture, image content, and depth of field, to obtain image features corresponding to the preset dimension.
  • the prompt words corresponding to the character image and the image features can be input into at least one stylization model, such as comic style or 3D style, to perform corresponding style transfer according to the stylization model and obtain a stylized image.
  • a stylization model is used for style transfer based on the character image to meet the requirements of diverse art styles.
  • a preset generation model with many parameters and high performance can be used in the character image generation stage to ensure the richness of character image details and the accuracy of the image.
  • a stylization model with fewer parameters but strong stylization can be used to save computational resources.
  • the image generation method after generating the character images, it may also include: generating a target video sequentially based on the character images corresponding to the content text.
  • the target video after generating character images for each role based on the content text, the target video can be generated sequentially based on the corresponding character images according to the order of the content text.
  • the character images can be switched at preset intervals (e.g., 3 seconds) to generate the target video.
  • existing image-based video generation methods can be used to generate videos based on the character images.
  • the technical solutions of this disclosure provide a detailed description of the stylization and video generation of character images.
  • stylizing the character images according to a stylization model with a small number of parameters By stylizing the character images according to a stylization model with a small number of parameters, low-cost, high-quality style transfer can be achieved while maintaining character consistency.
  • Generating videos from the character images further enhances the user's visual experience.
  • the image generation method provided in this disclosure belongs to the same disclosed concept as the image generation method provided in the above embodiments.
  • Technical details not described in detail in this embodiment can be found in the above embodiments, and the same technical features have the same beneficial effects in this embodiment and the above embodiments.
  • Figure 6 is a schematic diagram of an image generation apparatus provided in an embodiment of this disclosure.
  • the image generation apparatus provided in this embodiment is suitable for generating character images, such as generating character images corresponding to long texts.
  • the image generation apparatus provided in this embodiment may include:
  • the text receiving module 610 is used to receive content text and determine the description text of the characters in the content text;
  • the text understanding module 620 is used to generate prompt words based on the descriptive text using a preset language model, and to determine the role label based on the prompt words;
  • Matching module 630 is used to determine the degree of matching between character tags and preset character models
  • the control condition determination module 640 is used to determine control conditions based on at least one of prompt words, role tags, and preset role models in response to the degree of matching.
  • Image generation module 650 is used to generate character images of characters according to control conditions based on preset generation models.
  • control condition determination module can be used for:
  • control conditions are determined based on the prompt words and the preset role model.
  • control condition determination module can be used for:
  • auxiliary tags are determined based on the prompt words
  • the system identifies auxiliary models that match the auxiliary tags from a pre-defined auxiliary library and associates the character with the auxiliary model; the auxiliary models include clothing and/or age adjustment models.
  • the control conditions are determined based on the prompts, preset role models, and auxiliary models.
  • control condition determination module can be used for:
  • a new character model that matches the character tag is determined from the preset character library, and the character is associated with the new character model;
  • the character model includes a low-rank adaptive model of the face.
  • the image generating apparatus may further include:
  • the similarity determination module is used to determine the similarity between the generated character image and the corresponding historical character images after the character image is generated.
  • the image generation module can also be used to regenerate character images based on control conditions when the character similarity is less than a preset threshold.
  • the matching module can also be used to: determine the character model that matches the character tag from the preset character library, and use the matched character model as the preset character model.
  • the image generating apparatus may further include:
  • the model building module is used to generate an image set based on the character tags by using a preset generation model when the character model fails to be determined.
  • Facial features are extracted from each image in the image set, and the images are clustered based on these facial features to obtain the most similar clusters;
  • the constructed role model is used as the preset role model.
  • the image generating apparatus may further include:
  • the stylization module is used to extract features from the character image in at least one preset dimension after the character image is generated, so as to obtain image features.
  • Stylized images are generated based on image features by using a pre-defined stylization model.
  • the image generating apparatus may further include:
  • the video generation module is used to generate target videos sequentially based on the character images corresponding to the content text after generating character images.
  • the image generation apparatus provided in this disclosure can execute the image generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.
  • FIG7 a schematic diagram of the structure of an electronic device (e.g., the terminal device or server in FIG7) 700 suitable for implementing embodiments of the present disclosure is shown.
  • the terminal device in embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers.
  • the electronic device shown in FIG7 is merely an example and should not impose any limitation on the functionality and scope of use of embodiments of the present disclosure.
  • the electronic device 700 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 into a random access memory (RAM) 703.
  • the RAM 703 also stores various programs and data required for the operation of the electronic device 700.
  • the processing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704.
  • An input/output (I/O) interface 705 is also connected to the bus 704.
  • I/O interface 705 input devices 706 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 707 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 708 including, for example, magnetic tapes, hard disks, etc.; and communication devices 709.
  • Communication device 709 allows electronic device 700 to communicate wirelessly or wiredly with other devices to exchange data.
  • Figure 7 shows electronic device 700 with various devices, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.
  • embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts.
  • the computer program can be downloaded and installed from a network via communication device 709, or installed from storage device 708, or installed from ROM 702.
  • processing device 701 it performs the functions defined above in the image generation method of embodiments of this disclosure.
  • the electronic device provided in this embodiment and the image generation method provided in the above embodiments belong to the same disclosed concept.
  • Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.
  • This disclosure provides a storage medium for computer-executable instructions, which, when executed by a computer processor, can be used to perform the image generation method provided in the above embodiments.
  • the storage medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof.
  • a computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory (FLASH), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.
  • a computer-readable storage medium can be any tangible medium that contains or stores executable instructions that can be used by or in conjunction with an instruction execution system, apparatus, or device.
  • a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable executable instructions. Such transmitted data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.
  • the computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit executable instructions for use by or in connection with an instruction execution system, apparatus, or device. Executable instructions contained on the storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.
  • clients and servers may communicate using any currently known or future-developed network protocol, such as HTTP (Hypertext Transfer Protocol), and may interconnect with digital data communication (e.g., communication networks) of any form or medium.
  • network protocols such as HTTP (Hypertext Transfer Protocol)
  • communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad-hoc end-to-end networks), as well as any currently known or future-developed networks.
  • LANs local area networks
  • WANs wide area networks
  • the Internet e.g., the Internet of Things
  • end-to-end networks e.g., ad-hoc end-to-end networks
  • the aforementioned storage medium may be included in the aforementioned electronic device; or it may exist independently and not be assembled into the electronic device.
  • the aforementioned storage medium carries one or more executable instructions, which, when executed by the electronic device, cause the electronic device to:
  • Executable instructions for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof, including but not limited to object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as the "C" language or similar programming languages.
  • the executable instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.
  • the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
  • LAN local area network
  • WAN wide area network
  • This disclosure also provides a computer program product, including a computer program that, when executed by a processor, can implement the image generation method provided in any embodiment of this disclosure.
  • computer program code for performing the operations of this disclosure can be written in one or more programming languages or a combination thereof.
  • Programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as C or similar languages.
  • the program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server.
  • the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
  • LAN local area network
  • WAN wide area network
  • each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function.
  • the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.
  • the units described in the embodiments of this disclosure can be implemented in software or hardware.
  • the names of the units and modules do not, in certain circumstances, constitute a limitation on the unit or module itself.
  • exemplary types of hardware logic components include: Field Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • ASSP Application Specific Standard Parts
  • SOC System on Chip
  • CPLD Complex Programmable Logic Device
  • machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
  • RAM random access memory
  • ROM read-only memory
  • EPROM or flash memory erasable programmable read-only memory
  • CD-ROM compact disk read-only memory
  • magnetic storage devices or any suitable combination of the foregoing.
  • prompt words are generated based on the descriptive text, and character tags are determined based on the prompt words;
  • control conditions are determined based on at least one of the prompt words, the character tags, and the preset character models;
  • the character image is generated based on the preset generation model and the control conditions.
  • an image generation method further comprising:
  • control conditions are determined in response to the matching degree based on at least one of the role tags, the prompt words, and the preset role model, including:
  • control conditions are determined based on the prompt words and the preset role model.
  • an image generation method further comprising:
  • control conditions are determined in response to the matching degree based on at least one of the role tags, the prompt words, and the preset role model, including:
  • auxiliary tags are determined based on the prompt words
  • Auxiliary models matching the auxiliary tags are determined from a preset auxiliary library, and the character is associated with the auxiliary model; wherein, the auxiliary model includes clothing and/or age adjustment models;
  • the control conditions are determined based on the prompt words, the preset role model, and the auxiliary model.
  • an image generation method further comprising:
  • control conditions are determined in response to the matching degree based on at least one of the role tags, the prompt words, and the preset role model, including:
  • a new character model that matches the character tag is determined from the preset character library, and the character is associated with the new character model;
  • an image generation method further comprising:
  • the character model includes a low-rank adaptive model of the face.
  • an image generation method further comprising:
  • the character image is regenerated according to the control conditions.
  • an image generation method further comprising:
  • the steps for determining the preset role model include:
  • an image generation method further comprising:
  • An image set is generated based on the character tags using the preset generation model
  • Facial features are extracted from each image in the image set, and image clustering is performed based on each facial feature to obtain the most similar cluster;
  • the constructed character model is loaded into the preset generation model to generate a new image set
  • the constructed role model is used as the preset role model.
  • an image generation method further comprising:
  • the character image is subjected to feature extraction in at least one preset dimension to obtain image features
  • a stylized image is generated based on the image features using a preset stylization model.
  • an image generation method further comprising:
  • the target video is generated sequentially based on the character images corresponding to the content text.
  • an image generation apparatus comprising:
  • a text receiving module is used to receive content text and determine the description text of the characters in the content text
  • the text understanding module is used to generate prompt words based on the description text using a preset language model, and to determine the role label based on the prompt words;
  • a matching module is used to determine the degree of matching between the character tags and preset character models
  • a control condition determination module is used to determine control conditions based on at least one of the prompt words, the character tags, and the preset character models in response to the matching degree.
  • the image generation module is used to generate a character image of the character according to the control conditions based on a preset generation model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Medical Informatics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Databases & Information Systems (AREA)
  • Multimedia (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本公开实施例提供了一种图像生成方法、装置、电子设备及存储介质，该图像生成方法包括：接收内容文本，确定内容文本中角色的描述文本；通过预设语言模型，根据描述文本生成提示词，并根据提示词确定角色标签；确定角色标签与预设的角色模型的匹配程度；响应于匹配程度，根据提示词、角色标签和预设的角色模型中的至少一项，确定控制条件；通过预设生成模型，根据控制条件生成角色的角色图像。该图像生成方法能够保证图像中相同角色的一致性。

Description

图像生成方法、装置、电子设备及存储介质
本申请要求于2024年6月11日递交的中国专利申请第202410745318.6号的优先权，在此全文引用上述中国专利申请公开的内容以作为本申请的一部分。
技术领域
本公开实施例涉及一种图像生成方法、装置、电子设备及存储介质。
背景技术
目前，可通过语言模型进行文本理解，通过文生图技术基于文本理解，生成与文本对应的图像。在文本篇幅大于模型可处理篇幅的情况下，需将文本分批次来进行图像生成，这将无法保证分批图像中相同角色的一致性。
发明内容
本公开实施例提供了一种图像生成方法、装置、电子设备及存储介质，能够保证图像中相同角色的一致性。
第一方面，本公开实施例提供了一种图像生成方法，包括：
接收内容文本，确定所述内容文本中角色的描述文本；
通过预设语言模型，根据所述描述文本生成提示词，并根据所述提示词确定角色标签；
确定所述角色标签与预设的角色模型的匹配程度；
响应于所述匹配程度，根据所述提示词、所述角色标签和所述预设的角色模型中的至少一项，确定控制条件；
通过预设生成模型，根据所述控制条件生成所述角色的角色图像。
第二方面，本公开实施例还提供了一种图像生成装置，包括：
文本接收模块，用于接收内容文本，确定所述内容文本中角色的描述文本；
文本理解模块，用于通过预设语言模型，根据所述描述文本生成提示词，并根据所述提示词确定角色标签；
匹配模块，用于确定所述角色标签与预设的角色模型的匹配程度；
控制条件确定模块，用于响应于所述匹配程度，根据所述提示词、所述角色标签和所述预设的角色模型中的至少一项，确定控制条件；
图像生成模块，用于通过预设生成模型，根据所述控制条件生成所述角色的角色图像。
第三方面，本公开实施例还提供了一种电子设备，所述电子设备包括：
一个或多个处理器；
存储装置，用于存储一个或多个程序，
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如本公开实施例任一所述的图像生成方法。
第四方面，本公开实施例还提供了一种包含计算机可执行指令的存储介质，所述计算机可执行指令在由计算机处理器执行时用于执行如本公开实施例任一所述的图像生成方法。
附图说明
结合附图并参考以下具体实施方式，本公开各实施例的上述和其他特征、优点及方面将变得更加明显。贯穿附图中，相同或相似的附图标记表示相同或相似的元素。应当理解附图是示意性的，原件和元素不一定按照比例绘制。
图1为本公开实施例所提供的一种图像生成方法的流程示意图；
图2为本公开实施例所提供的一种图像生成方法的流程示意图；
图3为本公开实施例所提供的一种图像生成方法的流程示意图；
图4为本公开实施例所提供的一种图像生成方法中构建角色模型的数据流框图；
图5为本公开实施例所提供的一种图像生成方法中图像风格化的数据流框图；
图6为本公开实施例所提供的一种图像生成装置的结构示意图；以及
图7为本公开实施例所提供的一种电子设备的结构示意图。
具体实施方式
下面将参照附图更详细地描述本公开的实施例。虽然附图中显示了本公开的某些实施例，然而应当理解的是，本公开可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加透彻和完整地理解本公开。应当理解的是，本公开的附图及实施例仅用于示例性作用，并非用于限制本公开的保护范围。
应当理解，本公开的方法实施方式中记载的各个步骤可以按照不同的顺序执行，和/或并行执行。此外，方法实施方式可以包括附加的步骤和/或省略执行示出的步骤。本公开的范围在此方面不受限制。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”；术语“一些实施例”表示“至少一些实施例”。其他术语的相关定义将在下文描述中给出。
需要注意，本公开中提及的“第一”、“第二”等概念仅用于对不同的装置、模块或单元进行区分，并非用于限定这些装置、模块或单元所执行的功能的顺序或者相互依存关系。
需要注意，本公开中提及的“一个”、“多个”的修饰是示意性而非限制性的，本领域技术人员应当理解，除非在上下文另有明确指出，否则应该理解为“一个或多个”。
本公开实施方式中的多个装置之间所交互的消息或者信息的名称仅用于说明性的目的，而并不是用于对这些消息或信息的范围进行限制。
可以理解的是，本技术方案所涉及的数据(包括但不限于数据本身、数据的获取或使用)应当遵循相应法律法规及相关规定的要求。
图1为本公开实施例所提供的一种图像生成方法的流程示意图。本公开实施例适用于生成角色图像的情形，例如在较长文本篇幅情况下生成文本对应的角色图像的情形。该方法可以由图像生成装置来执行，该装置可以通过软件和/或硬件的形式实现，该装置可配置于电子设备中，例如配置于计算机中。
如图1所示，本实施例提供的图像生成方法，可以包括：
S110、接收内容文本，确定内容文本中角色的描述文本。
本公开实施例中，内容文本可以包括但不限于小说、散文、诗歌和戏剧等形式作品的内容文本。在内容文本篇幅较短的情况下，可直接接收内容文本。在内容文本篇幅较长的情况下，例如大于语言模型可处理篇幅的情况下，可对内容文本进行预先处理。其中，对内容文本进行预先处理，例如可对内容文本进行切分和分镜设计等处理，以得到至少两部分内容文本。之后，可按照各部分内容文本在整体内容文本中的行文顺序，依序接收各部分内容文本。
其中，角色的描述文本，可以包括角色的名称、容貌、体型、穿着、动作、神态和语言等至少一个描述维度的文本。在接收到内容文本后，可判断其中是否包含上述至少一个描述维度的文本。例如，可根据词语匹配的方式，确定是否包含角色的描述文本；其中，词语可包括预先设置的上述至少一个描述维度的常用词，例如神态维度的常用词，可以包括但不限于威武、淡然、羞怯和嚣张等。又如，可根据已有的语言模型，来识别内容文本中是否包含上述至少一个描述维度的文本等。
在内容文本不包含角色的描述文本的情况下，可停止图像生成流程；在内容文本包含至少一个角色的描述文本的情况下，可依照行文顺序分别提取至少一个角色的描述文本。
S120、通过预设语言模型，根据描述文本生成提示词，并根据提示词确定角色标签。
本公开实施例中，提示词可认为是描述文本包含的描述维度的文本的关键词。其中，可预先设置至少一个描述维度的关键词作为角色标签，例如角色标签可以包括提示词中与容貌和神态等维度相关的关键词。
其中，预设语言模型可以包括具备自然语言理解能力的语言模型。其中，预设语言模型可根据样本描述文本，以及与样本描述文本对应的样本提示词和样本角色标签进行构建。从而，基于构建完成的预设语言模型，能够执行根据描述文本提取提示词，并从提示词中确定角色标签的任务。
S130、确定角色标签与预设的角色模型的匹配程度。
其中，角色模型包括面部的低秩自适应模型。低秩自适应(Low-Rank Adaptation of Large Language Models，LoRA)模型，可以理解为文生图模型的插件。通过将低秩自适应模型加载至文生图模型，可在文生图模型中引入低秩权重更新，以实现文生图模型的参数微调和性能提升，控制文生图模型生成稳定的定制图像。本实施例中，角色模型包括控制面部稳定生成的LoRA模型。
本公开实施例中，可预先存储角色与预设的角色模型的对应关系。例如，可采用Jason格式存储角色与预设的角色模型的对应关系；其中，可以将角色的名称作为键(key)，将角色预设的角色模型作为值(value)。此外，还可以采用其他方式存储该对应关系，例如列表存储对应关系等，在此不做穷举。相应的，可根据预先存储的对应关系，判断内容文本中出现的角色是否已经与角色模型进行关联。
其中，各角色模型在构建过程中，可基于设置的标签进行构建，即各角色模型对应设置有标签。在角色已关联角色模型的情况下，可通过判断角色标签与预设的角色模型的标签的标签相似度，来确定角色标签与预设的角色模型的匹配程度。其中，可通过已有的文本相似度确定方式，确定角色标签与预设的角色模型的标签相似度。
其中，可以预先定义标签相似度与匹配程度的对应关系。示例性的，匹配程度可定义为第一匹配程度、第二匹配程度和第三匹配程度三个等级；其中，第一匹配程度可表征标签相似度大于等于第一阈值；第二匹配程度可表征标签相似度大于等于第二阈值、小于第一阈值；第三匹配程度可表征标签相似度小于第二阈值。其中，可根据具体业务需求，对匹配程度进行多种等级的划分，且用于划分不同等级的阈值可根据实际应用进行设置，此处仅示例性阐述各等级的匹配程度不同。
相应的，可根据预先定义的标签相似度与匹配程度的对应关系，根据角色标签与预设的角色模型的标签的标签相似度，来确定角色标签与预设的角色模型的匹配程度。
S140、响应于匹配程度，根据提示词、角色标签和预设的角色模型中的至少一项，确定控制条件。
本公开实施例中，在确定匹配程度较高的情况下，可直接将提示词和预设的角色模型，作为控制条件。在确定匹配程度中等的情况下，可根据提示词确定调整信息，并将提示词、调整信息和预设的角色模型，作为控制条件。在匹配程度较低的情况下，可根据角色标签重新关联角色模型，并可将提示词和重新关联的角色模型，作为控制条件。
S150、通过预设生成模型，根据控制条件生成角色的角色图像。
本公开实施例中，预设生成模型可以包括具备基于文本生成图像能力的扩散模型。其中，预设生成模型可根据样本控制条件和样本角色图像进行构建。从而，通过构建完成的预设生成模型，能够根据控制条件，控制由随机噪声图像到角色图像的扩散过程，以使生成的角色图像与控制条件相匹配。
由于在描述文本和预设的角色模型不同匹配程度下，可对应确定控制条件，从而通过根据控制条件生成的角色图像，即能保证角色未发生较大形象变化情况下的一致性，还能保证角色发生较大形象变化情况下的呈现正确性。
在一些实际应用中，描述文本中的描述维度，存在不包括容貌等维度的情况，即此时的提示词中可能不包含角色标签。示例性的，当角色“张三”第二次出现时，描述文本可能仅包含“张三”的名称和语言，而并未对张三的容貌再次描述。此时，可默认张三在形象上并无发生较大改变，可无需判断角色标签与已关联角色模型的匹配程度，直接根据提示词和预设的角色模型，确定控制条件，以保证角色的前后一致性。
此外，除了可预先存储角色与已关联角色模型的对应关系之外，还可以预先存储角色与部分描述维度的提示词的对应关系。例如，可预先存储角色与角色的瞳孔颜色、头发颜色等提示词，作为对角色模型的细节进行补充的辅助信息。相应的，在生成已关联角色模型的角色的图像时，还可以同时根据辅助信息确定控制条件，以进一步保证角色的前后一致性。
对于整个内容文本中无关紧要的次要角色，例如仅出现一次的角色、或未对容貌、穿着等维度描述的角色等，可通过预设生成模型仅根据提示词随机生成角色图像。从而，可避免针对任意角色进行角色模型等设计，可在一定程度上避免资源浪费。
本公开实施例的技术方案，接收内容文本，确定内容文本中角色的描述文本；通过预设语言模型，根据描述文本生成提示词，并根据提示词确定角色标签；确定角色标签与预设的角色模型的匹配程度；响应于匹配程度，根据提示词、角色标签和预设的角色模型中的至少一项，确定控制条件；通过预设生成模型，根据控制条件生成角色的角色图像。
通过预设语言模型，可对角色当前的描述文本进行理解，得到用于生成角色图像的提示词。进而，可从提示词中确定出包含面部描述的角色标签，并可根据角色标签与角色预设的角色模型的匹配程度，来对应确定用于生成角色图像的控制条件。从而，可以实现在匹配程度较高时，利用提示词和预设的角色模型确定控制条件，以使基于控制条件生成的角色图像，可保证相同角色的前后一致性。此外，也可以在匹配程度较低时，根据角色标签来确定新的角色模型，利用提示词和新的角色模型确定控制条件，可保证角色形象发生变化时，角色的正确呈现。
本公开实施例与上述实施例中所提供的图像生成方法中各个可选方案可以结合。本实施例所提供的图像生成方法，对不同匹配程度的情况下控制条件的确定进行了详细描述。在第一匹配程度的情况下，可直接根据提示词和预设的角色模型来确定控制条件；在第二匹配程度下，可根据提示词确定辅助模型，根据提示词、预设的角色模型和辅助模型来确定控制条件。在这两种匹配程度下，可通过调整提示词和设置辅助模型，来适应角色一定程度上的改变，且可保证角色的一致性。在第三匹配程度下，可根据角色标签重新确定角色模型，并可根据提示词和重新确定的角色模型确定控制条件。从而，可保证角色形象发生较大改变时，图像中角色呈现的正确性。
示例性的，图2为本公开实施例所提供的一种图像生成方法的流程示意图。如图2所示，本实施例提供的图像生成方法，可以包括：
S210、接收内容文本，确定内容文本中角色的描述文本。
S220、通过预设语言模型，根据描述文本生成提示词，并根据提示词确定角色标签。
S230、确定角色标签与预设的角色模型的匹配程度。
本实施例中，其中角色模型包括面部的低秩自适应模型。
S241、在匹配程度满足第一匹配程度的情况下，根据提示词和预设的角色模型，确定控制条件。
本实施例中，第一匹配程度可表征角色标签与已关联模型基本匹配。在此情况下，可直接将当前的提示词和预设的角色模型，作为控制条件。并且，针对辅助信息中的部分差异(例如瞳孔、头发和服装等对象的颜色差异)，也可根据当前的提示词更新预先存储的辅助信息。
S242、在匹配程度满足第二匹配程度的情况下，根据提示词确定辅助标签。
本实施例中，第二匹配程度可表征角色标签与预设的模型存在差异。例如，角色年龄在一定程度内变化时，角色的容貌因年龄变化而发生细小、复杂的改变(例如脸上出现了细纹等)；又如，角色在特殊场合穿上了特定服装，例如角色身着婚纱等。在此情况下，可从提示词中提取出辅助标签；其中，可以预先设置至少一个描述维度的关键词作为辅助标签，例如辅助标签可以包括提示词中与穿着和年龄等维度相关的关键词。
S252、从预设辅助库中确定与辅助标签匹配的辅助模型，并关联角色与辅助模型。
本实施例中，辅助模型包括服装和/或年龄的调节模型，且调节模型也可包括低秩自适应模型。在预设辅助库中可以包含预先构建的多个辅助模型，且各辅助模型在构建过程中，也可以基于设置的标签进行构建。相应的，可以从预设辅助库中筛选出命中各辅助标签的候选辅助模型。之后，可将候选辅助模型中已与其他角色关联的辅助模型进行筛除，并可从筛除后的候选辅助模型中随机选取，得到与辅助标签匹配的辅助模型。
其中，辅助模型也可包含于辅助信息，可以将角色与辅助模型的对应关系进行存储，以关联角色和辅助模型。此外，当角色已关联辅助模型的情况下，还可以基于当前匹配的辅助模型更新预设的辅助模型。
S262、根据提示词、预设的角色模型和辅助模型，确定控制条件。
本实施例中，可将提示词、预设的角色模型和当前匹配的辅助模型，作为控制条件。
S243、在匹配程度满足第三匹配程度的情况下，从预设角色库中确定与角色标签匹配的新的角色模型，并关联角色与新的角色模型。
本实施例中，第三匹配程度可表征角色标签与预设的角色模型基本不同。例如，角色年龄进行了较大跨度的改变(例如从少年到老年)，导致角色形象变化很大等。在此情况下，可从预设角色库中确定与角色标签匹配的新的角色模型。
其中，预设角色库可以根据预先构建的角色模型生成，且预设角色库中可存储各角色模型的标签。其中，可以从预设角色库中筛选出命中各角色标签的候选角色模型。之后，可将候选角色模型中已与其他角色关联的角色模型进行筛除，并可从筛除后的候选角色模型中随机选取角色模型作为新的角色模型。在确定与角色匹配的新的角色模型后，可基于新的角色模型更新预设的角色模型，即存储角色与新的角色模型的对应关系。
S253、根据提示词和新的角色模型，确定控制条件。
本实施例中，可将提示词和新的角色模型作为控制条件。
S270、通过预设生成模型，根据控制条件生成角色的角色图像。
S280、根据当前生成的角色图像，以及角色对应的历史角色图像，确定角色相似度。
本实施例中，可确定当前生成的角色图像，与该角色对应的预设数量的历史角色图像进行角色相似度确定。示例性的，可以确定当前生成的角色图像，与该角色对应的前一历史角色图像进行角色相似度确定。
其中，角色相似度的确定方法，例如可以包括：提取当前生成的角色图像的第一面部特征，以及提取历史角色图像中的第二面部特征；根据已有特征相似度确定方式，确定第一面部特征与第二面部特征的特征相似度；根据特征相似度，确定角色相似度。又如还可以包括：将当前生成的角色图像、与该角色对应的历史角色图像，以及角色相似度判定的提示词(例如包括“请输出图像内角色的相似度”)，输入具备多模态理解能力的语言模型中；通过该多模态语言模型输出角色相似度。
S290、响应于角色相似度小于预设阈值，根据控制条件重新生成角色图像。
本实施例中，相同角色的角色图像基于相同角色模型生成的情况下，图像中角色的一致性较高；相同角色的角色图像基于不同角色模型生成的情况下，图像中角色在脸型、瞳色等细节上仍需具备相似性。
其中，可根据业务需求来设置预设阈值。在角色相似度小于预设阈值的情况下，可认为不满足同一角色的相似性需求。此时，可根据控制条件，控制新的随机噪声图像到角色图像的扩散过程，以重新生成角色图像。直至角色相似度大于等于预设阈值，可停止角色图像的重复生成过程，得到满足相似性需求的角色图像。
本公开实施例的技术方案，对不同匹配程度的情况下控制条件的确定进行了详细描述。在第一匹配程度的情况下，可直接根据提示词和预设的角色模型来确定控制条件；在第二匹配程度下，可根据提示词确定辅助模型，根据提示词、预设的角色模型和辅助模型来确定控制条件。在这两种匹配程度下，可通过调整提示词和设置辅助模型，来适应角色一定程度上的改变，且可保证角色的一致性。在第三匹配程度下，可根据角色标签重新确定角色模型，并可根据提示词和重新确定的角色模型确定控制条件。从而，可保证角色形象发生较大改变时，图像中角色呈现的正确性。
此外，本公开实施例提供的图像生成方法与上述实施例提供的图像生成方法属于同一公开构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且相同的技术特征在本实施例与上述实施例中具有相同的有益效果。
本公开实施例与上述实施例中所提供的图像生成方法中各个可选方案可以结合。本实施例所提供的图像生成方法，对未确定预设的角色模型的情况下，确定控制条件进行了详细描述，能够在角色未确定预设的角色模型的情况下，实现首次关联角色模型，且可根据提示词和角色模型确定控制条件。
图3为本公开实施例所提供的一种图像生成方法的流程示意图。如图3所示，本实施例提供的图像生成方法，可以包括：
S301、接收内容文本，确定内容文本中角色的描述文本。
S302、通过预设语言模型，根据描述文本生成提示词，并根据提示词确定角色标签。
S303、判断角色是否确定预设的角色模型；若否，则跳转至S304；若是，则跳转至S306；
其中，可根据预先存储角色与预设的角色模型的对应关系，确定当前角色是否已经确定预设的角色模型，即是否与预设的角色模型关联。
S304、从预设角色库中确定与角色标签匹配的角色模型，将匹配的角色模型作为预设的角色模型。
本公开实施例中，当整体内容文本首次图像化或内容文本出现新角色的情况下，内容文本中的角色还未关联角色模型。在此情况下，可从预设角色库中确定与角色标签匹配的角色模型，并关联角色与角色模型。其中，确定匹配的角色模型、关联角色与角色模型的过程，可参考上述确定匹配的新的角色模型，以及关联角色与新的角色模型的过程，在此不做赘述。
此外，为角色首次匹配角色模型情况下，若角色的描述文本中包含穿着特定服装的文本，则还可以根据穿着描述维度的提示词，确定辅助标签；并可从预设辅助库中确定与辅助标签匹配的辅助模型，并关联角色与辅助模型。
其中，在确定与角色相关的角色模型、辅助模型和其他辅助信息后，可存储角色与角色模型、辅助模型和其他辅助信息的对应关系，以关联角色与角色模型、辅助模型和其他辅助信息。
S305、根据提示词和预设的角色模型，确定控制条件。
S306、确定角色标签与预设的角色模型的匹配程度。
S307、响应于匹配程度，根据提示词、角色标签和预设的角色模型中的至少一项，确定控制条件。
S308、通过预设生成模型，根据控制条件生成角色的角色图像。
在实际应用中，可能出现预设角色库中不包含命中角色标签的角色模型的情况，可认为从预设角色库中确定角色模型失败。此时，可针对角色标签自动构建匹配的角色模型。图4为本公开实施例所提供的一种图像生成方法中构建角色模型的数据流框图。参见图4，在一些可选的实现方式中，在角色模型确定失败的情况下，还可以包括：
通过预设生成模型，根据角色标签生成图像集合；提取图像集合中各图像的面部特征，并根据各面部特征进行图像聚类，得到最相似簇；根据最相似簇内的图像，构建与角色标签匹配的角色模型；将构建的角色模型加载至预设生成模型，以生成新的图像集合；响应于最相似簇满足预设条件，将构建的角色模型作为预设的角色模型。
其中，可将角色标签作为控制条件。构建角色模型的过程可包括：首先，可通过预设生成模型，根据该控制条件，控制不同噪声图像到角色图像的扩散过程，以生成包含足够多图像的图像集合。其次，可通过已有的面部特征提取方式，对图像集合中各图像的面部特征进行提取。再次，可根据已有聚类算法(如K-means算法等)，根据各图像的面部特征进行图像聚类，且可选择最有凝聚力的一个聚类簇作为最相似簇。其中，凝聚力可以基于聚类簇中各图像的面部特征与质心之间的平均距离确定，且凝聚力与平均距离呈反相关；其中，质心可根据各图像的面部特征来确定。然后，可根据最相似簇内的图像进行身份提取，以构建角色模型。接着，可将当前轮次构建的角色模型也作为控制条件，加载至预设生成模型中，以根据不同早上图像生成新的图像集合。
之后，可重复进行图像集合中图像面部特征提取、基于面部特征进行图像聚类、根据最相似簇构建角色模型，以及将最新构建的角色模型加载至预设生成模型，以生成新的图像集合的步骤，直至最相似簇满足预设条件(例如凝聚力满足预设数值)，可认为角色模型收敛，即得到构建完成的角色模型。
在这些可选的实现方式中，角色模型构建流程可实现无监督或半监督的角色模型构建。其中，也可以基于该流程，来预先构建预设角色库。并且，在预设角色库具备一定规模的角色模型后，也可以通过不同角色模型加权组合等方式，生成新的角色模型。且对于相同角色标签，也可训练多个对应的角色模型，以丰富预设角色库。
此外，也可以通过多种渠道获取符合角色标签的角色图像，基于该些角色图像直接进行角色模型的构建。可以理解的是，本技术方案中涉及的角色图像(包括但不限于角色图像本身、角色图像的获取或使用)应当遵循相应法律法规及相关规定的要求。
可以理解的是，在角色标签与预设的角色模型匹配程度较低的情况下，新的角色模型确定失败时，也可以执行上述操作进行新的角色模型的构建，且构建好的新的角色模型也可存储于预设角色库中。在辅助模型确定失败的情况下，也可以执行上述操作进行辅助模型的构建，且构建好的辅助模型也可存储于预设辅助库中。此外，也可以基于上述流程，来预先构建预设辅助库。
本公开实施例的技术方案，对未确定预设的角色模型的情况下，确定控制条件进行了详细描述，能够在角色未确定预设的角色模型的情况下，实现首次关联角色模型，且可根据提示词和角色模型确定控制条件。此外，本公开实施例提供的图像生成方法与上述实施例提供的图像生成方法属于同一公开构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且相同的技术特征在本实施例与上述实施例中具有相同的有益效果。
本公开实施例与上述实施例中所提供的图像生成方法中各个可选方案可以结合。本实施例所提供的图像生成方法，对生成角色图像后的风格化和视频化进行了详细描述。通过根据参数量较小的风格化模型对角色图像进行风格化，能够实现在保证角色一致性基础上，进行低成本、高质量地风格迁移。通过根据角色图像生成视频，可进一步提升用户视觉体验。
本实施例所提供的图像生成方法中，在生成角色图像后，还可以包括：对角色图像进行至少一个预设维度的特征提取，得到图像特征；通过预设风格化模型，根据图像特征生成风格化图像。
其中，风格化模型可以包括具备图生图能力的扩散模型。其中，风格化模型可根据样本角色图像和样本风格化图像进行构建。相较于预设生成模型，风格化模块可认为是参数小，但具备强烈风格化的小模型。
图5为本公开实施例所提供的一种图像生成方法中图像风格化的数据流框图。如图5所示，对角色图像进行至少一个预设维度的特征提取，可以包括对角色图像进行表情、姿态、画面内容和景深等至少一个预设维度的特征提取，得到对应预设维度的图像特征。在提取图像特征后，可以将角色图像对应的提示词以及图像特征输入漫画风格、三维风格等至少一个风格化模型中，以根据风格化模型进行相应的风格迁移，得到风格化图像。
本实施例中，在角色图像基础上，由风格化模型进行风格迁移，以满足画风的多样性要求。在实际生产中，在角色图像生成阶段可采用参数多、性能强的预设生成模型，以保证角色图像细节的丰富程度和画面准确性。在风格化图像生成阶段，可采参数小，但具备强烈风格化的风格化模型，以节约计算性能。通过串联预设生成模型和风格化模型，可在保证角色一致性基础上，适配多种画风、减少适配成本。
此外，本实施例所提供的图像生成方法中，在生成角色图像后，还可以包括：依序根据内容文本对应的各角色图像，生成目标视频。
本实施例中，在根据内容文本生成各角色的角色图像后，可按照内容文本的行文顺序，依序根据对应的各角色图像生成目标视频。例如，可以每隔预设时间(例如3秒)切换角色图像，以生成目标视频。此外，还可以根据已有的基于图像生成视频的方式，根据各角色图像生成视频。
本公开实施例的技术方案，对生成角色图像后的风格化和视频化进行了详细描述。通过根据参数量较小的风格化模型对角色图像进行风格化，能够实现在保证角色一致性基础上，进行低成本、高质量地风格迁移。通过根据角色图像生成视频，可进一步提升用户视觉体验。此外，本公开实施例提供的图像生成方法与上述实施例提供的图像生成方法属于同一公开构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且相同的技术特征在本实施例与上述实施例中具有相同的有益效果。
图6为本公开实施例所提供的一种图像生成装置的结构示意图。本实施例提供的图像生成装置适用于生成角色图像的情形，例如在较长文本篇幅情况下生成文本对应的角色图像的情形。
如图6所示，本公开实施例提供的图像生成装置，可以包括：
文本接收模块610，用于接收内容文本，确定内容文本中角色的描述文本；
文本理解模块620，用于通过预设语言模型，根据描述文本生成提示词，并根据提示词确定角色标签；
匹配模块630，用于确定角色标签与预设的角色模型的匹配程度；
控制条件确定模块640，用于响应于匹配程度，根据提示词、角色标签和预设的角色模型中的至少一项，确定控制条件；
图像生成模块650，用于通过预设生成模型，根据控制条件生成角色的角色图像。
在一些可选的实现方式中，控制条件确定模块，可以用于：
在匹配程度满足第一匹配程度的情况下，根据提示词和预设的角色模型，确定控制条件。
在一些可选的实现方式中，控制条件确定模块，可以用于：
在匹配程度满足第二匹配程度的情况下，根据提示词确定辅助标签；
从预设辅助库中确定与辅助标签匹配的辅助模型，并关联角色与辅助模型；其中，辅助模型包括服装和/或年龄的调节模型；
根据提示词、预设的角色模型和辅助模型，确定控制条件。
在一些可选的实现方式中，控制条件确定模块，可以用于：
在匹配程度满足第三匹配程度的情况下，从预设角色库中确定与角色标签匹配的新的角色模型，并关联角色与新的角色模型；
根据提示词和新的角色模型，确定控制条件。
在一些可选的实现方式中，角色模型包括面部的低秩自适应模型。
在一些可选的实现方式中，图像生成装置，还可以包括：
相似度确定模块，用于在生成角色图像后，根据当前生成的角色图像，以及角色对应的历史角色图像，确定角色相似度；
图像生成模块，还可以用于响应于角色相似度小于预设阈值，根据控制条件重新生成角色图像。
在一些可选的实现方式中，匹配模块，还可以用于：从预设角色库中确定与角色标签匹配的角色模型，将匹配的角色模型作为预设的角色模型。
在一些可选的实现方式中，图像生成装置，还可以包括：
模型构建模块，用于在角色模型确定失败的情况下，通过预设生成模型，根据角色标签生成图像集合；
提取图像集合中各图像的面部特征，并根据各面部特征进行图像聚类，得到最相似簇；
根据最相似簇内的图像，构建与角色标签匹配的角色模型；
将构建的角色模型加载至预设生成模型，以生成新的图像集合；
响应于最相似簇满足预设条件，将构建的角色模型作为预设的角色模型。
在一些可选的实现方式中，图像生成装置，还可以包括：
风格化模块，用于在生成角色图像后，对角色图像进行至少一个预设维度的特征提取，得到图像特征；
通过预设风格化模型，根据图像特征生成风格化图像。
在一些可选的实现方式中，图像生成装置，还可以包括：
视频化化模块，用于在生成角色图像后，依序根据内容文本对应的各角色图像，生成目标视频。
本公开实施例所提供的图像生成装置，可执行本公开任意实施例所提供的图像生成方法，具备执行方法相应的功能模块和有益效果。
值得注意的是，上述装置所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本公开实施例的保护范围。
下面参考图7，其示出了适于用来实现本公开实施例的电子设备(例如图7中的终端设备或服务器)700的结构示意图。本公开实施例中的终端设备可以包括但不限于诸如移动电话、笔记本电脑、数字广播接收器、PDA(个人数字助理)、PAD(平板电脑)、PMP(便携式多媒体播放器)、车载终端(例如车载导航终端)等等的移动终端以及诸如数字TV、台式计算机等等的固定终端。图7示出的电子设备仅仅是一个示例，不应对本公开实施例的功能和使用范围带来任何限制。
如图7所示，电子设备700可以包括处理装置(例如中央处理器、图形处理器等)701，其可以根据存储在只读存储器(Read-Only Memory，ROM)702中的程序或者从存储装置708加载到随机访问存储器(Random Access Memory，RAM)703中的程序而执行各种适当的动作和处理。在RAM 703中，还存储有电子设备700操作所需的各种程序和数据。处理装置701、ROM 702以及RAM 703通过总线704彼此相连。输入/输出(I/O)接口705也连接至总线704。
通常，以下装置可以连接至I/O接口705：包括例如触摸屏、触摸板、键盘、鼠标、摄像头、麦克风、加速度计、陀螺仪等的输入装置706；包括例如液晶显示器(LCD)、扬声器、振动器等的输出装置707；包括例如磁带、硬盘等的存储装置708；以及通信装置709。通信装置709可以允许电子设备700与其他设备进行无线或有线通信以交换数据。虽然图7示出了具有各种装置的电子设备700，但是应理解的是，并不要求实施或具备所有示出的装置。可以替代地实施或具备更多或更少的装置。
特别地，根据本公开的实施例，上文参考流程图描述的过程可以被实现为计算机软件程序。例如，本公开的实施例包括一种计算机程序产品，其包括承载在非暂态计算机可读介质上的计算机程序，该计算机程序包含用于执行流程图所示的方法的程序代码。在这样的实施例中，该计算机程序可以通过通信装置709从网络上被下载和安装，或者从存储装置708被安装，或者从ROM702被安装。在该计算机程序被处理装置701执行时，执行本公开实施例的图像生成方法中限定的上述功能。
本公开实施例提供的电子设备与上述实施例提供的图像生成方法属于同一公开构思，未在本实施例中详尽描述的技术细节可参见上述实施例，并且本实施例与上述实施例具有相同的有益效果。
本公开实施例提供了一种计算机可执行指令的存储介质，计算机可执行指令在由计算机处理器执行时可用于执行上述实施例所提供的图像生成方法。
需要说明的是，本公开上述的存储介质可以是计算机可读信号介质或者计算机可读存储介质或者是上述两者的任意组合。计算机可读存储介质例如可以是——但不限于——电、磁、光、电磁、红外线、或半导体的系统、装置或器件，或者任意以上的组合。计算机可读存储介质的更具体的例子可以包括但不限于：具有一个或多个导线的电连接、便携式计算机磁盘、硬盘、随机访问存储器(RAM)、只读存储器(ROM)、可擦式可编程只读存储器(Erasable Programmable Read-Only Memory，EPROM)或闪存(FLASH)、光纤、便携式紧凑磁盘只读存储器(CD-ROM)、光存储器件、磁存储器件、或者上述的任意合适的组合。在本公开中，计算机可读存储介质可以是任何包含或存储可执行指令的有形介质，该可执行指令可以被指令执行系统、装置或者器件使用或者与其结合使用。而在本公开中，计算机可读信号介质可以包括在基带中或者作为载波一部分传播的数据信号，其中承载了计算机可读的可执行指令。这种传播的数据信号可以采用多种形式，包括但不限于电磁信号、光信号或上述的任意合适的组合。计算机可读信号介质还可以是计算机可读存储介质以外的任何计算机可读介质，该计算机可读信号介质可以发送、传播或者传输用于由指令执行系统、装置或者器件使用或者与其结合使用的可执行指令。存储介质上包含的可执行指令可以用任何适当的介质传输，包括但不限于：电线、光缆、RF(射频)等等，或者上述的任意合适的组合。
在一些实施方式中，客户端、服务器可以利用诸如HTTP(Hyper Text Transfer Protocol，超文本传输协议)之类的任何当前已知或未来研发的网络协议进行通信，并且可以与任意形式或介质的数字数据通信(例如，通信网络)互连。通信网络的示例包括局域网(“LAN”)，广域网(“WAN”)，网际网(例如，互联网)以及端对端网络(例如，ad hoc端对端网络)，以及任何当前已知或未来研发的网络。
上述存储介质可以是上述电子设备中所包含的；也可以是单独存在，而未装配入该电子设备中。
上述存储介质承载有一个或者多个可执行指令，当上述一个或者多个可执行指令被该电子设备执行时，使得该电子设备：
接收内容文本，确定内容文本中角色的描述文本；通过预设语言模型，根据描述文本生成提示词，并根据提示词确定角色标签；确定角色标签与预设的角色模型的匹配程度；响应于匹配程度，根据提示词、角色标签和预设的角色模型中的至少一项，确定控制条件；通过预设生成模型，根据控制条件生成角色的角色图像。
可以以一种或多种程序设计语言或其组合来编写用于执行本公开的操作的可执行指令，上述程序设计语言包括但不限于面向对象的程序设计语言—诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言—诸如“C”语言或类似的程序设计语言。可执行指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
本公开实施例还提供了一种计算机程序产品，包括计算机程序，该计算机程序在被处理器执行时可实现如本公开任意一个实施例所提供的图像生成方法。
计算机程序产品在实现的过程中，可以以一种或多种程序设计语言或其组合来编写用于执行本公开操作的计算机程序代码，程序设计语言包括面向对象的程序设计语言，诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言，诸如“C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。
附图中的流程图和框图，图示了按照本公开各种实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段、或代码的一部分，该模块、程序段、或代码的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。也应当注意，在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个接连地表示的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或操作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
描述于本公开实施例中所涉及到的单元可以通过软件的方式实现，也可以通过硬件的方式来实现。其中，单元、模块的名称在某种情况下并不构成对该单元、模块本身的限定。
本文中以上描述的功能可以至少部分地由一个或多个硬件逻辑部件来执行。例如，非限制性地，可以使用的示范类型的硬件逻辑部件包括：现场可编程门阵列(Field Programmable Gate Array，FPGA)、专用集成电路(Application Specific Integrated Circuit，ASIC)、专用标准产品(Application Specific Standard Parts，ASSP)、片上系统(System on Chip，SOC)、复杂可编程逻辑设备(CPLD)等等。
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。
根据本公开的一个或多个实施例，提供了一种图像生成方法，该方法包括：
接收内容文本，确定所述内容文本中角色的描述文本；
通过预设语言模型，根据所述描述文本生成提示词，并根据所述提示词确定角色标签；
确定所述角色标签与预设的角色模型的匹配程度；
响应于所述匹配程度，根据所述提示词、所述角色标签和所述预设的角色模型中的至少一项，确定控制条件；
通过预设生成模型，根据所述控制条件生成所述角色的角色图像。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，所述响应于所述匹配程度，根据所述角色标签、所述提示词和所述预设的角色模型中的至少一项，确定控制条件，包括：
在所述匹配程度满足第一匹配程度的情况下，根据所述提示词和所述预设的角色模型，确定控制条件。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，所述响应于所述匹配程度，根据所述角色标签、所述提示词和所述预设的角色模型中的至少一项，确定控制条件，包括：
在所述匹配程度满足第二匹配程度的情况下，根据所述提示词确定辅助标签；
从预设辅助库中确定与所述辅助标签匹配的辅助模型，并关联所述角色与所述辅助模型；其中，所述辅助模型包括服装和/或年龄的调节模型；
根据所述提示词、所述预设的角色模型和所述辅助模型，确定控制条件。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，所述响应于所述匹配程度，根据所述角色标签、所述提示词和所述预设的角色模型中的至少一项，确定控制条件，包括：
在所述匹配程度满足第三匹配程度的情况下，从预设角色库中确定与所述角色标签匹配的新的角色模型，并关联所述角色与所述新的角色模型；
根据所述提示词和所述新的角色模型，确定控制条件。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，所述角色模型包括面部的低秩自适应模型。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，在生成所述角色图像后，还包括：
根据当前生成的所述角色图像，以及所述角色对应的历史角色图像，确定角色相似度；
响应于所述角色相似度小于预设阈值，根据所述控制条件重新生成角色图像。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，所述预设的角色模型的确定步骤，包括：
从预设角色库中确定与所述角色标签匹配的角色模型，将所述匹配的角色模型作为预设的角色模型。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，在所述角色模型确定失败的情况下，还包括：
通过所述预设生成模型，根据所述角色标签生成图像集合；
提取所述图像集合中各图像的面部特征，并根据各所述面部特征进行图像聚类，得到最相似簇；
根据所述最相似簇内的图像，构建与所述角色标签匹配的角色模型；
将构建的角色模型加载至所述预设生成模型，以生成新的图像集合；
响应于所述最相似簇满足预设条件，将所述构建的角色模型作为预设的角色模型。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，在生成所述角色图像后，还包括：
对所述角色图像进行至少一个预设维度的特征提取，得到图像特征；
通过预设风格化模型，根据所述图像特征生成风格化图像。
根据本公开的一个或多个实施例，提供了一种图像生成方法，还包括：
在一些可选的实现方式中，在生成所述角色图像后，还包括：
依序根据所述内容文本对应的各角色图像，生成目标视频。
根据本公开的一个或多个实施例，提供了一种图像生成装置，该装置包括：
文本接收模块，用于接收内容文本，确定所述内容文本中角色的描述文本；
文本理解模块，用于通过预设语言模型，根据所述描述文本生成提示词，并根据所述提示词确定角色标签；
匹配模块，用于确定所述角色标签与预设的角色模型的匹配程度；
控制条件确定模块，用于响应于所述匹配程度，根据所述提示词、所述角色标签和所述预设的角色模型中的至少一项，确定控制条件；
图像生成模块，用于通过预设生成模型，根据所述控制条件生成所述角色的角色图像。
以上描述仅为本公开的较佳实施例以及对所运用技术原理的说明。本领域技术人员应当理解，本公开中所涉及的公开范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离上述公开构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。例如上述特征与本公开中公开的(但不限于)具有类似功能的技术特征进行互相替换而形成的技术方案。
此外，虽然采用特定次序描绘了各操作，但是这不应当理解为要求这些操作以所示出的特定次序或以顺序次序执行来执行。在一定环境下，多任务和并行处理可能是有利的。同样地，虽然在上面论述中包含了若干具体实现细节，但是这些不应当被解释为对本公开的范围的限制。在单独的实施例的上下文中描述的某些特征还可以组合地实现在单个实施例中。相反地，在单个实施例的上下文中描述的各种特征也可以单独地或以任何合适的子组合的方式实现在多个实施例中。
尽管已经采用特定于结构特征和/或方法逻辑动作的语言描述了本主题，但是应当理解所附权利要求书中所限定的主题未必局限于上面描述的特定特征或动作。相反，上面所描述的特定特征和动作仅仅是实现权利要求书的示例形式。

Claims (14)

  1. 一种图像生成方法，包括：
    接收内容文本，确定所述内容文本中角色的描述文本；
    通过预设语言模型，根据所述描述文本生成提示词，并根据所述提示词确定角色标签；
    确定所述角色标签与预设的角色模型的匹配程度；
    响应于所述匹配程度，根据所述提示词、所述角色标签和所述预设的角色模型中的至少一项，确定控制条件；
    通过预设生成模型，根据所述控制条件生成所述角色的角色图像。
  2. 根据权利要求1所述的方法，其中，所述响应于所述匹配程度，根据所述角色标签、所述提示词和所述预设的角色模型中的至少一项，确定控制条件，包括：
    在所述匹配程度满足第一匹配程度的情况下，根据所述提示词和所述预设的角色模型，确定控制条件。
  3. 根据权利要求1所述的方法，其中，所述响应于所述匹配程度，根据所述角色标签、所述提示词和所述预设的角色模型中的至少一项，确定控制条件，包括：
    在所述匹配程度满足第二匹配程度的情况下，根据所述提示词确定辅助标签；
    从预设辅助库中确定与所述辅助标签匹配的辅助模型，并关联所述角色与所述辅助模型；其中，所述辅助模型包括服装和/或年龄的调节模型；
    根据所述提示词、所述预设的角色模型和所述辅助模型，确定控制条件。
  4. 根据权利要求1所述的方法，其中，所述响应于所述匹配程度，根据所述角色标签、所述提示词和所述预设的角色模型中的至少一项，确定控制条件，包括：
    在所述匹配程度满足第三匹配程度的情况下，从预设角色库中确定与所述角色标签匹配的新的角色模型，并关联所述角色与所述新的角色模型；
    根据所述提示词和所述新的角色模型，确定控制条件。
  5. 根据权利要求1-4中任一项所述的方法，其中，所述角色模型包括面部的低秩自适应模型。
  6. 根据权利要求1-5中任一项所述的方法，其中，在生成所述角色图像后，所述方法还包括：
    根据当前生成的所述角色图像，以及所述角色对应的历史角色图像，确定角色相似度；
    响应于所述角色相似度小于预设阈值，根据所述控制条件重新生成角色图像。
  7. 根据权利要求1-6任一项所述的方法，其中，所述预设的角色模型的确定步骤，包括：
    从预设角色库中确定与所述角色标签匹配的角色模型，将所述匹配的角色模型作为预设的角色模型。
  8. 根据权利要求7所述的方法，其中，在所述预设的角色模型确定失败的情况下，所述方法还包括：
    通过所述预设生成模型，根据所述角色标签生成图像集合；
    提取所述图像集合中各图像的面部特征，并根据各所述面部特征进行图像聚类，得到最相似簇；
    根据所述最相似簇内的图像，构建与所述角色标签匹配的角色模型；
    将构建的角色模型加载至所述预设生成模型，以生成新的图像集合；
    响应于所述最相似簇满足预设条件，将所述构建的角色模型作为预设的角色模型。
  9. 根据权利要求1-8任一项所述的方法，其中，在生成所述角色图像后，所述方法还包括：
    对所述角色图像进行至少一个预设维度的特征提取，得到图像特征；
    通过预设风格化模型，根据所述图像特征生成风格化图像。
  10. 根据权利要求1-9任一项所述的方法，其中，在生成所述角色图像后，所述方法还包括：
    依序根据所述内容文本对应的各角色图像，生成目标视频。
  11. 一种图像生成装置，包括：
    文本接收模块，被配置为接收内容文本，确定所述内容文本中角色的描述文本；
    文本理解模块，被配置为通过预设语言模型，根据所述描述文本生成提示词，并根据所述提示词确定角色标签；
    匹配模块，被配置为确定所述角色标签与预设的角色模型的匹配程度；
    控制条件确定模块，被配置为响应于所述匹配程度，根据所述提示词、所述角色标签和所述预设的角色模型中的至少一项，确定控制条件；
    图像生成模块，被配置为通过预设生成模型，根据所述控制条件生成所述角色的角色图像。
  12. 一种电子设备，包括：
    一个或多个处理器；
    存储装置，被配置为存储一个或多个程序，其中，
    当所述一个或多个程序被所述一个或多个处理器执行时，使得所述一个或多个处理器实现如权利要求1-10中任一项所述的图像生成方法。
  13. 一种包含计算机可执行指令的存储介质，其中，所述计算机可执行指令在由计算机处理器执行时用于执行如权利要求1-10中任一项所述的图像生成方法。
  14. 一种计算机程序产品，包括计算机程序，其中，所述计算机程序在被处理器执行时实现根据权利要求1-10中任一项所述的图像生成方法。
PCT/CN2025/100222 2024-06-11 2025-06-10 图像生成方法、装置、电子设备及存储介质 Pending WO2025256530A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410745318.6 2024-06-11
CN202410745318.6A CN121119105A (zh) 2024-06-11 2024-06-11 一种图像生成方法、装置、电子设备及存储介质

Publications (1)

Publication Number Publication Date
WO2025256530A1 true WO2025256530A1 (zh) 2025-12-18

Family

ID=97948721

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/100222 Pending WO2025256530A1 (zh) 2024-06-11 2025-06-10 图像生成方法、装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN121119105A (zh)
WO (1) WO2025256530A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230005284A1 (en) * 2021-09-18 2023-01-05 Beijing Baidu Netcom Science Technology Co., Ltd. Method for training image-text matching model, computing device, and storage medium
CN117032869A (zh) * 2023-08-14 2023-11-10 科大讯飞股份有限公司 图文内容生成方法、装置、设备及存储介质
CN117171369A (zh) * 2023-09-15 2023-12-05 北京字跳网络技术有限公司 一种内容生成方法、装置、计算机设备及存储介质
CN117392254A (zh) * 2023-10-18 2024-01-12 腾讯科技（深圳）有限公司 图像生成方法、装置、终端及存储介质
CN118052907A (zh) * 2024-02-19 2024-05-17 腾讯科技（深圳）有限公司 一种文本配图生成方法和相关装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230005284A1 (en) * 2021-09-18 2023-01-05 Beijing Baidu Netcom Science Technology Co., Ltd. Method for training image-text matching model, computing device, and storage medium
CN117032869A (zh) * 2023-08-14 2023-11-10 科大讯飞股份有限公司 图文内容生成方法、装置、设备及存储介质
CN117171369A (zh) * 2023-09-15 2023-12-05 北京字跳网络技术有限公司 一种内容生成方法、装置、计算机设备及存储介质
CN117392254A (zh) * 2023-10-18 2024-01-12 腾讯科技（深圳）有限公司 图像生成方法、装置、终端及存储介质
CN118052907A (zh) * 2024-02-19 2024-05-17 腾讯科技（深圳）有限公司 一种文本配图生成方法和相关装置

Also Published As

Publication number Publication date
CN121119105A (zh) 2025-12-12

Similar Documents

Publication Publication Date Title
US12190407B2 (en) Virtual image generation method, device, terminal and storage medium
US20230100423A1 (en) Crowdsourced on-boarding of digital assistant operations
CN109993150B (zh) 用于识别年龄的方法和装置
CN113407850B (zh) 一种虚拟形象的确定和获取方法、装置以及电子设备
CN109189544B (zh) 用于生成表盘的方法和装置
CN111539903B (zh) 训练人脸图像合成模型的方法和装置
CN110381368A (zh) 视频封面生成方法、装置及电子设备
WO2020238320A1 (zh) 用于生成表情包的方法和装置
US10950240B2 (en) Information processing device and information processing method
CN111444321B (zh) 问答方法、装置、电子设备和存储介质
KR20190074508A (ko) 챗봇을 위한 대화 모델의 데이터 크라우드소싱 방법
WO2021088790A1 (zh) 用于目标设备的显示样式调整方法和装置
CN113971048A (zh) 应用程序的启动方法及装置、存储介质及电子设备
WO2021068493A1 (zh) 用于处理信息的方法和装置
CN110619602A (zh) 一种图像生成方法、装置、电子设备及存储介质
CN110286776A (zh) 字符组合信息的输入方法、装置、电子设备和存储介质
CN113709383B (zh) 配置视频特效方法、装置、设备及存储介质
CN111062995B (zh) 生成人脸图像的方法、装置、电子设备和计算机可读介质
WO2025256530A1 (zh) 图像生成方法、装置、电子设备及存储介质
WO2024193489A1 (zh) 一种图像处理方法、装置、设备及存储介质
US12249345B2 (en) Ephemeral learning and/or federated learning of audio-based machine learning model(s) from stream(s) of audio data generated via radio station(s)
US10847135B2 (en) Sharing commands and command groups across digital assistant operations
CN117974172A (zh) 导购信息获取方法和装置
CN110188713A (zh) 用于输出信息的方法和装置
CN117034959A (zh) 数据处理方法、装置、电子设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25821240

Country of ref document: EP

Kind code of ref document: A1