WO2025251771A1 - 图像描述生成模型处理方法、装置、计算机设备及存储介质 - Google Patents

图像描述生成模型处理方法、装置、计算机设备及存储介质

Info

Publication number
WO2025251771A1
WO2025251771A1 PCT/CN2025/087426 CN2025087426W WO2025251771A1 WO 2025251771 A1 WO2025251771 A1 WO 2025251771A1 CN 2025087426 W CN2025087426 W CN 2025087426W WO 2025251771 A1 WO2025251771 A1 WO 2025251771A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
description
text
model
generation model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2025/087426
Other languages
English (en)
French (fr)
Inventor
邓鑫驰
熊江丰
芦清林
蒋杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Publication of WO2025251771A1 publication Critical patent/WO2025251771A1/zh
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/12—Use of codes for handling textual entities
    • G06F40/151—Transformation
    • G06F40/16—Automatic learning of transformation rules, e.g. from examples
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0475—Generative networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Definitions

  • This application relates to the field of computer technology, and in particular to an image description generation model processing method, apparatus, computer device, and storage medium.
  • Image description generation technology refers to the technique of generating text that describes an image given an image. With the development of computer vision technology and natural language processing technology, image description generation technology has been widely used.
  • an image description generation model is typically trained based on training samples. After training, this model can generate descriptive text corresponding to any image.
  • the training samples include images and their corresponding descriptive text.
  • This application provides an image description generation model processing method, apparatus, computer device, and storage medium.
  • the technical solution is as follows:
  • Some embodiments of this application provide an image description generation model processing method, the method comprising:
  • the first image description generation model generates the first description text corresponding to the first image, and the first description text is corrected by the description correction model to obtain the first corrected description text.
  • a second descriptive text corresponding to the first image is generated through a second image description generation model, a quality score of the second descriptive text is obtained through the description correction model, and a second corrected descriptive text is obtained by correcting the second descriptive text through the description correction model.
  • a loss value is determined based on the first descriptive text, the first corrected descriptive text, the second descriptive text, and the second corrected descriptive text.
  • the second image description generation model is updated with the goal of reducing the loss value.
  • the updated second image description generation model is used to generate text describing the input image based on the input image.
  • the second image description generation model before the update has the same structure and state as the first image description generation model.
  • Some embodiments of this application also provide an image description generation model processing apparatus, the apparatus comprising:
  • the first description generation module is used to generate a first description text corresponding to a first image through a first image description generation model, and to correct the first description text through a description correction model to obtain a first corrected description text.
  • the second description generation module is used to generate a second description text corresponding to the first image through a second image description generation model, obtain a quality score of the second description text through the description correction model, and correct the second description text through the description correction model to obtain a second corrected description text.
  • the model update module is used to determine a loss value based on the first description text, the first corrected description text, the second description text, and the second corrected description text when the quality score of the second description text does not meet the quality conditions.
  • the second image description generation model is updated with the goal of reducing the loss value.
  • the updated second image description generation model is used to generate text describing the input image based on the input image.
  • the second image description generation model before the update has the same structure and state as the first image description generation model.
  • Some embodiments of this application also provide a computer device including a processor and a memory, wherein the memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed by the model processing method as described above.
  • Some embodiments of this application also provide a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to perform the operations performed by the model processing method as described above.
  • Some embodiments of this application also provide a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the model processing method as described above.
  • Figure 1 is a schematic diagram of an implementation environment provided in an embodiment of this application.
  • FIG. 2 is a flowchart of an image description generation model processing method provided in an embodiment of this application.
  • Figure 3 is a schematic diagram of the update process of a second image description generation model provided in an embodiment of this application.
  • FIG. 4 is a flowchart of another image description generation model processing method provided in an embodiment of this application.
  • Figure 5 is a flowchart illustrating a combination of multi-dimensional descriptive text provided in an embodiment of this application.
  • Figure 6 is a schematic flowchart of a training image description generation model provided in an embodiment of this application.
  • Figure 7 is a schematic flowchart of a training description correction model provided in an embodiment of this application.
  • FIG. 8 is a flowchart of another image description generation model processing method provided in an embodiment of this application.
  • Figure 9 is an architecture diagram of an image description generation model provided in an embodiment of this application.
  • Figure 10 is a schematic diagram of a training phase and an update phase provided in an embodiment of this application.
  • FIG 11 is a schematic diagram of another training stage provided in an embodiment of this application.
  • Figure 12 is a schematic diagram of the structure of an image description generation model processing device provided in an embodiment of this application.
  • Figure 13 is a schematic diagram of another image description generation model processing device provided in an embodiment of this application.
  • Figure 14 is a schematic diagram of the structure of a terminal provided in an embodiment of this application.
  • Figure 15 is a schematic diagram of the structure of a server provided in an embodiment of this application.
  • first a first image
  • second a second image
  • At least two means two or more.
  • at least two images can be any integer number of images greater than or equal to two, such as two images, three images, etc.
  • Each means each of the at least two images.
  • each image means every single image in the at least two images. If the at least two images are three images, then "each image” means every single image in the three images.
  • the information including but not limited to user equipment information, user personal information, etc.
  • data including but not limited to data used for analysis, data stored, data displayed, etc.
  • signals including but not limited to signals transmitted between user terminals and other devices
  • the computer device is a terminal or a server.
  • the terminal is a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, etc., but is not limited thereto.
  • the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.
  • the embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.
  • the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network.
  • Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.
  • the computer device in this application is a node in a blockchain system.
  • the node can store the model in the blockchain, and then the node or other nodes in the blockchain can access the blockchain to call the model.
  • a description generation model is typically trained based on training samples. After training, this model can generate descriptive text corresponding to any image.
  • the training samples include images and their corresponding descriptive text. Since training samples are usually crawled from the internet, their quality is often poor, leading to poor performance of the trained image description generation model.
  • This application provides an image description generation model processing scheme. By using a first image description generation model and a description correction model, even if the quality score of the second descriptive text generated by the second image description generation model does not meet the quality requirements, the second descriptive text, along with the second corrected descriptive text obtained through the description correction model, the first descriptive text generated by the first image description generation model, and the first corrected descriptive text obtained through the description correction model, can be used together to update the second image description generation model. This improves the performance of the second image description generation model, saves the number of samples, and avoids wasting low-quality descriptive text.
  • Figure 1 is a schematic diagram of an implementation environment provided in an embodiment of this application.
  • the implementation environment includes a terminal 101 and a server 102, and the terminal 101 and the server 102 are connected through a wired network or a wireless network.
  • Terminal 101 has a client 111 installed and running.
  • This client 111 can be a video sharing client, a social application client, a game client, or the like.
  • client 111 runs client 111, the user interface of client 111 is displayed on the screen of terminal 101.
  • the number of terminals described above can be more or less. For example, there may be only one terminal, or there may be six, eight, or more terminals. This application does not limit the number of terminals or the type of device.
  • Figure 1 shows only one terminal, but in different embodiments, multiple other terminals 103 can access the server 102.
  • one or more terminals 103 are terminals corresponding to developers, and a client development and editing platform is installed on the terminal 103. Developers can edit and update the client on the terminal 103 and transmit the updated client installation package to the server 102 via wired or wireless network.
  • Terminal 101 can download the client installation package from the server 102 to update the client.
  • Terminal 101 and other terminals 103 are connected to server 102 via wired or wireless networks.
  • Server 102 is used to provide background services for client 111 on terminal 101.
  • server 102 undertakes the main computing work and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 use a distributed computing architecture for collaborative computing.
  • server 102 updates the second image description generation model using the method provided in this embodiment to improve its performance. Then, server 102 generates description text corresponding to the image using the updated second image description generation model to ensure that the description text accurately and comprehensively describes the image.
  • a user inputs an image using terminal 101, and terminal 101 or server 102 generates descriptive text corresponding to the image through a second image description generation model.
  • terminal 101 determines the image the user wants to post and sends the image to server 102.
  • Server 102 then generates a corresponding descriptive text for the image using an updated second image description generation model and sends the descriptive text to terminal 101.
  • Terminal 101 displays the image and the descriptive text. After the user clicks to confirm, the image and descriptive text are combined into a dynamic message and posted to the social media platform. The user only needs to select the image to be posted, and the corresponding descriptive text is automatically generated as the matching text, eliminating the need for the user to input text separately, thus enabling rapid posting of dynamic messages.
  • server 102 generates descriptive text corresponding to an image using an updated second image description generation model. Based on the image and its corresponding descriptive text, it trains the image generation model so that the trained model can generate an image that matches any descriptive text.
  • the user inputs descriptive text using terminal 101, and terminal 101 or server 102 generates an image that matches the descriptive text using the image generation model.
  • a user inputs descriptive text into terminal 101.
  • Terminal 101 sends the descriptive text to server 102, which then uses an image generation model to generate an image that matches the descriptive text and possesses a unique creative concept, and sends the image back to terminal 101.
  • the user can then share the image with friends, publish it on social media platforms, or generate a video containing the image.
  • Figure 2 is a flowchart of an image description generation model processing method provided in an embodiment of this application
  • Figure 3 is a schematic diagram of an update process for a second image description generation model provided in an embodiment of this application.
  • This embodiment of the application is executed by a computer device, such as a terminal 101 or server 102 as shown in Figure 1.
  • This embodiment of the application describes the process of updating the image description generation model. Referring to Figures 2 and 3, the method includes:
  • the computer device generates a first descriptive text corresponding to the first image through a first image description generation model, and corrects the first descriptive text through a description correction model to obtain a first corrected descriptive text.
  • the computer device acquires a first image description generation model, a second image description generation model, and a description correction model.
  • the three models will be described in detail below.
  • the first image description generation model is used to generate descriptive text corresponding to the image, thereby describing the image with descriptive text. Since the first image description generation model has been trained once or multiple times and already possesses the ability to generate descriptive text corresponding to images, it can be put into use. That is, for any image, the first image description generation model can generate descriptive text corresponding to that image.
  • the first image description generation model used in this application is a multimodal large language model.
  • a multimodal large language model is an artificial intelligence model capable of processing various types of data input, such as text, images, and audio, and capable of generating natural language output; therefore, it can be used to generate descriptive text corresponding to images.
  • the second image description generation model functions similarly to the first image description generation model, also used to generate descriptions corresponding to images, thus describing the images with descriptive text. Since the second image description generation model has been trained once or multiple times and already possesses the ability to generate descriptive text corresponding to images, it can be put into use. That is, for any image, the second image description generation model can generate the descriptive text corresponding to that image.
  • the image description generation model (e.g., the first image description generation model and the second image description generation model described above) may include two main components: an image feature extraction component and a text generation component.
  • the image feature extraction component converts images into feature vectors for subsequent text generation. This can be achieved using a Convolutional Neural Network (CNN).
  • CNN Convolutional Neural Network
  • the basic structure of a CNN includes an input layer, multiple convolutional layers, pooling layers, and fully connected layers.
  • the convolutional layers are the core of the CNN, used to extract local features from the image.
  • the pooling layers simplify the data and reduce computational complexity.
  • the fully connected layers integrate the features extracted by the convolutional and pooling layers, outputting a feature vector.
  • the text generation component maps feature vectors to text sequences, which can be achieved using recurrent neural networks (RNNs), long short-term memory networks (LSTMs), or Transformers.
  • RNNs recurrent neural networks
  • LSTMs long short-term memory networks
  • Transformers recurrent neural networks
  • the difference between the first image description generation model and the second image description generation model is that the first image description generation model remains unchanged, while the initial state of the second image description generation model is the same as that of the first image description generation model. That is, before updating the second image description generation model using the method provided in this application embodiment, the second image description generation model is the same as the first image description generation model. However, after updating the second image description generation model once or multiple times using the method provided in this application embodiment, the second image description generation model changes. Although both the first and second image description generation models are used to generate description text corresponding to images, the second image description generation model has better performance and the accuracy of the generated description text is higher.
  • the description correction model is used to correct the descriptive text corresponding to the image, removing erroneous words and inappropriate phrases to obtain a more accurate description. Additionally, the description correction model is also used to perform quality detection on the descriptive text corresponding to the image to obtain a quality score, which indicates the quality of the descriptive text.
  • the description correction model has been trained once or multiple times and already possesses the ability to correct and detect the quality of descriptive text.
  • the description correction model used in this embodiment is a multimodal large language model.
  • this embodiment updates the second image description generation model after training it to improve its performance.
  • the second image description generation model can be trained and updated by a computer device, or it can be trained by another device, sent to the computer device, and then updated by the computer device.
  • the computer device uses the first image description generation model as a reference model for the second image description generation model, and uses the description text generated by the first image description generation model as reference description text. These two models are used together to update the second image description generation model.
  • the first image description generation model can also be referred to as the reference image description generation model; the first description text generated by the first image description generation model can also be referred to as the reference description text.
  • the first image can be any image from which descriptive text needs to be generated, such as an image input onto a computer device, an image sent to the computer device from another device (such as a user's terminal or a technician's terminal), or an image obtained through other means.
  • the computer device generates first descriptive text corresponding to the first image (also called a sample image) using a first image description generation model.
  • the first descriptive text is used to describe the first image, such as describing the theme, background, style, composition, culture, and emotion of the first image.
  • the quality of the first descriptive text reflects the performance of the first image description generation model.
  • a first corrected descriptive text corresponding to the first descriptive text is obtained through a description correction model. Since the first corrected descriptive text has been corrected by the description correction model, its accuracy can be considered higher than that of the first descriptive text.
  • the first descriptive text can also be applied to other tasks besides updating the first image description generation model, such as semantic analysis tasks, text classification tasks, and dialogue response tasks.
  • This application embodiment does not limit this.
  • the computer device after the computer device generates the first descriptive text based on the image input by the user, it performs semantic analysis on the first descriptive text, generates response text based on the semantics of the first descriptive text, and feeds the response text back to the user, thereby achieving the effect of a dialogue between the computer device and the user's image.
  • the computer device generates a second descriptive text corresponding to a first image through a second image description generation model, obtains a quality score of the second descriptive text through a description correction model, and corrects the second descriptive text through the description correction model to obtain a second corrected descriptive text.
  • the computer device will also generate a second descriptive text corresponding to the first image through a second image description generation model, and obtain the quality score of the second descriptive text and the second corrected descriptive text corresponding to the second descriptive text through a description correction model.
  • the second descriptive text is used to describe the first image, such as its subject, background, style, composition, culture, and emotion.
  • the quality of the second descriptive text reflects the performance of the second image description generation model.
  • a second corrected descriptive text is obtained through a description correction model.
  • the second description text was "A man wearing a green shirt and gray pants”
  • the revised second description text was "A man wearing a green shirt and brown pants, with green prints and English on the front of the shirt”.
  • the color of the pants was corrected to the correct color, and more detailed descriptions of the shirt were added.
  • the first and second image description generation models are identical, so the accuracy of the second description text can be considered equal to that of the first. Subsequently, since the first image description generation model remains unchanged while the second image description generation model undergoes one or more updates, its performance gradually improves. Therefore, the accuracy of the second description text can be considered higher than that of the first. Because the second corrected description text has already been corrected by the description correction model, its accuracy can be considered higher than that of the first corrected description text, and it is also highly likely to be higher than that of the first corrected description text.
  • the quality score of the second descriptive text indicates its quality and, to some extent, the performance of the second image description generation model. In some embodiments, a higher quality score indicates that the second descriptive text accurately and comprehensively describes the first image, while a lower quality score indicates that the second descriptive text fails to accurately and comprehensively describe the first image.
  • the quality score can be expressed in numerical or alphabetical form.
  • the quality score can be represented as a value of 1, 2, 3, etc., with the level of the quality score determined by the value.
  • the quality score can be represented as a grade A, grade B, etc., with the level of the quality score determined by the grade.
  • the computer device determines a loss value based on the first descriptive text, the first corrected descriptive text, the second descriptive text, and the second corrected descriptive text, and updates the second image description generation model with the goal of reducing the loss value.
  • the quality threshold refers to the conditions that a high-quality descriptive text should meet in terms of quality score. These quality conditions can be set by default by computer equipment or by technicians according to their needs.
  • the descriptive text is divided according to quality conditions. For descriptive text that meets the quality conditions and descriptive text that does not meet the quality conditions, different methods are used to update the second image description generation model.
  • This embodiment uses only the first image as an example. After generating the second descriptive text corresponding to the first image through the second image description generation model and obtaining the quality score of the second descriptive text through the description correction model, it is determined whether the quality score of the second descriptive text meets the quality conditions. If the quality score of the second descriptive text does not meet the quality conditions, it indicates that the second descriptive text is low-quality descriptive text. In this case, if the first image and the second descriptive text are directly used to update the second image description generation model, the second image description generation model will learn incorrect knowledge, which will affect the performance of the second image description generation model. On the other hand, if the second descriptive text is directly filtered out, training samples will be wasted, reducing the number of samples.
  • the computer device obtains the four descriptive texts corresponding to the first image: the first descriptive text, the first corrected descriptive text, the second descriptive text, and the second corrected descriptive text, if the quality parameters of the second descriptive text do not meet the quality conditions, it can take all four descriptive texts into account and update the second image description generation model using these four descriptive texts.
  • the update goal of the second image description generation model is to increase the probability of obtaining the second corrected description text through the updated second image description generation model and the description correction model, that is, to generate more accurate description text.
  • the quality score of the second description text does not meet the quality condition, as long as the second image description generation model is updated according to the above update goal, it can still be guaranteed that the performance of the updated second image description generation model is higher than that of the unupdated second image description generation model.
  • step 203 includes: determining a first probability that the first image description generation model outputs the first description text, a second probability that the description correction model outputs the first corrected description text, a third probability that the second image description generation model outputs the second description text, and a fourth probability that the description correction model outputs the second corrected description text. Based on the first probability, second probability, third probability, and fourth probability, a loss value is determined, wherein the loss value is positively correlated with the logarithm of the ratio of the third probability to the first probability and negatively correlated with the logarithm of the ratio of the fourth probability to the second probability.
  • the second image description generation model is updated with the goal of reducing the loss value.
  • the logarithm of the ratio of the third probability to the first probability represents the difference between the logarithm of the third probability and the logarithm of the first probability
  • the logarithm of the ratio of the fourth probability to the second probability represents the difference between the logarithm of the fourth probability and the logarithm of the second probability
  • the update objective of the second image description generation model is to reduce the loss value.
  • a smaller loss value results in a smaller difference between the logarithm of the third probability and the logarithm of the first probability. This indicates a smaller deviation between the probability of the second image description generation model generating the second descriptive text and the probability of the first image description generation model generating the first descriptive text.
  • the probability of outputting both the first and second descriptive text decreases, and since the quality score of the second descriptive text does not meet the quality requirements, the probability of outputting the second descriptive text decreases even more.
  • a smaller loss value results in a larger difference between the logarithm of the fourth probability and the logarithm of the second probability.
  • the loss value, the first probability, the second probability, the third probability, and the fourth probability satisfy the following relationship:
  • L( ⁇ ; ⁇ ref ) represents the loss value
  • E represents the expectation
  • ⁇ refers to the second image description generation model
  • ref refers to the first image description generation model
  • x refers to the image
  • yl refers to the description text generated by the image description generation model
  • yw refers to the description text corrected by the description correction model
  • ⁇ represents the hyperparameter
  • ⁇ represents the sigmoid function (S-shaped function). Therefore, ⁇ ( yw /x) represents the fourth probability, ⁇ ref ( yw /x) represents the second probability, ⁇ ( yl /x) represents the third probability, and ⁇ ref ( yl /x) represents the first probability.
  • the method may further include:
  • the computer device updates the second image description generation model based on the first image and the second descriptive text.
  • the first image and the second descriptive text can be used to update the second image description generation model, thereby increasing the number of samples and enabling the second image description generation model to learn the correct knowledge contained in the second descriptive text, thus effectively improving the performance of the second image description generation model.
  • the first image and the second descriptive text constitute a set of samples and are added to the training set of the second image description generation model. Then, based on one or more sets of samples in the training set, the second image description generation model is updated. Since the training set only stores high-quality descriptive text, updating the second image description generation model based on high-quality descriptive text allows the second image description generation model to learn the correct knowledge, thereby effectively improving performance.
  • image description generation models are typically trained based on training samples. After training, these models can generate descriptive text corresponding to any image.
  • the training samples include images and their corresponding descriptive texts.
  • the accuracy of the descriptive text generated by these models is insufficient, and they suffer from a serious "illusion" problem—the descriptive text may contain incorrect descriptive information. Furthermore, they cannot achieve self-learning updates.
  • the method provided in this application by means of a first image description generation model and a description correction model, can use the second description text, along with the second corrected description text, the first description text, and the first corrected description text to update the second image description generation model even when the quality of the second description text generated by the second image description generation model does not meet the quality conditions. This improves the performance of the second image description generation model, saves the number of samples, and avoids wasting low-quality description text.
  • the quality score of the second descriptive text generated by the second image description generation model meets the quality condition, it indicates that the second descriptive text is a high-quality descriptive text.
  • the high-quality second descriptive text and the first image corresponding to the second descriptive text are used to update the second image description generation model, thereby enabling the second image description generation model to achieve self-learning update and improving the performance of the second image description generation model.
  • the second image description generation model can perform self-learning updates based on newly processed images, continuously iterating and updating the performance of the second image description generation model to adapt to the needs of continuous image processing in real-time scenarios.
  • the following describes the method for determining whether the quality score of the second descriptive text meets the quality conditions.
  • the quality condition includes a quality score threshold, and if the quality score of the second descriptive text is less than or equal to the quality score threshold, it is determined that the second descriptive text does not meet the quality condition.
  • the quality score threshold is a threshold used to distinguish between high-quality and low-quality descriptive text. For example, if the quality score range is [0, 1], the quality score threshold may be 0.5 or 0.8, or if the quality score range is [0, 10], the quality score threshold may be 5 or 8, or the quality score may belong to the set ⁇ A, B, C, D, E ⁇ , with the level increasing from A to E, and the quality score threshold may be C. In other words, descriptive text with a quality score of A, B, or C is considered low-quality descriptive text, while descriptive text with a quality score of D or E is considered high-quality descriptive text.
  • the quality score threshold can be set by computer equipment or by technicians as needed. Alternatively, if the computer equipment receives multiple descriptive texts, it can determine the quality score threshold based on the quality scores of the multiple descriptive texts. For example, the quality score threshold could be equal to the average of the quality scores of the multiple descriptive texts, or it could be equal to the product of the sum of the quality scores of the multiple descriptive texts and a preset ratio.
  • the quality score of the second descriptive text is greater than the quality score threshold, it indicates that the second descriptive text is high-quality and meets the quality condition. In another embodiment, if the quality score of the second descriptive text is not greater than the quality score threshold, it indicates that the second descriptive text is low-quality and does not meet the quality condition.
  • descriptive text with a quality score greater than a quality score threshold is selected as high-quality descriptive text
  • descriptive text with a quality score not greater than a quality score threshold is selected as low-quality descriptive text.
  • the second image description generation model has the ability to generate multi-dimensional descriptive text.
  • the second descriptive text generated by the second image description generation model can describe the first image from N dimensions, where N is an integer greater than 1. These N dimensions include at least two of the following: theme, background, style, composition, culture, and emotion, and may also include other dimensions.
  • the description correction model has the ability to perform quality detection on the multi-dimensional descriptive text. Therefore, the quality score obtained by quality detection of the second descriptive text will include sub-scores for N dimensions. Each sub-score represents the quality of the second descriptive text in describing the first image in the dimension corresponding to that sub-score. Quality conditions include quality score thresholds for N dimensions, used to determine whether the quality of the second descriptive text meets the requirements from each of the N dimensions.
  • Subscores that are greater than the quality score threshold of their respective dimension are called high-quality subscores.
  • the method then further includes any of the following:
  • the second descriptive text is determined to meet the quality condition.
  • the preset number is set by the computer equipment by default or by technicians based on the value of N. If the number of high-quality sub-scores in the N dimensions of the second descriptive text is greater than the preset number, it means that the second descriptive text can accurately describe the first image from a sufficient number of dimensions, and therefore belongs to high-quality descriptive text, satisfying the quality condition. If the number of high-quality sub-scores in the N dimensions of the second descriptive text is not greater than the preset number, it means that the second descriptive text can accurately describe the first image from fewer dimensions, and therefore belongs to low-quality descriptive text, not satisfying the quality condition.
  • the preset ratio is set by the computer equipment by default or by technicians according to requirements. If the proportion of high-quality sub-scores in the N dimensions of the second descriptive text is greater than the preset ratio, it means that the second descriptive text can accurately describe the first image from a sufficient number of dimensions, and therefore belongs to high-quality descriptive text, meeting the quality condition. If the proportion of high-quality sub-scores in the N dimensions of the second descriptive text is not greater than the preset ratio, it means that the second descriptive text can accurately describe the first image from fewer dimensions, and therefore belongs to low-quality descriptive text, not meeting the quality condition.
  • the second descriptive text generated by the second image description generation model is multi-dimensional descriptive text
  • a filtering scheme suitable for multi-dimensional descriptive text is provided.
  • the sub-scores in the quality score of the second descriptive text that are greater than the quality score threshold of their respective dimensions are called high-quality sub-scores.
  • high-quality sub-scores Based on the number of high-quality sub-scores among the N dimensions of the second descriptive text, high-quality and low-quality descriptive texts are filtered out.
  • This embodiment provides an accurate scheme for filtering high-quality and low-quality multi-dimensional descriptive text, improving the accuracy of filtering and thus improving the performance of the second image description generation model.
  • the computer device can generate multiple second descriptive texts corresponding to multiple first images, thereby obtaining multiple second descriptive texts.
  • the quality scores of the multiple second descriptive texts are obtained.
  • the multiple second descriptive texts are filtered to distinguish high-quality second descriptive texts and their corresponding first images, as well as low-quality second descriptive texts and their corresponding first images, thereby updating the second image description generation model in different ways.
  • the computer device generates a second descriptive text corresponding to a first image each time, stores the first image and the second descriptive text as a set of samples in a sample set. As the number of samples gradually increases, after the number of samples in the sample set reaches a target number, a quality score for each second descriptive text in the sample set is obtained through a description correction model. Based on the quality scores of multiple second descriptive texts, the multiple second descriptive texts are divided.
  • the computer device generates a second descriptive text corresponding to a first image each time, stores the first image and the second descriptive text as a set of samples in a sample set, and at preset time intervals, for the samples collected in the sample set within the preset time interval, a quality score for the second descriptive text is obtained through a description correction model. Based on the quality scores of multiple second descriptive texts, the multiple second descriptive texts are divided.
  • the quality conditions include: a quality ranking threshold
  • the quality score of the second descriptive text is after the quality ranking threshold in the ranking of multiple second descriptive texts from high to low, it is determined that the second descriptive text does not meet the quality condition; wherein, the multiple second descriptive texts are descriptive texts generated by the second image description model and corresponding to multiple images respectively.
  • the method also includes:
  • the computer device determines that the second description texts not ranked in the top K positions do not meet the quality criteria, based on the quality scores of multiple second description texts in descending order.
  • K is a positive integer, and K is less than the number of multiple second description texts.
  • K can be set by the computer device by default, or by technicians according to their needs.
  • K is equal to the product of the number of multiple second description texts and a preset ratio, or K is equal to other values.
  • the computer device sorts the second descriptive texts according to their quality scores from highest to lowest. If any second descriptive text is among the top K in the ranking, it is considered a high-quality descriptive text and meets the quality criteria. Conversely, if it is not among the top K, it is considered a low-quality descriptive text and does not meet the quality criteria.
  • This application embodiment sorts multiple second descriptive texts according to their quality scores from highest to lowest.
  • the second descriptive texts ranked in the top K positions are considered high-quality descriptive texts, while those not ranked in the top K positions are considered low-quality descriptive texts.
  • This allows for different methods to be used to update the second image description generation model based on high-quality and low-quality descriptive texts, thereby improving flexibility.
  • This application embodiment provides a scheme for accurately classifying descriptive texts, improving the accuracy of the selection process, and thus enhancing the performance of the second image description generation model.
  • the quality score of the second descriptive text includes N sub-scores, each sub-score representing the quality of the second descriptive text in describing the first image in the dimension corresponding to that sub-score.
  • the method further includes any one of the following: The top K sub-scores in the same dimension of multiple second descriptive texts are referred to as high-quality sub-scores.
  • the second descriptive text is determined to meet the quality condition.
  • the preset number is set by the computer equipment by default or by technicians based on the value of N. If the number of high-quality sub-scores in the N dimensions of the second descriptive text is greater than the preset number, it means that the second descriptive text can accurately describe the first image from a sufficient number of dimensions, and therefore belongs to high-quality descriptive text, satisfying the quality condition. If the number of high-quality sub-scores in the N dimensions of the second descriptive text is not greater than the preset number, it means that the second descriptive text can accurately describe the first image from fewer dimensions, and therefore belongs to low-quality descriptive text, not satisfying the quality condition.
  • the preset ratio is set by the computer equipment by default or by technicians according to requirements. If the proportion of high-quality sub-scores in the N dimensions of the second descriptive text is greater than the preset ratio, it means that the second descriptive text can accurately describe the first image from a sufficient number of dimensions, and therefore belongs to high-quality descriptive text, meeting the quality condition. If the proportion of high-quality sub-scores in the N dimensions of the second descriptive text is not greater than the preset ratio, it means that the second descriptive text can accurately describe the first image from fewer dimensions, and therefore belongs to low-quality descriptive text, not meeting the quality condition.
  • computer devices can also use other methods to classify multiple second description texts by quality.
  • This application embodiment does not limit the classification method.
  • the second descriptive text generated by the second image description generation model is multi-dimensional descriptive text
  • a filtering scheme suitable for multi-dimensional descriptive text is provided.
  • the sub-scores of multiple second descriptive texts are sorted according to different dimensions.
  • the top K sub-scores in the same dimension of multiple second descriptive texts are called high-quality sub-scores.
  • high-quality and low-quality descriptive texts are filtered out.
  • This embodiment provides a scheme for accurately filtering high-quality multi-dimensional descriptive text, improving the accuracy of filtering and thus improving the performance of the second image description generation model.
  • the process of training the second image description generation model includes:
  • a description text corresponding to the first image is generated based on the current second image description generation model. Based on the error between the generated description text and the second description text, the model parameters of the second image description generation model are adjusted with the goal of reducing the error.
  • the performance of the second image description generation model can be improved, increasing the probability that the second image description generation model generates high-quality descriptive text.
  • the computer device after the computer device updates the second image description generation model, it can generate descriptive text corresponding to any image using the updated second image description generation model. Since the updated second image description generation model has stronger performance, the quality of the generated descriptive text is improved. Alternatively, the updated second image description generation model can be used to train the image generation model.
  • the image generation model is used to generate an image that matches any descriptive text. For example, if a user inputs descriptive text describing what they want the image to look like, the image generation model can generate an image that meets the user's requirements.
  • the process of training the image generation model includes: generating sample description text corresponding to the sample image through the updated second image description generation model, the sample description text being used to describe the sample image, and training the image generation model based on the sample image and the sample description text.
  • sample images the images used to train the image generation model are called sample images, and the descriptive text corresponding to the sample images is called sample descriptive text.
  • sample images may be the same as or different from the first image in the above embodiments.
  • the sample images may be sent to the computer device by other devices, stored in the computer device by technicians, or obtained by other means.
  • the sample description text generated by the second image description generation model is of high quality and can accurately and comprehensively describe the sample image. Therefore, using the sample image as the reference sample and the sample description text as the input sample to train the image generation model can effectively improve the performance of the image generation model, enabling the image generation model to generate images that are more consistent with the input description text.
  • training an image generation model based on sample images and sample descriptive text includes: generating an image corresponding to the sample descriptive text using the image generation model; and adjusting the model parameters of the image generation model based on the error between the generated image and the sample image, with the goal of reducing the error.
  • the performance of the image generation model can be improved, enabling the image generation model to generate high-quality images.
  • the training samples used to train image generation models in related technologies may contain a lot of noise, resulting in poor performance of the trained image generation models.
  • This application also provides a scheme for training an image generation model, which can generate descriptive text corresponding to an image using a second image description generation model, thereby training the image generation model based on the image and its corresponding descriptive text. Since the current second image description generation model has been trained and updated and has strong performance, the descriptive text generated by the second image description generation model is of high quality and can accurately and comprehensively describe the image. Therefore, it can effectively improve the performance of the image generation model, enabling it to generate images that better match the input descriptive text. Furthermore, it eliminates the need for manual determination of the descriptive text corresponding to the image, saving labor costs and improving training efficiency.
  • the second image description generation model implements self-learning updates and is suitable for real-time scenarios, it can not only continuously improve the performance of the second image description generation model, but also provide more high-quality descriptive text for the image generation model, enriching the sample quantity of the image generation model, thereby effectively improving the performance of the image generation model.
  • FIG 4 is a flowchart of another image description generation model processing method provided by this application.
  • This application is executed by a computer device, such as a terminal 101 or server 102 as shown in Figure 1.
  • the first image description generation model and the second image description generation model are obtained by training an image description generation model to be trained.
  • This application describes the process of training the image description generation model to obtain the first image description generation model and the second image description generation model. Referring to Figure 4, the method includes:
  • generating the combined descriptive text corresponding to the second image includes:
  • Multiple third image description generation models are used to generate description text for the second image in multiple dimensions; wherein, the description text for each dimension is used to describe the second image from that dimension; each third image description generation model is used to generate description text for the second image in one dimension.
  • the image used to train the first image description generation model is referred to as the second image.
  • the second image may be different from the first image in the above embodiments.
  • the second image may be sent to the computer device by other devices, stored in the computer device by technicians, or obtained by other means.
  • step 401 is similar to step 201 above.
  • the first and third image descriptive generation models are different descriptive generation models.
  • the first and third image descriptive generation models may have the same model architecture but use different training samples during training; or, the first and third image descriptive generation models may have different model architectures and use both the same and different training samples during training.
  • multiple third-party image description generation models can be the same model or different models.
  • multiple third-party image description generation models may have the same model architecture but use different training samples during training, which may result in different description texts generated by multiple third-party image description generation models for the same image.
  • the generation results of multiple third image description generation models can be used as training samples to train the first image description generation model.
  • the third image description generation model used in this application is a multimodal large language model.
  • a multimodal large language model is an artificial intelligence model that can process various types of data input, such as text, images, and audio, and can generate natural language output. Therefore, it can be used to generate descriptive text corresponding to images.
  • multiple third image description generation models are stored on a computer device, and the computer device can generate description text by calling the third image description generation models.
  • multiple third image description generation models are stored on other devices, wherein different third image description generation models can be stored on the same device or different devices.
  • the computer device sends a request to the device storing the third image description generation models to generate description text, the request carrying a second image.
  • the device storing the third image description generation models then generates description text corresponding to the second image based on the third image description generation models and sends the third description text to the computer device.
  • each third image description generation model has one or more target dimensions for generating descriptive text belonging to the target dimensions.
  • multiple dimensions include at least two of the following: theme, background, style, composition, culture, and emotion. Other dimensions may also be included.
  • the target dimensions of different third image description generation models may include the same dimension or different dimensions. If multiple third image description generation models include third image description generation models with different target dimensions, then the multiple third image description generation models correspond to a total of N dimensions, where N is an integer greater than 1.
  • the training samples used during training include images and descriptive text belonging to the target dimension, but not descriptive text belonging to other dimensions. This can effectively reduce the complexity of the descriptive text and the amount of information in the training samples, thereby improving the training speed.
  • the third image description generation model trained based on the above training samples has the ability to generate descriptive text belonging to the target dimension based on the image, thereby improving the targeting and being able to better describe the image from the target dimension.
  • the dimensions involved in the generated descriptive text are not comprehensive enough.
  • the first major language model is an artificial intelligence model designed to understand and generate human language. It requires training on a large amount of text to ensure accurate task performance, including text summarization, translation, and sentiment analysis.
  • the first major language model may include GPT (Generative Pre-Trained Transformer), GLM (Generalized Linear Models), CPM (Chinese Pre-trained Models), and mixed-model approaches, but this application does not limit the specific first major language model used.
  • multiple third image description generation models generate a total of N-dimensional description texts, where N is an integer greater than 1, the description texts generated by the multiple third image description generation models are combined through the first large language model.
  • the multiple description texts complement and correct each other, thereby generating more accurate combined description texts.
  • This can provide multi-dimensional training samples for the first image description generation model, thereby not only improving the performance of the first image description generation model, but also improving its comprehensiveness. This enables the first image description generation model to generate multi-dimensional description texts for the input image, providing a more comprehensive description of the image.
  • the first language model can still perform semantic analysis on the description text generated by each third image description generation model.
  • it can consider whether the dimensions of different description texts are the same, thereby removing the description content of repeated dimensions in multiple description texts and retaining the description content of non-repeating dimensions, so that the resulting combined description text can describe the second image from multiple dimensions without having duplicate content.
  • Figure 5 is a flowchart of a multi-dimensional descriptive text combination provided in an embodiment of this application.
  • the second image is processed by multiple third image description generation models to obtain multi-dimensional descriptive text, which is then combined by the first major language model.
  • the resulting combined descriptive text can describe the second image from multiple dimensions, including: theme, background, style, composition, culture, and emotion.
  • the computer device uses the second image and the combined descriptive text as training sample pairs to train the image description generation model to be trained.
  • the combined descriptive text is obtained by integrating the generation results of multiple third image description generation models, it can be considered as a more comprehensive and higher-quality descriptive text. Therefore, training the first image description generation model based on the second image and the combined descriptive text can effectively improve the performance and comprehensiveness of the first image description generation model.
  • the method before training the first image description generation model based on the second image and the combined description text, the method further includes: displaying the second image and the combined description text, and obtaining the corrected combined description text based on the correction operation performed on the combined description text.
  • the computer device can display a second image and combined descriptive text on a local display screen, allowing the user to check the accuracy and completeness of the combined descriptive text, and perform correction operations, such as adding new text content, deleting text content, or modifying text content in the combined descriptive text. After correction, the computer device can obtain the corrected combined descriptive text.
  • the computer device can send a second image and combined descriptive text to the operator's device, thereby displaying the second image and combined descriptive text on the operator's device, performing correction operations on the combined descriptive text, such as adding new text content to the combined descriptive text, deleting text content from the combined descriptive text, or modifying the text content in the combined descriptive text.
  • correction operations such as adding new text content to the combined descriptive text, deleting text content from the combined descriptive text, or modifying the text content in the combined descriptive text.
  • the operator can obtain the corrected combined descriptive text and send it to the computer device.
  • the process of training the first image description generation model based on the combined descriptive text shown in Figure 5 is shown in Figure 6.
  • performing correction operations on the combined descriptive text will consume some manpower, since the combined descriptive text has already been generated, corrections only need to be made based on the combined descriptive text, so the manpower cost will not be too high.
  • it can improve the accuracy of the combined descriptive text, thereby ensuring that the subsequent training of the first image description generation model based on the second image and the combined descriptive text can effectively improve the performance of the first image description generation model.
  • the training process shown in the embodiments of this application can be performed before the update process in the embodiment shown in FIG2 above. That is, after training the first image description generation model using the embodiments of this application, the second image description generation model is updated using the embodiment shown in FIG2. Alternatively, the training process shown in the embodiments of this application can be performed concurrently with the update process in the embodiment shown in FIG2 above. For example, after training the first image description generation model using the embodiments of this application, the second image description generation model is updated using the embodiment shown in FIG2. Afterwards, the first image description generation model can be trained again using the embodiments of this application. The embodiments of this application do not limit the time for training the first image description generation model and updating the second image description generation model.
  • the computer device copies the trained description generation model to obtain the first image description generation model and the second image description generation model.
  • the trained image description generation model can be used as the first image description generation model. Then, the first image description generation model is copied to obtain the second image description generation model. Therefore, the initial states of the first and second image description generation models are the same. Subsequently, the second image description generation model will be updated while keeping the first image description generation model unchanged.
  • the example of obtaining a second image description generation model by copying the first image description generation model can be used as an example.
  • the trained image description generation model can also be used as the second image description generation model, and then the second image description generation model can be copied to obtain the first image description generation model. That is, after training an image description generation model and copying it to obtain another image description generation model, either of the two image description generation models can be used as the first image description generation model and kept unchanged, while the other can be used as the second image description generation model, and the update method in the above embodiment can be used for updating.
  • image description generation models are typically trained based on training samples. After training, these models can generate descriptive text for any image.
  • the training samples include images and their corresponding descriptive text.
  • image description generation models are limited by the dimensionality of the descriptive text in the training samples, resulting in generated descriptive text that is single-dimensional and cannot cover multiple dimensions, thus failing to comprehensively describe the image.
  • This application provides a scheme for training an image description generation model. It can generate multi-dimensional descriptive text corresponding to a second image using multiple third image description generation models. Then, a first language model is used to complement and correct these multi-dimensional descriptive texts, resulting in a multi-dimensional combined descriptive text.
  • This combined descriptive text can describe the second image from multiple dimensions, providing a more comprehensive and higher-quality description. For example, it can cover multiple dimensions such as theme, background, style, composition, culture, and emotion. Therefore, training a first image description generation model based on the second image and the combined descriptive text can effectively improve its performance and comprehensiveness, enabling it to generate multi-dimensional descriptive text for input images, providing a more accurate and comprehensive description. Furthermore, it eliminates the need for manual determination of the descriptive text, saving labor costs and improving training efficiency.
  • image description generation models can generate multi-dimensional descriptive text for input images, providing a more accurate and comprehensive description of the images, they can also provide more multi-dimensional descriptive text for image generation models. Therefore, training image generation models based on the descriptive text generated by image description generation models can effectively improve the performance of image generation models.
  • the method further includes a process of training a description correction model based on the combined description text, i.e., the method also includes steps 404 and 405:
  • Step 404 The computer device obtains the quality score for the combined descriptive text annotation.
  • the quality score includes N sub-scores in N dimensions, where N is an integer greater than 1.
  • Each sub-score represents the quality of the combined descriptive text in describing the second image in the dimension corresponding to that sub-score.
  • the quality score can be manually labeled, with labelers determining the quality score of the combined descriptive text by performing quality checks on the combined descriptive text. Since the combined descriptive text describes the second image from N dimensions, labelers can perform quality checks on the descriptive text for each dimension separately, thus obtaining sub-scores for N dimensions. Then, based on the combined descriptive text and its quality score, a description correction model is trained, enabling the model to perform quality checks on the descriptive text for each dimension of the image.
  • the combined description text is manually scored to obtain scores in multiple dimensions, and the scoring model is trained based on the combined description text and the scores in multiple dimensions.
  • the process of training the description correction model in step 404 and the process of training the image description generation model in step 402 can be executed in parallel or one after the other.
  • the training time of the description correction model and the first image description generation model is not limited.
  • Step 405 The computer device trains a description correction model based on the combined description text and the quality score of the combined description text.
  • This application also provides a scheme for training a description correction model. It can generate multi-dimensional descriptive text corresponding to a second image using multiple third-image description generation models, and then use a first large language model to complement and correct these multi-dimensional descriptive texts, thereby combining them to obtain a multi-dimensional combined descriptive text.
  • This combined descriptive text can describe the second image from multiple dimensions, resulting in a more comprehensive and higher-quality description. Therefore, training the description correction model based on the combined descriptive text, the quality scores labeled for the combined descriptive text, and the corrected combined descriptive text can effectively improve the performance and comprehensiveness of quality detection.
  • This allows the description correction model to determine sub-scores for multiple dimensions of the multi-dimensional descriptive text, enabling more accurate and comprehensive quality detection and more precise correction of the descriptive text. Furthermore, it eliminates the need for manual determination of the descriptive text, saving labor costs and improving training efficiency.
  • FIG 8 is a flowchart of another image description generation model processing method provided by this application.
  • This application is executed by a computer device, such as a terminal 101 or server 102 as shown in Figure 1.
  • training the image description generation model before step 401, further includes at least one of a first training stage and a second training stage, wherein: the first training stage is used to train the visual model and the connector; and the second training stage is used to train the first large language model and the connector.
  • This application embodiment illustrates the process of training the image description generation model.
  • the method includes:
  • a computer device generates an image description generation model to be trained, the image description generation model including a visual model, a second major language model, and a connector located between the visual model and the second major language model.
  • the architecture of the image description generation model is shown in Figure 9.
  • the visual model is a model for processing visual information, which can be in the form of images or videos.
  • This visual model can include the Transformer model (a neural network model based on a self-attention mechanism), the ViT model, etc.
  • the second major language model is an artificial intelligence model designed to understand and generate human language. This second major language model needs to be trained on a large amount of text to ensure that the first major language model can accurately perform tasks, including text summarization, translation, sentiment analysis, etc.
  • the second major language model can include GPT, GLM, CPM, and mixed-model approaches, etc. This application embodiment does not limit the second major language model.
  • This connector links the visual model and the second language model, converting the first feature space to which the features output by the visual model belong to the second feature space to which the features input by the second language model belong to the second feature space. This enables the second language model to recognize the features output by the visual model, i.e., aligning the visual model with the second language model.
  • the model parameters in the visual model, connector, and second major language model can be initialized, and then steps 802 and 803 below are executed to train the image description generation model.
  • the visual model and the second major language model acquired by the computer device have already been trained, that is, the model parameters of the visual model and the second major language model have already been trained.
  • the computer device then combines the visual model, the second major language model, and the connector to obtain the image description generation model.
  • the model parameters of any unit (including the visual model, connector, and second major language model) in the image description generation model include the weights or biases in that unit, or other types of model parameters.
  • the computer device trains a visual model and a connector based on a first training sample while keeping the model parameters of the second language model unchanged.
  • the first training sample includes a third image and the third descriptive text corresponding to the third image.
  • Keeping the model parameters of the second largest language model unchanged includes freezing the model parameters of the second largest language model. After step 802 is completed, the model parameters of the second largest language model are then unfrozen.
  • the image used in training the image description generation model at this stage is referred to as the third image.
  • This third image may be different from the first image in the above embodiments, and may be the same as or different from the second image in the above embodiments.
  • the third image may be sent to a computer device by another device, stored in a computer device by a technician, or obtained through other means.
  • the descriptive text corresponding to the third image is referred to as the third descriptive text.
  • the third descriptive text may be determined manually or through the third descriptive generation model; this embodiment does not limit this.
  • training the visual model and connector can align the first feature space to which the features output by the visual model belong with the second feature space to which the features of the second language model belong. This allows the connector to convert the features output by the visual model that belong to the first feature space into features that belong to the second feature space and input them into the second language model. In this way, the second language model can recognize the features that belong to the second feature space and process them.
  • the computer device trains a second large language model and connector based on the second training samples while keeping the model parameters of the visual model unchanged.
  • the second training samples include the fourth image and the dialogue text for the fourth image.
  • Keeping the model parameters of the visual model unchanged includes freezing the model parameters of the visual model. After step 803 is completed, the model parameters of the visual model are then unfrozen.
  • the image used to train the image description generation model at this stage is referred to as the fourth image.
  • This fourth image may be different from the first image in the above embodiments, and may be the same as or different from the second image and the third image in step 802.
  • the fourth image can be sent to a computer device by another device, stored in a computer device by a technician, or obtained in other ways.
  • the fourth image corresponds to dialogue text used for dialogue about the fourth image.
  • the dialogue text includes at least two dialogue statements, each related to the fourth image.
  • This dialogue text also belongs to descriptive text used to describe the fourth image. Compared to ordinary descriptive text, the descriptive text for the fourth image has a contextual relationship between the at least two dialogue statements and contains more information.
  • the dialogue text for the fourth image includes two dialogue statements: the first dialogue statement is the question posed to the fourth image, "What is in this image?", and the second dialogue statement is the answer to the question, "There are two animals in this image.”
  • the dialogue text for the fourth image can be determined manually or obtained by crawling from the Internet.
  • the dialogue text for the fourth image can be composed of comments on the fourth image and replies to those comments. This application embodiment does not limit this.
  • steps 802 and 803 can be considered as two training stages of the image description generation model. These two training stages can be executed sequentially.
  • This embodiment of the application takes step 802 being executed before step 803 as an example; in another embodiment, step 803 can also be executed before step 802.
  • steps 802 and 803 are optional solutions; that is, in this embodiment, only step 802 can be executed without step 803, or only step 803 can be executed without step 802. This embodiment of the application does not limit the training process of the image description generation model.
  • This application provides another scheme for training an image description generation model, which can perform training in stages.
  • a visual model and connector are trained based on a third image and the corresponding third descriptive text, aligning the visual model with the second language model.
  • the second language model and connector are trained based on a fourth image and the dialogue text for the fourth image, which can enhance the understanding ability of the second language model.
  • the description correction model has the same architecture as the image description generation model, and the description correction model can also be trained using the embodiment shown in Figure 8.
  • the training samples for the image description generation model include images and the corresponding descriptive text or dialogue text
  • the training samples for the description correction model include the descriptive text corresponding to the image, the quality score of the descriptive text, and the corrected descriptive text.
  • This application embodiment does not limit the training process of the description correction model.
  • training method provided in this application embodiment can be combined with the embodiments shown in Figures 2 and 4 above, so as to train the image description generation model in multiple ways, thereby more comprehensively improving the performance of the image description generation model.
  • step 802 is taken as the first training stage, step 803 as the second training stage, steps 401-402 and 404-405 as the third training stage, and steps 201-204 as the update stage following the third training stage.
  • the trained image description generation model is obtained as the first image description generation model, and the second image description generation model and the trained description correction model are copied.
  • the update stage is executed based on the first image, the first image description generation model, the second image description generation model, and the description correction model.
  • step 802 is taken as the first training phase
  • step 803 as the second training phase
  • steps 401-402 and 404-405 as the third training phase.
  • steps 201-204 are also included as part of the third training phase, meaning the third training phase includes steps 401-405 and 201-204.
  • a trained image description generation model is obtained.
  • the third training phase is executed.
  • the image description generation model and the description correction model are trained.
  • the trained image description generation model is used as the first image description generation model, and the second image description generation model and the description correction model are copied to obtain them.
  • the second image description generation model is updated.
  • This application embodiment integrates multi-dimensional descriptive text generated by multiple third-party description generation models using a first major language model to obtain multi-dimensional combined descriptive text. Multi-dimensional quality scores of the combined descriptive text are then obtained through manual annotation. A multi-dimensional description correction model is trained using these annotated quality scores, while an image description generation model is trained using the multi-dimensional combined descriptive text. The description correction model performs quality checks and filtering on the results of the image description generation model, thereby updating the performance of the image description generation model and achieving multi-dimensional descriptive text generation and self-learning of the image description generation model.
  • Figure 12 is a schematic diagram of a model processing device provided in an embodiment of this application. Referring to Figure 12, the device includes:
  • the first description generation module 1201 is used to generate a first description text corresponding to a first image through a first image description generation model, and to correct the first description text through a description correction model to obtain a first corrected description text.
  • the second description generation module 1202 is used to generate a second description text corresponding to the first image through a second image description generation model, obtain a quality score of the second description text through a description correction model, and correct the second description text through the description correction model to obtain a second corrected description text.
  • the model update module 1203 is used to determine a loss value based on the first description text, the first corrected description text, the second description text, and the second corrected description text when the quality score of the second description text does not meet the quality conditions.
  • the second image description generation model is updated with the goal of reducing the loss value.
  • the updated second image description generation model is used to generate text describing the input image based on the input image.
  • the second image description generation model before the update has the same structure and state as the first image description generation model.
  • the model update module 1203 includes:
  • the first update unit 1213 is used to determine a first probability that the first image description generation model outputs the first description text, a second probability that the description correction model outputs the first corrected description text, a third probability that the second image description generation model outputs the second description text, and a fourth probability that the description correction model outputs the second corrected description text; determine a loss value based on the first probability, the second probability, the third probability, and the fourth probability, wherein the loss value is positively correlated with the logarithm of the ratio of the third probability to the first probability and negatively correlated with the logarithm of the ratio of the fourth probability to the second probability; and update the second image description generation model with the goal of reducing the loss value.
  • the model update module 1203 includes:
  • the second update unit 1223 is used to update the second image description generation model based on the first image and the second description text, provided that the quality score of the second description text meets the quality conditions.
  • model processing apparatus provided in the above embodiments is only an example of the division of the above functional modules.
  • the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above.
  • the model processing apparatus and the model processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.
  • This application also provides a computer device, which includes a processor and a memory.
  • the memory stores at least one computer program, which is loaded and executed by the processor to perform the operations performed in the model processing method of the above embodiments.
  • the computer device is provided as a terminal.
  • Figure 14 shows a schematic diagram of the structure of a terminal 1400 provided in an exemplary embodiment of this application.
  • Terminal 1400 includes a processor 1401 and a memory 1402.
  • Processor 1401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc.
  • Processor 1401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array).
  • Processor 1401 may also include a main processor and a coprocessor.
  • the main processor also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.
  • processor 1401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen.
  • processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.
  • AI Artificial Intelligence
  • Memory 1402 may include one or more computer-readable storage media, which may be non-transitory. Memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in memory 1402 are used to store at least one computer program, which is used by processor 1401 to implement the model processing method provided in the method embodiments of this application.
  • the terminal 1400 may further include a peripheral device interface 1403 and at least one peripheral device.
  • the processor 1401, memory 1402, and peripheral device interface 1403 can be connected via a bus or signal line.
  • Each peripheral device can be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board.
  • the peripheral device includes at least one of a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, and a power supply 1407.
  • Peripheral device interface 1403 can be used to connect at least one I/O (Input/Output) related peripheral device to processor 1401 and memory 1402.
  • processor 1401, memory 1402 and peripheral device interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1401, memory 1402 and peripheral device interface 1403 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.
  • FIG14 does not constitute a limitation on the terminal 1400, and may include more or fewer components than shown, or combine certain components, or employ different component arrangements.
  • the computer device is provided as a server.
  • Figure 15 is a schematic diagram of the structure of a server provided in an embodiment of this application.
  • the server 1500 can vary considerably due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 1501 and one or more memories 1502.
  • the memories 1502 store at least one computer program, which is loaded and executed by the processor 1501 to implement the methods provided in the various method embodiments described above.
  • the server may also have wired or wireless network interfaces, a keyboard, and input/output interfaces for input and output.
  • the server may also include other components for implementing device functions, which will not be elaborated here.
  • This application also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to implement the operations performed by the model processing method of the above embodiments.
  • This application also provides a computer program product, including a computer program loaded and executed by a processor to perform the operations performed by the model processing method of the above embodiments.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Processing Or Creating Images (AREA)

Abstract

本申请实施例公开了一种图像描述生成模型处理方法、装置、计算机设备及存储介质，属于计算机技术领域。该方法包括：通过第一图像描述生成模型生成第一图像对应的第一描述文本，并通过描述修正模型获取第一修正描述文本；通过第二图像描述生成模型生成第一图像对应的第二描述文本，并通过描述修正模型获取第二描述文本的质量分数以及第二修正描述文本；在第二描述文本的质量分数不满足质量条件的情况下，基于第一描述文本、第一修正描述文本、第二描述文本和第二修正描述文本，确定损失值，以减小所述损失值为目标，更新第二图像描述生成模型。

Description

图像描述生成模型处理方法、装置、计算机设备及存储介质
本申请要求于2024年6月5日提交中国专利局、申请号为202410729166.0，发明名称为“模型处理方法、装置、计算机设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及计算机技术领域，特别涉及一种图像描述生成模型处理方法、装置、计算机设备及存储介质。
背景技术
图像描述生成技术是指给定一个图像，生成描述该图像的文本的技术，随着计算机视觉技术和自然语言处理技术的发展，图像描述生成技术取得了广泛的应用。
相关技术中，通常会基于训练样本训练图像描述生成模型，训练之后即可通过该图像描述生成模型生成任一图像对应的描述文本。其中，训练样本包括图像和图像对应的描述文本。
技术内容
本申请实施例提供了一种图像描述生成模型处理方法、装置、计算机设备及存储介质。所述技术方案如下：
本申请一些实施例提供了一种图像描述生成模型处理方法，所述方法包括：
通过第一图像描述生成模型生成第一图像对应的第一描述文本，并通过描述修正模型修正所述第一描述文本得到第一修正描述文本；
通过第二图像描述生成模型生成所述第一图像对应的第二描述文本，通过所述描述修正模型获取所述第二描述文本的质量分数，以及通过所述描述修正模型修正所述第二描述文本得到第二修正描述文本；
在所述第二描述文本的质量分数不满足质量条件的情况下，基于所述第一描述文本、所述第一修正描述文本、所述第二描述文本和所述第二修正描述文本，确定损失值，以减小所述损失值为目标，更新所述第二图像描述生成模型，所述更新后的第二图像描述生成模型用于根据输入图像生成描述该输入图像的文本，更新前的所述第二图像描述生成模型具有与所述第一图像描述生成模型相同的结构和状态。
本申请一些实施例还提供了一种图像描述生成模型处理装置，所述装置包括：
第一描述生成模块，用于通过第一图像描述生成模型生成第一图像对应的第一描述文本，并通过描述修正模型修正所述第一描述文本得到第一修正描述文本；
第二描述生成模块，用于通过第二图像描述生成模型生成所述第一图像对应的第二描述文本，通过所述描述修正模型获取所述第二描述文本的质量分数，以及通过所述描述修正模型修正所述第二描述文本得到第二修正描述文本；
模型更新模块，用于在所述第二描述文本的质量分数不满足质量条件的情况下，基于所述第一描述文本、所述第一修正描述文本、所述第二描述文本和所述第二修正描述文本，确定损失值，以减小所述损失值为目标，更新所述第二图像描述生成模型，所述更新后的第二图像描述生成模型用于根据输入图像生成描述该输入图像的文本，更新前的所述第二图像描述生成模型具有与所述第一图像描述生成模型相同的结构和状态。
本申请一些实施例还提供了一种计算机设备，所述计算机设备包括处理器和存储器，所述存储器中存储有至少一条计算机程序，所述至少一条计算机程序由所述处理器加载并执行，以实现如上述方面所述的模型处理方法所执行的操作。
本申请一些实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有至少一条计算机程序，所述至少一条计算机程序由处理器加载并执行，以实现如上述方面所述的模型处理方法所执行的操作。
本申请一些实施例还提供了一种计算机程序产品，包括计算机程序，所述计算机程序由处理器加载并执行，以实现如上述方面所述的模型处理方法所执行的操作。
附图简要说明
图1是本申请实施例提供的一种实施环境的示意图；
图2是本申请实施例提供的一种图像描述生成模型处理方法的流程图；
图3是本申请实施例提供的一种第二图像描述生成模型的更新流程的示意图；
图4是本申请实施例提供的另一种图像描述生成模型处理方法的流程图；
图5是本申请实施例提供的一种组合多维度描述文本的流程示意图；
图6是本申请实施例提供的一种训练图像描述生成模型的流程示意图；
图7是本申请实施例提供的一种训练描述修正模型的流程示意图；
图8是本申请实施例提供的另一种图像描述生成模型处理方法的流程图；
图9是本申请实施例提供的一种图像描述生成模型的架构图；
图10是本申请实施例提供的一种训练阶段和更新阶段的示意图；
图11是本申请实施例提供的另一种训练阶段的示意图；
图12是本申请实施例提供的一种图像描述生成模型处理装置的结构示意图；
图13是本申请实施例提供的另一种图像描述生成模型处理装置的结构示意图；
图14是本申请实施例提供的一种终端的结构示意图；
图15是本申请实施例提供的一种服务器的结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合附图对本申请实施方式作进一步地详细描述。
可以理解，本申请所使用的术语“第一”、“第二”等可在本文中用于描述各种概念，但除非特别说明，这些概念不受这些术语限制。这些术语仅用于将一个概念与另一个概念区分。举例来说，在不脱离本申请的范围的情况下，可以将第一图像称为第二图像，且类似地，可将第二图像称为第一图像。
其中，至少两个是指两个或者两个以上，例如，至少两个图像可以是两个图像、三个图像等任一大于等于二的整数个图像。每个是指至少两个中的每一个，例如，每个图像是指至少两个图像中的每一个图像，若至少两个图像为3个图像，则每个图像是指3个图像中的每一个图像。
需要说明的是，本申请所涉及的信息(包括但不限于用户设备信息、用户个人信息等)、数据(包括但不限于用于分析的数据、存储的数据、展示的数据等)以及信号(包括但不限于用户终端与其他设备之间传输的信号等)，均为经用户或相关方面充分授权的，且相关数据的收集、使用和处理需要遵守相关国家和地区的相关法律法规和标准。
本申请实施例提供的方案涉及人工智能的计算机视觉和自然语言处理等技术，具体通过如下实施例进行说明：
本申请实施例提供的方法，用于计算机设备中。在一些实施例中，该计算机设备为终端或服务器。在一些实施例中，该终端是智能手机、平板电脑、笔记本电脑、台式计算机、智能音箱、智能手表、智能语音交互设备、智能家电、车载终端、飞行器等，但并不局限于此。在一些实施例中，该服务器是独立的物理服务器，或者，是多个物理服务器构成的服务器集群或者分布式系统，或者，是提供云服务、云数据库、云计算、云函数、云存储、网络服务、云通信、中间件服务、域名服务、安全服务、CDN(Content Delivery Network，内容分发网络)以及大数据和人工智能平台等基础云计算服务的云服务器。本申请实施例可应用于各种场景，包括但不限于云技术、人工智能、智慧交通、辅助驾驶等。
在一些实施例中，本申请实施例所涉及的计算机程序可被部署在一个计算机设备上执行，或者在位于一个地点的多个计算机设备上执行，又或者，在分布在多个地点且通过通信网络互连的多个计算机设备上执行，分布在多个地点且通过通信网络互连的多个计算机设备能够组成区块链系统。
在一些实施例中，本申请实施例中的计算机设备是区块链系统中的节点，该节点能够将模型存储在区块链中，之后该节点或者该区块链中的其他设备对应的节点可通过访问区块链调用模型。
相关技术中，通常会基于训练样本训练描述生成模型，训练之后即可通过该描述生成模型生成任一图像对应的描述文本。其中，训练样本包括图像和图像对应的描述文本。由于训练样本通常是从互联网中抓取得到，训练样本的质量较差，导致训练出的图像描述生成模型的性能较差。本申请实施例提供了一种图像描述生成模型处理方案，借助第一图像描述生成模型和描述修正模型，即使在第二图像描述生成模型所生成的第二描述文本的质量分数不满足质量条件的情况下，也能将第二描述文本、与通过描述修正模型修正后得到的第二修正描述文本、第一图像描述生成模型生成的第一描述文本和描述修正模型修正后得到的第一修正描述文本一起，用于更新第二图像描述生成模型，从而提升第二图像描述生成模型的性能，并且节省了样本数量，避免了浪费低质量的描述文本。
图1是本申请实施例提供的一种实施环境的示意图，参见图1，该实施环境包括：终端101和服务器102，终端101与服务器102之间通过有线网络或无线网络连接。
终端101安装和运行有客户端111，该客户端111可以是视频分享客户端、社交应用客户端、游戏客户端等类型。当终端101运行客户端111时，终端101的屏幕上显示客户端111的用户界面。
本领域技术人员可以知晓，上述终端的数量可以更多或更少。比如上述终端可以仅为一个，或者上述终端为6个或8个或者更多数量。本申请实施例对终端的数量和设备类型不加以限定。
图1中仅示出了一个终端，但在不同实施例中存在多个其它终端103可以接入服务器102。在一些实施例中，还存在一个或多个终端103是开发者对应的终端，在终端103上安装有客户端的开发和编辑平台，开发者可在终端103上对客户端进行编辑和更新，并将更新后的客户端安装包通过有线或无线网络传输至服务器102，终端101可从服务器102下载客户端安装包实现对客户端的更新。
终端101以及其它终端103通过有线网络或无线网络与服务器102相连。
服务器102用于为终端101上的客户端111提供后台服务。在一些实施例中，服务器102承担主要计算工作，终端101承担次要计算工作；或者，服务器102承担次要计算工作，终端101承担主要计算工作；或者，服务器102和终端101之间采用分布式计算架构进行协同计算。
在本申请实施例中，服务器102采用本申请实施例提供的方法更新第二图像描述生成模型，以提升第二图像描述生成模型的性能。之后，服务器102通过更新后的第二图像描述生成模型生成图像对应的描述文本，以保证该描述文本能够准确全面地描述图像。
在一些实施例中，用户使用终端101输入图像，由终端101或服务器102通过第二图像描述生成模型生成图像对应的描述文本。
例如，在发布动态信息的场景下，终端101确定用户需要发布的图像，向服务器102发送该图像，则服务器102通过更新后的第二图像描述生成模型生成该图像对应的描述文本，向终端101发送该描述文本，终端101显示该图像和该描述文本，用户点击确认后，即可该图像和该描述文本组合成动态消息后发布到社交平台中。用户只需要选择需要发布的图像，就能自动地生成该图像对应的描述文本，作为与该图像匹配的文案，而不需要用户单独输入文案，实现了动态消息的快速发布。
在另一种可能实现方式中，服务器102通过更新后的第二图像描述生成模型，生成图像对应的描述文本，基于图像和图像对应的描述文本，训练图像生成模型，以使训练后的图像生成模型能够根据任一描述文本生成符合该描述文本的图像。用户使用终端101输入描述文本，由终端101或服务器102通过图像生成模型生成符合该描述文本的图像。
例如，在生成创意图像的场景下，用户在终端101中输入一段描述文本，终端101向服务器102发送描述文本，服务器102即可通过图像生成模型生成符合该描述文本且具有独特创意的图像，向终端101发送该图像。后续用户可以将该图像分享给好友，或者发布到社交平台上，或者生成包括该图像的视频等。
需要说明的是，上述实施环境仅是示例，本申请实施例提供的方法还可以由服务器102单独执行，或者由其他的计算机设备执行，本申请实施例对此不做限定。
图2是本申请实施例提供的一种图像描述生成模型处理方法的流程图，图3是本申请实施例提供的一种第二图像描述生成模型的更新流程的示意图。本申请实施例由计算机设备执行，该计算机设备为如图1所示的终端101或服务器102等设备，本申请实施例对更新该图像描述生成模型的过程进行说明。参见图2和图3，该方法包括：
201、计算机设备通过第一图像描述生成模型，生成第一图像对应的第一描述文本，并通过描述修正模型修正所述第一描述文本得到第一修正描述文本。
本申请实施例中，计算机设备获取第一图像描述生成模型、第二图像描述生成模型和描述修正模型。以下将对上述三种模型分别进行介绍。
首先，第一图像描述生成模型用于生成图像对应的描述文本，从而以描述文本来描述图像。由于第一图像描述生成模型经过了一次或多次训练，已经具备了生成图像对应的描述文本的能力，因此可以将第一图像描述生成模型投入使用，即针对任一图像，通过第一图像描述生成模型可以生成该图像对应的描述文本。在一些实施例中，本申请实施例采用的第一图像描述生成模型为多模态大语言模型，多模态大语言模型是一种人工智能模型，能够处理多种类型的数据输入，如文本、图像、音频等，并能够生成自然语言输出，因此可以用于生成图像对应的描述文本。
其次，第二图像描述生成模型与第一图像描述生成模型的功能同理，也是用于生成图像对应的描述，从而以描述文本来描述图像。由于第二图像描述生成模型经过了一次或多次训练，已经具备了生成图像对应的描述文本的能力，因此可以将第二图像描述生成模型投入使用，即针对任一图像，通过第二图像描述生成模型可以生成该图像对应的描述文本。
在一些实施例中，图像描述生成模型(例如上述的第一图像描述生成模型和第二图像描述生成模型)可以包括两个主要组件：图像特征提取组件和文本生成组件。
其中，图像特征提取组件，用于将图像转换为特征向量，以便于后续的文本生成。这可以通过卷积神经网络(CNN)实现。CNN的基本结构包括输入层、多个卷积层、池化层和全连接层。卷积层是CNN的核心部分，用于提取图像的局部特征，池化层用于简化数据和降低计算复杂度，全连接层用于整合卷积层和池化层提取的特征，输出一个特征向量。
文本生成组件，用于将特征向量映射到文本序列，这可以通过递归神经网络(RNN)、长短期记忆网络(LSTM)或Transformer实现。
第一图像描述生成模型与第二图像描述生成模型的区别在于：第一图像描述生成模型保持不变，而第二图像描述生成模型的初始状态与第一图像描述生成模型相同，即，在采用本申请实施例提供的方法更新第二图像描述生成模型之前，第二图像描述生成模型与第一图像描述生成模型相同，而在一次或多次采用本申请实施例提供的方法更新第二图像描述生成模型之后，第二图像描述生成模型发生了变化，虽然与第一图像描述生成模型都是用于生成图像对应的描述文本，但是第二图像描述生成模型的性能更好，所生成的描述文本的准确性更高。
再次，描述修正模型用于对图像对应的描述文本进行修正，能够将图像对应的描述文本中的错误用词、不合适用语等进行修正，以得到更为准确的描述文本。另外，描述修正模型还用于对图像对应的描述文本进行质量检测，以获取描述文本的质量分数，该质量分数表示描述文本的质量好坏。本申请实施例中，描述修正模型也经过了一次或多次训练，已经具备了对描述文本进行修正和质量检测的能力。在一些实施例中，本申请实施例采用的描述修正模型为多模态大语言模型。
本申请实施例中，考虑到虽然第二图像描述生成模型已经基于训练样本进行了训练，但是第二图像描述生成模型的性能会受到训练样本的影响，为了提升第二图像描述生成模型的性能，本申请实施例在训练第二图像描述生成模型之后，还会更新第二图像描述生成模型。其中，该第二图像描述生成模型可以由计算机设备训练后进行更新，也可以由其他设备训练后发送给该计算机设备，再由该计算机设备进行更新。
并且，更新第二图像描述生成模型的过程还需要第一图像描述生成模型的参与，计算机设备会将第一图像描述生成模型作为第二图像描述生成模型的参考模型，将第一图像描述生成模型生成的描述文本作为参考描述文本，与第二图像描述生成模型生成的描述文本一起用于更新第二图像描述生成模型。在一些实施例中，所述第一图像描述生成模型也可以称为参考图像描述生成模型；所述第一图像描述生成模型生成的第一描述文本也可以称为参考描述文本。
本申请实施例以第一图像为例，该第一图像为需要生成描述文本的任一图像，如在计算机设备上输入的图像、其他设备(如用户使用的终端或者技术人员使用的终端等)发送给该计算机设备的图像或者采用其他方式得到的图像等。计算机设备通过第一图像描述生成模型，生成第一图像(也可以称为样本图像)对应的第一描述文本，第一描述文本是用来描述第一图像的文本，如描述第一图像的主题、背景、风格、构图、文化和情感等，且第一描述文本的质量能够反映出第一图像描述生成模型的性能好坏。并且，通过描述修正模型获取第一描述文本对应的第一修正描述文本。由于第一修正描述文本已经经过了描述修正模型的修正，因此可以认为第一修正描述文本的准确度高于第一描述文本的准确度。
需要说明的是，在步骤201中计算机设备生成第一描述文本后，还可以将第一描述文本应用于除更新第一图像描述生成模型以外的其他任务，如语义分析任务、文本分类任务、回复对话任务等，本申请实施例对此不做限定。例如，在智能问答的场景下，计算机设备根据用户输入的图像生成第一描述文本后，对第一描述文本进行语义分析，基于第一描述文本的语义，生成回复文本，将该回复文本反馈给用户，从而实现了一种计算机设备与用户的图像进行对话的效果。
202、计算机设备通过第二图像描述生成模型，生成第一图像对应的第二描述文本，并通过描述修正模型获取第二描述文本的质量分数，以及通过所述描述修正模型修正所述第二描述文本得到第二修正描述文本。
针对相同的第一图像，计算机设备还会通过第二图像描述生成模型，生成第一图像对应的第二描述文本，并通过描述修正模型，获取第二描述文本的质量分数以及第二描述文本对应的第二修正描述文本。
其中，第二描述文本是用来描述第一图像的文本，如描述第一图像的主题、背景、风格、构图、文化和情感等，且第二描述文本的质量能够反映出第二图像描述生成模型的性能好坏。并且，通过描述修正模型获取第二描述文本对应的第二修正描述文本。
例如，第二描述文本为“一位男子穿着绿色上衣和灰色裤子”，而经过修正后得到的第二修正描述文本为“一位男子穿着绿色上衣和褐色裤子，上衣正面有绿色印花和英文”，将裤子的颜色修正为了正确的颜色，并且，添加了上衣的更多细节描述。
在初始状态下，第一图像描述生成模型与第二图像描述生成模型为相同模型，可以认为第二描述文本的准确度等于第一描述文本的准确度，之后，由于第一图像描述生成模型一直保持不变，而第二图像描述生成模型会经过一次或多次更新，性能逐渐提升，因此可以认为第二描述文本的准确度高于第一描述文本的准确度。由于第二修正描述文本已经经过了描述修正模型的修正，因此可以认为第二修正描述文本的准确度高于第二描述文本的准确度，并且也很大可能高于第一修正描述文本的准确度。
第二描述文本的质量分数表示第二描述文本的质量好坏，也在一定程度上表示第二图像描述生成模型的性能好坏。在一些实施例中，质量分数越高，表示第二描述文本越能准确全面地描述该第一图像，质量分数越低，表示第二描述文本越不能准确全面地描述该第一图像。
在一些实施例中，质量分数可以表示为数值形式或者为字母形式，例如质量分数可以表示为1、2、3等分值，质量分数的高低由分值的大小确定。或者质量分数可以表示为A级、B级等级别，质量分数的高低由级别的高低确定。
203、计算机设备在第二描述文本的质量分数不满足质量条件的情况下，基于第一描述文本、第一修正描述文本、第二描述文本和第二修正描述文本，确定损失值，以减小所述损失值为目标，更新所述第二图像描述生成模型。
其中，质量阈值是指高质量的描述文本的质量分数应当满足的条件，该质量条件可以由计算机设备默认设置或者由技术人员根据需求设置。
本申请实施例中，考虑到描述文本的质量对第二图像描述生成模型的性能会造成影响，因此采用质量条件对描述文本进行划分，针对满足质量条件的描述文本和不满足质量条件的描述文本，分别采用不同的方式来更新第二图像描述生成模型。
本申请实施例仅是以第一图像为例，则在通过第二图像描述生成模型生成第一图像对应的第二描述文本并且通过描述修正模型获取到第二描述文本的质量分数之后，判断第二描述文本的质量分数是否满足质量条件。在第二描述文本的质量分数不满足质量条件的情况下，表示第二描述文本为低质量的描述文本，此时如果直接将第一图像和第二描述文本用来更新第二图像描述生成模型，会使第二图像描述生成模型学习到错误的知识，将影响第二图像描述生成模型的性能。而如果直接过滤掉第二描述文本，将浪费训练样本，减少了样本数量。
因此，计算机设备在获取到第一图像对应的四种描述文本：第一描述文本、第一修正描述文本、第二描述文本和第二修正描述文本后，在第二描述文本的质量参数不满足质量条件的情况下，可以将这四种描述文本一起考虑进来，通过这四种描述文本来更新第二图像描述生成模型。
在给定第一图像的情况下，可以采用多种描述文本来描述第一图像，而在多种描述文本中每种描述文本出现的概率存在差异，出现概率高的描述文本往往可以更准确描述第一图像。而若要提升第二图像描述生成模型的性能，就要求第二图像描述生成模型尽可能地生成准确性高的描述文本，而少生成准确性低的描述文本。
由于上述四种描述文本中，第二修正描述文本的准确度很可能是最高的，因此第二图像描述生成模型的更新目标是通过更新后的第二图像描述生成模型和描述修正模型得到的第二修正描述文本的概率增大，即生成了更加准确的描述文本，这样虽然第二描述文本的质量分数不满足质量条件，但是只要按照上述更新目标来更新第二图像描述生成模型，仍然能够保证更新后的第二图像描述生成模型的性能高于更新前的第二图像描述生成模型的性能。
在一些实施例中，步骤203包括：确定所述第一图像描述生成模型输出所述第一描述文本的第一概率、所述描述修正模型输出所述第一修正描述文本的第二概率、所述第二图像描述生成模型输出所述第二描述文本的第三概率和所述描述修正模型输出所述第二修正描述文本的第四概率。根据所述第一概率、第二概率、第三概率以及第四概率，确定所述损失值，其中，所述损失值与第三概率与第一概率的比值的对数正相关，与第四概率与第二概率的比值的对数负相关，以损失值减小为目标，更新第二图像描述生成模型。
其中，第三概率与第一概率的比值的对数，表示第三概率的对数与第一概率的对数之间的差值，而第四概率与第二概率的比值的对数，表示第四概率的对数与第二概率的对数之间的差值。
第二图像描述生成模型的更新目标为损失值减小，而损失值越小，第三概率的对数与第一概率的对数之间的差值越小，表示第二图像描述生成模型生成第二描述文本的概率与第一图像描述生成模型生成第一描述文本的概率之间的偏差越小，即针对第一图像，输出第一描述文本和第二描述文本的概率都下降，而且由于第二描述文本的质量分数不满足质量条件，因此输出第二描述文本的概率下降的更多。损失值越小，第四概率的对数与第二概率的对数之间的差值越大，即第四概率越大、第二概率越小，即针对第一图像更加倾向于输出第二修正描述文本，而不倾向于输出第一修正描述文本。因此，采用此更新目标更新第二图像描述生成模型，可以使通过更新后的第二图像描述生成模型和描述修正模型得到的第二修正描述文本的概率增大，从而提升第二图像描述生成模型的性能。
例如，损失值、第一概率、第二概率、第三概率和第四概率满足以下关系：
其中，L(πθ；πref)表示损失值，E表示期望，θ指代第二图像描述生成模型，ref指代第一图像描述生成模型，x指代图像，yl指代通过图像描述生成模型生成的描述文本，yw指代通过描述修正模型修正后的描述文本，β表示超参数，σ表示sigmiod函数(S型函数)，因此，πθ(yw/x)表示第四概率，πref(yw/x)表示第二概率，πθ(yl/x)表示第三概率，πref(yl/x)表示第一概率。
在一些实施例中，所述方法可以进一步包括：
204、计算机设备在第二描述文本的质量分数满足质量条件的情况下，基于所述第一图像和第二描述文本，更新第二图像描述生成模型。
在第二描述文本的质量分数满足质量条件的情况下，表示第二描述文本为高质量的描述文本，第一图像和第二描述文本可以用来更新第二图像描述生成模型，从而增加了样本数量，使第二图像描述生成模型学习到第二描述文本中包含的正确知识，从而有效提升第二图像描述生成模型的性能。
在一些实施例中，在第二描述文本的质量分数满足质量条件的情况下，将第一图像和第二描述文本构成一组样本，添加到第二图像描述生成模型的训练集中，之后基于训练集中的一组或多组样本，更新第二图像描述生成模型。由于训练集中只存储高质量的描述文本，因此基于高质量的描述文本更新第二图像描述生成模型，可以使第二图像描述生成模型学习到正确的知识，从而有效地提升性能。
相关技术中，通常会基于训练样本训练图像描述生成模型，训练之后即可通过该图像描述生成模型生成任一图像对应的描述文本。其中，训练样本包括图像和图像对应的描述文本。但是描述生成模型生成的描述文本的精确度不足，而且存在严重的“幻觉”问题，即描述文本会包含错误的描述信息，而且无法实现自学习更新。
本申请实施例提供的方法，借助第一图像描述生成模型和描述修正模型，即使在第二图像描述生成模型所生成的第二描述文本的质量不满足质量条件的情况下，也能将第二描述文本、与第二修正描述文本、第一描述文本和第一修正描述文本一起，用于更新第二图像描述生成模型，从而提升第二图像描述生成模型的性能，并且节省了样本数量，避免了浪费低质量的描述文本。
另外，在第二图像描述生成模型所生成的第二描述文本的质量分数满足质量条件的情况下，表示第二描述文本为高质量的描述文本，则将高质量的第二描述文本和第二描述文本对应的第一图像用于更新第二图像描述生成模型，从而使第二图像描述生成模型实现了自学习更新，提升了第二图像描述生成模型的性能。
例如，在持续处理图像的场景中，第二图像描述生成模型能够基于新处理的图像实现自学习更新，不断迭代更新第二图像描述生成模型的性能，以适应实时场景中持续处理图像的需求。
在上述实施例的基础上，以下对确定第二描述文本的质量分数是否满足质量条件的方式进行说明。
在一些实施例中，质量条件包括质量分数阈值，在所述第二描述文本的质量分数小于或等于所述质量分数阈值的情况下，确定所述第二描述文本不满足所述质量条件。
质量分数阈值是区分高质量描述文本和低质量描述文本的阈值，例如，质量分数的取值范围是区间[0，1]，质量分数阈值为0.5或0.8等，或者，质量分数的取值范围是区间[0，10]，质量分数阈值为5或8等，或者质量分数属于集合{A，B，C，D，E}，从A到E，级别依次升高，且质量分数阈值为C，即质量分数为A，B或C的描述文本为低质量描述文本，而质量分数为D或E的描述文本为高质量描述文本。
质量分数阈值可以由计算机设备进行设置或者由技术人员根据需求设置。或者，计算机设备获取到多个描述文本的情况下，基于多个描述文本的质量分数确定质量分数阈值，例如，该质量分数阈值等于该多个描述文本的质量分数的平均值，或者该质量分数阈值等于该多个描述文本的质量分数的总和与预设比例的乘积等。
在第二描述文本的质量分数大于质量分数阈值的情况下，表示第二描述文本为高质量描述文本，满足质量条件。而在另一实施例中，在第二描述文本的质量分数不大于质量分数阈值的情况下，表示第二描述文本为低质量描述文本，不满足质量条件。
本申请实施例按照质量分数阈值，筛选出了质量分数大于质量分数阈值的描述文本，作为高质量的描述文本，以及质量分数不大于质量分数阈值的描述文本，作为低质量的描述文本，以便基于高质量的描述文本和基于低质量的描述文本，分别采用不同的方式来更新第二图像描述生成模型，从而提高了灵活性。
在一些实施例中，第二图像描述生成模型具有生成多维度描述文本的能力，则通过第二图像描述生成模型生成的第二描述文本能够从N个维度上描述第一图像，N为大于1的整数，这N个维度包括主题、背景、风格、构图、文化和情感中的至少两个，另外还可以包括其他的维度。而描述修正模型具有对多维度的描述文本进行质量检测的能力，因此对第二描述文本进行质量检测得到的质量分数会包括N个维度的子分数，每个子分数表示第二描述文本在该子分数对应的维度上描述第一图像的质量。质量条件包括N个维度的质量分数阈值，用来从N个维度上分别判断第二描述文本的质量是否满足要求。
将大于所属维度的质量分数阈值的子分数称为高质量子分数，则该方法还包括以下任一项：
(1)、在N个维度的子分数不全是高质量子分数的情况下，确定第二描述文本不满足质量条件。
在另一实施例中，第二描述文本的N个维度的子分数均为高质量子分数，表示第二描述文本从每个维度上都能准确地描述第一图像，属于高质量的描述文本，则确定第二描述文本满足质量条件。
(2)、在N个维度的子分数中的高质量子分数的数量不大于预设数量的情况下，确定第二描述文本不满足质量条件，预设数量小于N。
其中，预设数量由计算机设备默认设置或者由技术人员根据N的数值设置。第二描述文本的N个维度的子分数中的高质量子分数的数量大于预设数量，表示第二描述文本从足够多的维度上都能准确地描述第一图像，因此属于高质量的描述文本，满足质量条件。而第二描述文本的N个维度的子分数中的高质量子分数的数量不大于预设数量，表示第二描述文本较少的维度上才能准确地描述第一图像，因此属于低质量的描述文本，不满足质量条件。
(3)、在N个维度的子分数中的高质量子分数所占的比例不大于预设比例的情况下，确定第二描述文本不满足质量条件，预设比例小于100％。
其中，预设比例由计算机设备默认设置或者由技术人员根据需求设置。第二描述文本的N个维度的子分数中的高质量子分数的所占的比例大于预设比例，表示第二描述文本从足够多的维度上都能准确地描述第一图像，因此属于高质量的描述文本，满足质量条件。而第二描述文本的N个维度的子分数中的高质量子分数所占的比例不大于预设比例，表示第二描述文本较少的维度上才能准确地描述第一图像，因此属于低质量的描述文本，不满足质量条件。
本申请实施例中，考虑到第二图像描述生成模型所生成的第二描述文本为多维度描述文本，因此提供了一种适用于多维度描述文本的筛选方案，将第二描述文本的质量分数中，大于所属维度的质量分数阈值的子分数称为高质量子分数，基于第二描述文本的N个维度的子分数中的高质量子分数的多少，筛选出高质量的描述文本和低质量的描述文本。本申请实施例提供了准确筛选高质量和低质量的多维度描述文本的方案，提升了筛选的准确性，从而提升了第二图像描述生成模型的性能。
在另一种可能实现方式中，计算机设备可以分别生成多个第一图像对应的第二描述文本，从而得到了多个第二描述文本，并且通过描述修正模型，分别获取到了多个第二描述文本的质量分数，则基于多个第二描述文本的质量分数，对多个第二描述文本进行过滤，区分出高质量的第二描述文本和对应的第一图像，以及低质量的第二描述文本和对应的第一图像，从而采用不同的方式更新第二图像描述生成模型。
在一些实施例中，计算机设备每次生成一个第一图像对应的第二描述文本，将第一图像和第二描述文本作为一组样本，存储到样本集中，随着样本的数量逐渐增多，在样本集中的样本数量达到目标样本数量后，通过描述修正模型获取样本集中的每个第二描述文本的质量分数，基于多个第二描述文本的质量分数，对多个第二描述文本进行划分。或者，计算机设备每次生成一个第一图像对应的第二描述文本，将第一图像和第二描述文本作为一组样本，存储到样本集中，并且每隔预设时长，针对该预设时长内样本集中收集的样本，通过描述修正模型获取第二描述文本的质量分数，基于多个第二描述文本的质量分数，对多个第二描述文本进行划分。
在一些实施例中，质量条件包括：质量排序阈值；
当所述第二描述文本的质量分数在多个第二描述文本按照质量分数从高到低的排序中的位置在所述质量排序阈值之后时，确定所述第二描述文本不满足所述质量条件；其中，所述多个第二描述文本是所述第二图像描述模型生成的、多个图像分别对应的描述文本。
例如，质量条件为质量分数排在多个第二描述文本中的前K个，该方法还包括：
计算机设备按照多个第二描述文本的质量分数从大到小的顺序，确定未排在前K位的第二描述文本不满足质量条件。
其中，K为正整数，K小于多个第二描述文本的数量。K可以由计算机设备默认设置、由技术人员根据需求设置，另外K等于多个第二描述文本的数量与预设比例的乘积或者K等于其他数值等。
计算机设备按照多个第二描述文本的质量分数从大到小的顺序，对第二描述文本进行排序，对于任一第二描述文本来说，在该第二描述文本排在前K位的情况下，表示第二描述文本为高质量描述文本，满足质量条件。而在该第二描述文本未排在前K位的情况下，表示第二描述文本为低质量描述文本，不满足质量条件。
本申请实施例按照多个第二描述文本的质量分数从大到小的顺序，对多个第二描述文本进行排序，将排在前K位的第二描述文本作为高质量的描述文本，将未排在前K位的第二描述文本作为低质量的描述文本，以便基于高质量的描述文本和基于低质量的描述文本，分别采用不同的方式来更新第二图像描述生成模型，从而提高了灵活性。本申请实施例提供了准确划分描述文本的方案，提升了筛选的准确性，从而提升了第二图像描述生成模型的性能。
在一些实施例中，第二描述文本的质量分数包括N个维度的子分数，每个子分数表示第二描述文本在该子分数对应的维度上描述第一图像的质量。将多个第二描述文本的同一个维度的子分数中排在前K位的子分数称为高质量子分数，则该方法还包括以下任一项：
(1)、在N个维度的子分数不全是高质量子分数的情况下，确定第二描述文本不满足质量条件。
在另一实施例中，第二描述文本的N个维度的子分数均为高质量子分数，表示第二描述文本从每个维度上都能准确地描述第一图像，属于高质量的描述文本，则确定第二描述文本满足质量条件。
(2)、在N个维度的子分数中的高质量子分数的数量不大于预设数量的情况下，确定第二描述文本不满足质量条件，预设数量小于N。
其中，预设数量由计算机设备默认设置或者由技术人员根据N的数值设置。第二描述文本的N个维度的子分数中的高质量子分数的数量大于预设数量，表示第二描述文本从足够多的维度上都能准确地描述第一图像，因此属于高质量的描述文本，满足质量条件。而第二描述文本的N个维度的子分数中的高质量子分数的数量不大于预设数量，表示第二描述文本较少的维度上才能准确地描述第一图像，因此属于低质量的描述文本，不满足质量条件。
(3)、在N个维度的子分数中的高质量子分数所占的比例不大于预设比例的情况下，确定第二描述文本不满足质量条件，预设比例小于100％。
其中，预设比例由计算机设备默认设置或者由技术人员根据需求设置。第二描述文本的N个维度的子分数中的高质量子分数的所占的比例大于预设比例，表示第二描述文本从足够多的维度上都能准确地描述第一图像，因此属于高质量的描述文本，满足质量条件。而第二描述文本的N个维度的子分数中的高质量子分数所占的比例不大于预设比例，表示第二描述文本较少的维度上才能准确地描述第一图像，因此属于低质量的描述文本，不满足质量条件。
另外，计算机设备还可以采用其他的方式对多个第二描述文本进行质量划分，本申请实施例对划分方式不做限定。
本申请实施例中，考虑到第二图像描述生成模型所生成的第二描述文本为多维度描述文本，因此提供了一种适用于多维度描述文本的筛选方案，按照不同的维度分别对多个第二描述文本的子分数进行排序，将多个第二描述文本的同一个维度的子分数中排在前K位的子分数称为高质量子分数，基于第二描述文本的N个维度的子分数中的高质量子分数的多少，筛选出高质量的描述文本和低质量的描述文本。本申请实施例提供了准确筛选高质量的多维度描述文本的方案，提升了筛选的准确性，从而提升了第二图像描述生成模型的性能。
在上述实施例的基础上，以下对训练第二图像描述生成模型的过程进行说明。
在一些实施例中，在计算机设备确定多组高质量的样本(第一图像和满足质量条件的第二描述文本)后，训练所述第二图像描述生成模型的过程包括：
对于每组样本，基于当前的第二图像描述生成模型生成第一图像对应的描述文本，基于生成的描述文本与第二描述文本之间的误差，以将该误差减小为目标，调整第二图像描述生成模型的模型参数。
在经过一组或多组样本的训练之后，即可提升第二图像描述生成模型的性能，使得第二图像描述生成模型生成高质量的描述文本的概率增大。
在上述实施例的基础上，计算机设备更新第二图像描述生成模型后，可以通过更新后的第二图像描述生成模型，生成任一图像对应的描述文本，由于更新后的第二图像描述生成模型的性能更强，因此提高了所生成的描述文本的质量。或者，还可以通过更新后的第二图像描述生成模型，训练图像生成模型。
其中，图像生成模型用于基于任一描述文本生成符合该描述文本的图像。例如，用户输入一段描述文本，描述了想要生成的图像的样子，则通过图像生成模型可以生成符合用户要求的图像。
训练图像生成模型的过程包括：通过更新后的第二图像描述生成模型，生成样本图像对应的样本描述文本，样本描述文本用于描述样本图像，基于样本图像和样本描述文本，训练图像生成模型。
其中，为了便于区分，将训练图像生成模型所用的图像称为样本图像，将样本图像对应的描述文本称为样本描述文本，该样本图像与上述实施例中的第一图像可以包括相同的图像，也可以包括不同的图像，样本图像可以由其他设备发送给计算机设备、由技术人员存储至计算机设备，或者采用其他方式得到。
由于当前的第二图像描述生成模型已经经过了训练和更新，性能较强，因此通过第二图像描述生成模型生成的样本描述文本的质量较高，能够准确全面地描述样本图像，则以样本图像为参考样本，以样本描述文本为输入样本，训练图像生成模型，可以有效地提升图像生成模型的性能，使得图像生成模型能够基于输入的描述文本生成更加符合该描述文本的图像。
在一些实施例中，基于样本图像和样本描述文本，训练图像生成模型，包括：通过图像生成模型，生成样本描述文本对应的图像，基于所生成的图像与样本图像之间的误差，以将该误差减小为目标，调整图像生成模型的模型参数。
在经过一组或多组样本的训练之后，即可提升图像生成模型的性能，使得图像生成模型能够生成高质量的图像。
相关技术中的图像生成模型训练所用的训练样本可能会包含较多的噪声信息，导致训练出的图像生成模型的性能较差。
而本申请实施例还提供了一种训练图像生成模型的方案，能够借助第二图像描述生成模型生成图像对应的描述文本，从而基于图像和图像对应的描述文本训练图像生成模型。由于当前的第二图像描述生成模型已经经过了训练和更新，性能较强，因此通过第二图像描述生成模型生成的描述文本的质量较高，能够准确全面地描述图像，因此可以有效地提升图像生成模型的性能，使得图像生成模型能够基于输入的描述文本生成更加符合该描述文本的图像。而且无需人工确定图像对应的描述文本，节省了人工成本，提高了训练效率。
并且，由于第二图像描述生成模型实现了自学习更新，适用于实时场景，在不断地提升第二图像描述生成模型的性能的同时，还能够为图像生成模型提供更多高质量的描述文本，丰富了图像生成模型的样本数量，从而也能有效提升图像生成模型的性能。
在上述实施例的基础上，本申请实施例还提供了另一种图像描述生成模型处理方法，图4是本申请实施例提供的另一种图像描述生成模型处理方法的流程图，本申请实施例由计算机设备执行，该计算机设备为如图1所示的终端101或服务器102等设备。在一些实施例中，所述第一图像描述生成模型和所述第二图像描述生成模型是通过训练一个待训练的图像描述生成模型得到的，本申请实施例对训练所述图像描述生成模型得到所述第一图像描述生成模型和所述第二图像描述生成模型的过程进行说明。参见图4，该方法包括：
401、生成与第二图像对应的组合描述文本，其中，所述组合描述文本用于从多个维度描述所述第二图像。
在一些实施例中，所述生成与第二图像对应的组合描述文本，包括：
4011，通过多个第三图像描述生成模型，分别生成第二图像对应的多个维度的描述文本；其中，每个维度的描述文本用于从该维度上描述所述第二图像；每个第三图像描述生成模型用于生成所述第二图像对应的一个维度的描述文本；
4012，通过第二大语言模型，组合所述多个维度的描述文本，得到组合描述文本。
其中，为了便于区分，将训练第一图像描述生成模型所用的图像称为第二图像，该第二图像与上述实施例中的第一图像可以包括不同的图像，第二图像可以由其他设备发送给计算机设备、由技术人员存储至计算机设备，或者采用其他方式得到。
考虑到针对第二图像，如果人工确定第二图像对应的描述文本，会导致人工成本较大，训练效率很低，因此训练第一图像描述生成模型的过程中，需要借助第三图像描述生成模型来生成第二图像对应的描述文本，才能节省人工成本，提高训练效率。第三图像描述生成模型与第一图像描述生成模型的功能类似，也是用于生成图像对应的描述文本的模型。因此，步骤401与上述步骤201同理。区别在于，第一图像描述生成模型与第三图像描述生成模型为不同的描述生成模型。例如，第一图像描述生成模型与第三图像描述生成模型的模型架构相同，训练过程中使用的训练样本不同，或者，第一图像描述生成模型与第三图像描述生成模型的模型架构不同，训练过程中使用的训练样本包括相同的训练样本，也包括不同的训练样本。
另外，多个第三图像描述生成模型也可以为相同的模型或者不同的模型，例如，多个第三图像描述生成模型的模型架构相同，训练过程中使用的训练样本不同，导致针对相同的图像，多个第三图像描述生成模型生成的描述文本可能会不同。
本申请实施例中，由于多个第三图像描述生成模型经过了一次或多次训练，已经具备了生成图像对应的描述文本的能力，因此可以将多个第三图像描述生成模型的生成结果作为训练样本，用于训练第一图像描述生成模型。
在一些实施例中，本申请实施例采用的第三图像描述生成模型为多模态大语言模型，多模态大语言模型是一种人工智能模型，能够处理多种类型的数据输入，如文本、图像、音频等，并能够生成自然语言输出，因此可以用于生成图像对应的描述文本。
在一些实施例中，多个第三图像描述生成模型存储于计算机设备上，计算机设备调用第三图像描述生成模型即可生成描述文本。或者，多个第三图像描述生成模型存储于其他的设备上，其中，不同的第三图像描述生成模型可以存储于相同的设备或者不同的设备上。计算机设备向存储第三图像描述生成模型的设备发送生成描述文本的请求，该请求携带第二图像，则存储第三图像描述生成模型的设备基于第三图像描述生成模型生成第二图像对应的描述文本后，将第三描述文本发送给计算机设备。
在一些实施例中，每个第三图像描述生成模型具有一个或多个目标维度，用于生成属于目标维度的描述文本。例如，多个维度包括：主题、背景、风格、构图、文化和情感中的至少两个，另外还可以包括其他的维度，不同的第三图像描述生成模型的目标维度可以包括相同的维度，也可以包括不同的维度，且多个第三图像描述生成模型中包括目标维度不同的第三图像描述生成模型，则多个第三图像描述生成模型总共对应于N个维度，N为大于1的整数。
例如，针对每个第三图像描述生成模型，在训练过程中所使用的训练样本包括图像和属于目标维度的描述文本，而不包括属于其他维度的描述文本，这样能有效减少描述文本的复杂度，减少训练样本的信息量，从而提高训练速度，而基于上述训练样本训练出来的第三图像描述生成模型具有基于图像生成属于目标维度的描述文本的能力，从而提升了针对性，能够更好地从目标维度上描述图像，但是所生成的描述文本涉及的维度不够全面。
第一大语言模型是一种人工智能模型，旨在理解和生成人类语言，第一大语言模型需要在大量文本上进行训练，以确保通过第一大语言模型能够准确地执行任务，包括文本总结、翻译、情感分析等等。第一大语言模型可以包括GPT(Generative Pre-Trained Transformer，生成式预训练Transformer模型)、GLM(Generalized Linear Models，广义线性模型)、CPM(Chinese Pre-trained Models，中文预训练模型)、混元大模型等，本申请实施例对第一大语言模型不做限定。
本申请实施例中，由于多个第三图像描述生成模型总共生成了N个维度的描述文本，N为大于1的整数，通过第一大语言模型将多个第三图像描述生成模型生成的描述文本组合起来，对多个描述文本进行相互补充和修正，从而生成更为准确的组合描述文本，能够为第一图像描述生成模型提供多维度的训练样本，从而不仅能够提升第一图像描述生成模型的性能，还能提升第一图像描述生成模型的全面性，使得第一图像描述生成模型能够为输入的图像生成多维度的描述文本，更加全面地描述图像。
在一些实施例中，即使计算机设备无法确定每个第三图像描述生成模型的目标维度为哪一个维度，但是通过第一大语言模型，仍然可以对每个第三图像描述生成模型生成的描述文本进行语义分析，从而对多个描述文本进行组合时，能够考虑不同的描述文本所属维度是否相同，进而去除多个描述文本中重复维度的描述内容，并保留不重复维度的描述内容，以使得到的组合描述文本既能从多个维度描述第二图像，又不会存在重复内容。
例如，图5是本申请实施例提供的一种组合多维度描述文本的流程示意图，参见图5，通过多个第三图像描述生成模型对第二图像进行处理，得到多个维度的描述文本，再通过第一大语言模型进行组合，得到的组合描述文本中能够从多个维度描述第二图像，该多个维度包括：主题、背景、风格、构图、文化和情感等。
402、计算机设备将所述第二图像和所述组合描述文本作为训练样本对，训练所述待训练的图像描述生成模型。
由于组合描述文本是综合了多个第三图像描述生成模型的生成结果得到的，可以认为是更加全面、质量更高的描述文本，因此基于第二图像和组合描述文本，训练第一图像描述生成模型，可以有效提升第一图像描述生成模型的性能和全面性。
在一些实施例中，基于第二图像和组合描述文本，训练第一图像描述生成模型之前，方法还包括：显示第二图像和组合描述文本，基于对组合描述文本执行的修正操作，获取修正后的组合描述文本。
其中，计算机设备可以在本地的显示屏幕上显示第二图像和组合描述文本，供计算机设备的用户查看该组合描述文本是否准确全面，对组合描述文本执行修正操作，如在组合描述文本中增加新的文本内容，从组合描述文本中删除文本内容，或者修改组合描述文本中的文本内容等。修正之后计算机设备即可获取到修正后的组合描述文本。
或者，计算机设备还可以向修正人员的设备发送第二图像和组合描述文本，从而在修正人员的设备上显示第二图像和组合描述文本，对组合描述文本执行修正操作，如在组合描述文本中增加新的文本内容，从组合描述文本中删除文本内容，或者修改组合描述文本中的文本内容等。修正之后该设备即可获取到修正后的组合描述文本，发送给计算机设备。
例如，在图5所示的组合描述文本的基础上，训练第一图像描述生成模型的流程如图6所示。虽然对组合描述文本执行修正操作，会耗费一定的人力成本，但是由于已经生成了组合描述文本，只需要在组合描述文本的基础上进行修正，耗费的人力成本不会太高，而且能提高组合描述文本的准确性，从而保证了后续基于第二图像和组合描述文本训练第一图像描述生成模型，能够有效地提升第一图像描述生成模型的性能。
需要说明的是，本申请实施例所示的训练过程可以在上述图2所示的实施例中的更新过程之前，即采用本申请实施例训练第一图像描述生成模型后，再采用图2所示的实施例更新第二图像描述生成模型。或者，本申请实施例所示的训练过程可以与上述图2所示的实施例中的更新过程交叉进行，如采用本申请实施例训练第一图像描述生成模型后，再采用图2所示的实施例更新第二图像描述生成模型，之后还可以采用本申请实施例再次训练第一图像描述生成模型，本申请实施例对训练第一图像描述生成模型和更新第二图像描述生成模型的时间不做限定。
403、计算机设备将训练完成后的描述生成模型进行复制，得到所述第一图像描述生成模型和所述第二图像描述生成模型。
训练完成后，即可将训练后的图像描述生成模型作为第一图像模型生成模型，然后将第一图像描述生成模型进行复制得到第二图像描述生成模型，因此，第一图像描述生成模型与第二图像描述生成模型的初始状态相同。后续会保持第一图像描述生成模型不变的情况下，更新第二图像描述生成模型。
需要说明的是，本申请实施例以复制第一图像描述生成模型得到第二图像描述生成模型为例，在另一实施例中也可以将训练完成的图像描述生成模型作为第二图像描述生成模型，然后复制第二图像描述生成模型得到第一图像描述生成模型，也即是，在训练得到一个图像描述生成模型，并进行复制得到另一个图像描述生成模型，此时可以将这两个图像描述生成模型中的任一个作为第一图像描述生成模型，保持不变，而将另一个作为第二图像描述生成模型，采用上述实施例中的更新方法来进行更新。
相关技术中，通常会基于训练样本训练图像描述生成模型，训练之后即可通过该图像描述生成模型生成任一图像对应的描述文本。其中，训练样本包括图像和图像对应的描述文本。但是图像描述生成模型会受到训练样本中的描述文本的维度的限制，导致生成的描述文本属于单一的维度，不能涵盖多个维度，无法全面地描述图像。
本申请实施例提供了一种训练图像描述生成模型的方案，能够借助多个第三图像描述生成模型生成第二图像对应的多个维度的描述文本，再通过第一大语言模型对多个维度的描述文本进行相互补充和修正，从而组合得到多维度的组合描述文本，该组合描述文本能够从多个维度描述第二图像，是更加全面、质量更高的描述文本，例如该组合描述文本能够覆盖主题、背景、风格、构图、文化和情感等多个维度，因此基于第二图像和组合描述文本训练第一图像描述生成模型，可以有效提升第一图像描述生成模型的性能和全面性，使得第一图像描述生成模型能够为输入的图像生成多维度的描述文本，更加准确全面地描述图像。而且无需人工确定描述文本，节省了人工成本，提高了训练效率。
并且，生成组合描述文本后，只需要在组合描述文本的基础上进行修正即可得到更为准确的组合描述文本，耗费的人力成本不会太高，而且能提高组合描述文本的准确性，从而保证了后续基于第二图像和组合描述文本训练图像描述生成模型，能够有效地提升图像描述生成模型的性能。
并且，由于图像描述生成模型能够为输入的图像生成多维度的描述文本，更加准确全面地描述图像，从而也能为图像生成模型提供更多地多维度描述文本，则基于图像描述生成模型生成的描述文本训练图像生成模型，也能够有效地提升图像生成模型的性能。
在步骤401之后，该方法还包括基于组合描述文本训练描述修正模型的过程，即该方法还包括步骤404和405：
步骤404，计算机设备获取为组合描述文本标注的质量分数。
在一些实施例中，质量分数包括N个维度的子分数，N为大于1的整数，每个子分数表示组合描述文本在该子分数对应的维度上描述第二图像的质量，基于组合描述文本和组合描述文本的质量分数，训练描述修正模型。
其中，该质量分数可以由人工标注，标注人员通过对组合描述文本进行质量检测确定组合描述文本的质量分数。由于组合描述文本用于从N个维度描述第二图像，标注人员可以分别对每个维度的描述文本进行质量检测，从而得到N个维度的子分数。之后基于组合描述文本和组合描述文本的质量分数，训练描述修正模型，使得描述修正模型具备对图像的描述文本中各个维度的描述文本进行质量检测的能力。
例如，参见图7，以描述修正模型为评分模型为例，对组合描述文本进行人工评分，得到多个维度的分值，基于组合描述文本及多个维度的分值训练评分模型。
步骤404中训练描述修正模型的过程与步骤402中训练图像描述生成模型的过程可以并行执行，或者一前一后执行，本申请实施例对描述修正模型和第一图像描述生成模型的训练时间不做限定。
步骤405，计算机设备基于组合描述文本以及组合描述文本的质量分数，训练描述修正模型。
本申请实施例还提供了一种训练描述修正模型的方案，能够借助多个第三图像描述生成模型生成第二图像对应的多个维度的描述文本，再通过第一大语言模型对多个维度的描述文本进行相互补充和修正，从而组合得到多维度的组合描述文本，该组合描述文本能够从多个维度描述第二图像，是更加全面、质量更高的描述文本，因此基于组合描述文本和为组合描述文本标注的质量分数，以及组合描述文本和修正后的组合描述文本训练描述修正模型，可以有效提升质量检测的性能和全面性，使得描述修正模型能够为多维度的描述文本确定多个维度的子分数，更加准确全面地进行质量检测，而且更加准确地修正描述文本。而且无需人工确定描述文本，节省了人工成本，提高了训练效率。
在上述实施例的基础上，本申请实施例还提供了另一种模型处理方法，图8是本申请实施例提供的另一种图像描述生成模型处理方法的流程图，本申请实施例由计算机设备执行，该计算机设备为如图1所示的终端101或服务器102等设备。在一些实施例中，在步骤401之前，所述训练所述图像描述生成模型进一步包括：第一训练阶段和第二训练阶段中的至少一个，其中：所述第一训练阶段用于训练所述视觉模型和所述连接器；以及所述第二训练阶段用于训练所述第一大语言模型和所述连接器。
本申请实施例对训练该图像描述生成模型的过程进行说明。参见图8，该方法包括：
801、计算机设备生成待训练的图像描述生成模型，所述图像描述生成模型包括视觉模型、第二大语言模型和位于视觉模型和第二大语言模型之间的连接器。
该图像描述生成模型的架构图如图9所示。其中，视觉模型为处理视觉信息的模型，该视觉信息可以为图片或视频等形式的信息，该视觉模型可以包括Transformer模型(一种基于自注意力机制的神经网络模型)、ViT模型等。第二大语言模型是一种人工智能模型，旨在理解和生成人类语言，第二大语言模型需要在大量文本上进行训练，以确保通过第一大语言模型能够准确地执行任务，包括文本总结、翻译、情感分析等等。第二大语言模型可以包括GPT、GLM、CPM、混元大模型等，本申请实施例对第二大语言模型不做限定。
该连接器将视觉模型和第二大语言模型相连，用于将视觉模型输出的特征所属的第一特征空间转换为第二大语言模型输入的特征所属的第二特征空间，从而使第二大语言模型能够识别视觉模型输出的特征，即将视觉模型与第二大语言模型对齐。
需要说明的是，在一些实施例中，在计算机设备新生成图像描述生成模型时，可以对视觉模型、连接器和第二大语言模型中的模型参数进行初始化，之后执行下述步骤802和803以训练图像描述生成模型。在另一种可能实现方式中，计算机设备获取的视觉模型和第二大语言模型已经经过了训练，也即是视觉模型和第二大语言模型的模型参数已经经过了训练，之后计算机设备将视觉模型和第二大语言模型以及连接器组合起来，得到图像描述生成模型。其中，图像描述生成模型中任一个单元(包括视觉模型、连接器和第二大语言模型)的模型参数包括该单元中的权重或偏置项，或其他类型的模型参数等。
802、计算机设备在保持第二大语言模型的模型参数不变的情况下，基于第一训练样本，训练视觉模型和连接器，第一训练样本包括第三图像及第三图像对应的第三描述文本。
其中，保持第二大语言模型的模型参数不变包括：将第二大语言模型的模型参数冻结。在步骤802执行完成后，再将第二大语言模型的模型参数解冻。
本申请实施例中，为了便于区分，将此阶段训练图像描述生成模型所用的图像称为第三图像，该第三图像与上述实施例中的第一图像可以包括不同的图像，该第三图像与上述实施例中的第二图像可以包括相同的图像，也可以包括不同的图像。第三图像可以由其他设备发送给计算机设备、由技术人员存储至计算机设备，或者采用其他方式得到。将第三图像对应的描述文本称为第三描述文本，第三描述文本可以由人工确定，或者通过第三描述生成模型确定，本申请实施例对此不做限定。
在保持第二大语言模型的模型参数不变的情况下，训练视觉模型和连接器，能够将视觉模型输出的特征所属的第一特征空间与第二大语言模型的特征所属的第二特征空间对齐，以使连接器能够将视觉模型输出的、属于第一特征空间的特征转换为属于第二特征空间的特征，输入到第二大语言模型中，这样第二大语言模型就能识别属于第二特征空间的特征，从而对属于第二特征空间的特征进行处理。
803、计算机设备在保持视觉模型的模型参数不变的情况下，基于第二训练样本，训练第二大语言模型和连接器，第二训练样本包括第四图像及针对第四图像的对话文本。
其中，保持视觉模型的模型参数不变包括：将视觉模型的模型参数冻结。在步骤803执行完成后，再将视觉模型的模型参数解冻。
本申请实施例中，为了便于区分，将此阶段训练图像描述生成模型所用的图像称为第四图像，该第四图像与上述实施例中的第一图像可以包括不同的图像，该第四图像与上述实施例中的第二图像和上述步骤802中的第三图像可以包括相同的图像，也可以包括不同的图像。第四图像可以由其他设备发送给计算机设备、由技术人员存储至计算机设备，或者采用其他方式得到。第四图像对应于对话文本，该对话文本用于针对第四图像进行对话。在一些实施例中，该对话文本包括至少两条对话语句，每条对话语句与第四图像有关。该对话文本也属于用于描述第四图像的描述文本，与普通的描述文本相比，第四图像的描述文本中的至少两条对话语句之间存在上下文关系，包含着更多的信息量。例如，针对第四图像的对话文本包括两条对话语句，第一条对话语句是针对第四图像提出的问题“这张图像中有什么内容？”，第二条对话语句是针对上述问题的回答“这张图像中有两个动物”。其中，针对第四图像的对话文本可以由人工确定，或者通过从互联网中抓取得到，例如，将对第四图像的评论以及对该评论的回复组成针对第四图像的对话文本，本申请实施例对此不做限定。
在保持视觉模型的模型参数不变的情况下，训练第二大语言模型和连接器，能够增强第二大语言模型的理解能力，使得第二大语言模型能够准确地理解连接器输出的特征，从而处理得到准确的图像描述文本。需要说明的是，步骤802和步骤803可以看作是图像描述生成模型的两个训练阶段，这两个训练阶段可以依次执行，且本申请实施例以步骤802在步骤803之前执行为例，在另一实施例中还可以在步骤802之前执行步骤803。而且，步骤802和步骤803分别为可选方案，即本申请实施例中可以只执行步骤802，而不执行步骤803，或者，只执行步骤803，而不执行步骤802，本申请实施例对图像描述生成模型的训练过程不做限定。
本申请实施例提供了另一种训练图像描述生成模型的方案，能够分阶段地进行训练，首先在保持第二大语言模型的模型参数不变的情况下，基于第三图像及第三图像对应的第三描述文本，训练视觉模型和连接器，将视觉模型与第二大语言模型对齐，之后在保持视觉模型的模型参数不变的情况下，基于第四图像及针对第四图像的对话文本，训练第二大语言模型和连接器，能够增强第二大语言模型的理解能力，从而在每个阶段进行有针对性地训练，提升了图像描述生成模型的性能，提高了训练效果。
在一些实施例中，描述修正模型与图像描述生成模型的架构相同，描述修正模型也可以采用如图8所示的实施例进行训练。区别在于，图像描述生成模型的训练样本包括图像以及图像对应的描述文本或对话文本，而描述修正模型的训练样本包括图像对应的描述文本和该描述文本的质量分数和修正后的描述文本，本申请实施例对描述修正模型的训练过程不做限定。
另外，本申请实施例提供的训练方法可以与上述图2和图4所示的实施例相结合，以便采用多种方式训练图像描述生成模型，从而更加全面地提升图像描述生成模型的性能。
例如，如图10所示，将步骤802作为第一训练阶段，将步骤803作为第二训练阶段，将步骤401-402和步骤404-405作为第三训练阶段，将步骤201-204作为位于第三训练阶段之后的更新阶段，例如，在执行第一训练阶段、第二训练阶段和第三训练阶段后，得到训练后的图像描述生成模型，作为所述第一图像描述生成模型，并复制得到第二图像描述生成模型以及训练后的描述修正模型，之后再基于第一图像、第一图像描述生成模型、第二图像描述生成模型和描述修正模型执行更新阶段。
或者，如图11所示，将步骤802作为第一训练阶段，将步骤803作为第二训练阶段，将步骤401-402和步骤404-405作为第三训练阶段，将步骤201-204也作为第三训练阶段的一部分，即第三训练阶段包括步骤401-405以及步骤201-204。在执行第一训练阶段和第二训练阶段后，得到训练后的图图像描述生成模型，之后执行第三训练阶段，基于第二图像对应的多维度描述文本，训练图像描述生成模型和描述修正模型，将训练后的图像描述生成模型作为第一图像描述生成模型，复制得到第二图像描述生成模型和描述修正模型，再基于第一图像、第一图像描述生成模型、第二图像描述生成模型以及描述修正模型，更新第二图像描述生成模型。
本申请实施例通过第一大语言模型整合多个第三描述生成模型生成的多个维度的描述文本，得到多维度的组合描述文本，并通过人工标注的方式获取组合描述文本的多维度的质量分数。利用标注的质量分数训练多维度的描述修正模型，同时利用多维度的组合描述文本训练图像描述生成模型。通过描述修正模型对图像描述生成模型的结果进行质量检测和过滤，从而更新图像描述生成模型的性能，实现多维度描述文本的生成以及图像描述生成模型的自学习。
实验表明，本申请实施例提供的自学习方法具有卓越的性能，本申请实施例中的图像描述生成模型所生成的描述文本与图像之间的相关性可以达到78％，远远超过了相关技术中的描述生成模型。
图12是本申请实施例提供的一种模型处理装置的结构示意图。参见图12，该装置包括：
第一描述生成模块1201，用于通过第一图像描述生成模型生成第一图像对应的第一描述文本，并通过描述修正模型修正第一描述文本得到第一修正描述文本；
第二描述生成模块1202，用于通过第二图像描述生成模型生成第一图像对应的第二描述文本，通过描述修正模型获取第二描述文本的质量分数，以及通过所述描述修正模型修正第二描述文本得到第二修正描述文本；
模型更新模块1203，用于在第二描述文本的质量分数不满足质量条件的情况下，基于第一描述文本、第一修正描述文本、第二描述文本和第二修正描述文本，确定损失值，以减小所述损失值为目标，更新第二图像描述生成模型，所述更新后的第二图像描述生成模型用于根据输入图像生成描述该输入图像的文本，更新前的所述第二图像描述生成模型具有与所述第一图像描述生成模型相同的结构和状态。
在一些实施例中，参见图13，模型更新模块1203，包括：
第一更新单元1213，用于确定所述第一图像描述生成模型输出所述第一描述文本的第一概率、所述描述修正模型输出所述第一修正描述文本的第二概率、所述第二图像描述生成模型输出所述第二描述文本的第三概率和所述描述修正模型输出所述第二修正描述文本的第四概率；根据所述第一概率、第二概率、第三概率和第四概率，确定损失值，其中，所述损失值与第三概率与第一概率的比值的对数正相关，与第四概率与第二概率的比值的对数负相关；以损失值减小为目标，更新第二图像描述生成模型。
在一些实施例中，参见图13，模型更新模块1203，包括：
第二更新单元1223，用于在第二描述文本的质量分数满足质量条件的情况下，基于第一图像和第二描述文本，更新第二图像描述生成模型。
需要说明的是：上述实施例提供的模型处理装置，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将计算机设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的模型处理装置与模型处理方法实施例属于同一构思，其具体实现过程详见方法实施例，这里不再赘述。
本申请实施例还提供了一种计算机设备，该计算机设备包括处理器和存储器，存储器中存储有至少一条计算机程序，该至少一条计算机程序由处理器加载并执行，以实现上述实施例的模型处理方法中所执行的操作。
在一些实施例中，该计算机设备提供为终端。图14示出了本申请一个示例性实施例提供的终端1400的结构示意图。
终端1400包括有：处理器1401和存储器1402。
处理器1401可以包括一个或多个处理核心，比如4核心处理器、8核心处理器等。处理器1401可以采用DSP(Digital Signal Processing，数字信号处理)、FPGA(Field Programmable Gate Array，现场可编程门阵列)、PLA(Programmable Logic Array，可编程逻辑阵列)中的至少一种硬件形式来实现。处理器1401也可以包括主处理器和协处理器，主处理器是用于对在唤醒状态下的数据进行处理的处理器，也称CPU(Central Processing Unit，中央处理器)；协处理器是用于对在待机状态下的数据进行处理的低功耗处理器。在一些实施例中，处理器1401可以集成有GPU(Graphics Processing Unit，图像处理的交互器)，GPU用于负责显示屏所需要显示的内容的渲染和绘制。一些实施例中，处理器1401还可以包括AI(Artificial Intelligence，人工智能)处理器，该AI处理器用于处理有关机器学习的计算操作。
存储器1402可以包括一个或多个计算机可读存储介质，该计算机可读存储介质可以是非暂态的。存储器1402还可包括高速随机存取存储器，以及非易失性存储器，比如一个或多个磁盘存储设备、闪存存储设备。在一些实施例中，存储器1402中的非暂态的计算机可读存储介质用于存储至少一条计算机程序，该至少一条计算机程序用于被处理器1401所具有以实现本申请中方法实施例提供的模型处理方法。
在一些实施例中，终端1400还可以包括有：外围设备接口1403和至少一个外围设备。处理器1401、存储器1402和外围设备接口1403之间可以通过总线或信号线相连。各个外围设备可以通过总线、信号线或电路板与外围设备接口1403相连。在一些实施例中，外围设备包括：射频电路1404、显示屏1405、摄像头组件1406和电源1407中的至少一种。
外围设备接口1403可被用于将I/O(Input/Output，输入/输出)相关的至少一个外围设备连接到处理器1401和存储器1402。在一些实施例中，处理器1401、存储器1402和外围设备接口1403被集成在同一芯片或电路板上；在一些其他实施例中，处理器1401、存储器1402和外围设备接口1403中的任意一个或两个可以在单独的芯片或电路板上实现，本实施例对此不加以限定。
本领域技术人员可以理解，图14中示出的结构并不构成对终端1400的限定，可以包括比图示更多或更少的组件，或者组合某些组件，或者采用不同的组件布置。
在一些实施例中，该计算机设备提供为服务器。图15是本申请实施例提供的一种服务器的结构示意图，该服务器1500可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上处理器(Central Processing Units，CPU)1501和一个或一个以上的存储器1502，其中，所述存储器1502中存储有至少一条计算机程序，所述至少一条计算机程序由所述处理器1501加载并执行以实现上述各个方法实施例提供的方法。当然，该服务器还可以具有有线或无线网络接口、键盘以及输入输出接口等部件，以便进行输入输出，该服务器还可以包括其他用于实现设备功能的部件，在此不做赘述。
本申请实施例还提供了一种计算机可读存储介质，该计算机可读存储介质中存储有至少一条计算机程序，该至少一条计算机程序由处理器加载并执行，以实现上述实施例的模型处理方法所执行的操作。
本申请实施例还提供了一种计算机程序产品，包括计算机程序，计算机程序由处理器加载并执行，以实现如上述实施例的模型处理方法所执行的操作。
本领域普通技术人员可以理解实现上述实施例的全部或部分步骤可以通过硬件来完成，也可以通过程序来指令相关的硬件完成，所述的程序可以存储于一种计算机可读存储介质中，上述提到的存储介质可以是只读存储器，磁盘或光盘等。
以上所述仅为本申请实施例的可选实施例，并不用以限制本申请实施例，凡在本申请实施例的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (18)

  1. 一种图像描述生成模型处理方法，由电子设备执行，包括：
    通过第一图像描述生成模型生成第一图像对应的第一描述文本，并通过描述修正模型修正所述第一描述文本，得到第一修正描述文本；
    通过第二图像描述生成模型生成所述第一图像对应的第二描述文本，并通过所述描述修正模型获取所述第二描述文本的质量分数，以及通过所述描述修正模型修正所述第二描述文本得到第二修正描述文本；
    在所述第二描述文本的质量分数不满足质量条件的情况下，基于所述第一描述文本、所述第一修正描述文本、所述第二描述文本和所述第二修正描述文本，确定损失值，以减小所述损失值为目标，更新所述第二图像描述生成模型，得到更新后的第二图像描述生成模型，所述更新后的第二图像描述生成模型用于根据输入图像生成描述该输入图像的文本；其中，更新前的所述第二图像描述生成模型具有与所述第一图像描述生成模型相同的结构和状态。
  2. 根据权利要求1所述的方法，其中，所述基于所述第一描述文本、所述第一修正描述文本、所述第二描述文本和所述第二修正描述文本，确定损失值，包括：
    确定所述第一图像描述生成模型输出所述第一描述文本的第一概率、所述描述修正模型输出所述第一修正描述文本的第二概率、所述第二图像描述生成模型输出所述第二描述文本的第三概率和所述描述修正模型输出所述第二修正描述文本的第四概率；
    根据所述第一概率、第二概率、第三概率以及第四概率，确定所述损失值，其中，所述损失值与所述第三概率和所述第一概率的比值的对数正相关，与所述第四概率和所述第二概率的比值的对数负相关。
  3. 根据权利要求1或2所述的方法，其中，所述第一图像描述生成模型和所述第二图像描述生成模型是通过训练一个待训练的图像描述生成模型得到的，所述方法进一步包括：
    训练所述图像描述生成模型，得到所述第一图像描述生成模型和所述第二图像描述生成模型，所述待训练的图像描述生成模型包括：视觉模型、第一大语言模型和位于所述视觉模型和所述第一大语言模型之间的连接器；
    所述训练包括第一训练阶段和第二训练阶段中的至少一个，其中：
    所述第一训练阶段用于训练所述视觉模型和所述连接器；以及
    所述第二训练阶段用于训练所述第一大语言模型和所述连接器。
  4. 根据权利要求3所述的方法，其中，所述训练进一步包括：第三训练阶段；
    所述第三训练阶段包括：
    生成与第二图像对应的组合描述文本，其中，所述组合描述文本用于从多个维度描述所述第二图像；
    将所述第二图像和所述组合描述文本作为训练样本对，训练所述图像描述生成模型。
  5. 根据权利要求4所述的方法，其中，所述生成与第二图像对应的组合描述文本，包括：
    通过多个第三图像描述生成模型，分别生成第二图像对应的多个维度的描述文本；其中，每个维度的描述文本用于从该维度上描述所述第二图像；每个第三图像描述生成模型用于生成所述第二图像对应的一个维度的描述文本；
    通过第二大语言模型，组合所述多个维度的描述文本，得到组合描述文本。
  6. 根据权利要求4或5所述的方法，其中，所述组合描述文本为修正后的组合描述文本。
  7. 根据权利要求4至6任一项所述的方法，其中，所述描述修正模型为训练后的描述修正模型，所述方法进一步包括：
    获取为所述组合描述文本标注的质量分数；
    基于所述组合描述文本以及和所述组合描述文本的质量分数，训练所述描述修正模型。
  8. 根据权利要求7所述的方法，其中，所述质量分数包括所述多个维度上每个维度的子分数，每个子分数表示所述组合描述文本在该子分数对应的维度上描述所述第二图像的质量。
  9. 根据权利要求4至8任一项所述的方法，其中，所述多个维度包括：主题、背景、风格、构图、文化和情感中的至少两个。
  10. 根据权利要求1至9任一项所述的方法，进一步包括：
    在所述第二描述文本的质量分数满足所述质量条件的情况下，基于所述第一图像和所述第二描述文本，更新所述第二图像描述生成模型。
  11. 根据权利要求1至10任一项所述的方法，其中，所述质量条件包括质量分数阈值，所述方法还包括：
    在所述第二描述文本的质量分数小于或等于所述质量分数阈值的情况下，确定所述第二描述文本不满足所述质量条件。
  12. 根据权利要求1至10任一项所述的方法，其中，所述质量条件包括：质量排序阈值；所述方法还包括：
    当所述第二描述文本的质量分数在多个第二描述文本按照质量分数从高到低的排序中的位置在所述质量排序阈值之后时，确定所述第二描述文本不满足所述质量条件；其中，所述多个第二描述文本是所述第二图像描述模型生成的、多个第一图像分别对应的第二描述文本质量分数。
  13. 根据权利要求11或12所述的方法，其中，所述第二描述文本的质量分数包括N个维度的子分数，N为大于1的整数，每个子分数表示所述第二描述文本在该子分数对应的维度上描述所述第一图像的质量。
  14. 根据权利要求1至13任一项所述的方法，进一步包括：
    通过更新后的所述第二图像描述生成模型，生成样本图像对应的样本描述文本；
    将所述样本图像和所述样本描述文本作为训练样本，训练图像生成模型。
  15. 一种图像描述生成模型处理装置，包括：
    第一描述生成模块，用于通过第一图像描述生成模型生成第一图像对应的第一描述文本，并通过描述修正模型修正所述第一描述文本得到第一修正描述文本；
    第二描述生成模块，用于通过第二图像描述生成模型生成所述第一图像对应的第二描述文本，通过所述描述修正模型获取所述第二描述文本的质量分数，以及通过所述描述修正模型修正所述第二描述文本得到第二修正描述文本；
    模型更新模块，用于在所述第二描述文本的质量分数不满足质量条件的情况下，基于所述第一描述文本、所述第一修正描述文本、所述第二描述文本和所述第二修正描述文本，确定损失值，以减少所述损失值为目标，更新所述第二图像描述生成模型，所述更新后的第二图像描述生成模型用于根据输入图像生成描述该输入图像的文本，更新前的所述第二图像描述生成模型具有与所述第一图像描述生成模型相同的结构和状态。
  16. 一种计算机设备，所述计算机设备包括处理器和存储器，所述存储器中存储有至少一条计算机程序，所述至少一条计算机程序由所述处理器加载并执行，以实现如权利要求1至14任一项所述的模型处理方法所执行的操作。
  17. 一种计算机可读存储介质，所述计算机可读存储介质中存储有至少一条计算机程序，所述至少一条计算机程序由处理器加载并执行，以实现如权利要求1至14任一项所述的模型处理方法所执行的操作。
  18. 一种计算机程序产品，包括计算机程序，所述计算机程序由处理器加载并执行，以实现如权利要求1至14任一项所述的模型处理方法所执行的操作。
PCT/CN2025/087426 2024-06-05 2025-04-07 图像描述生成模型处理方法、装置、计算机设备及存储介质 Pending WO2025251771A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202410729166.0A CN121072613A (zh) 2024-06-05 2024-06-05 模型处理方法、装置、计算机设备及存储介质
CN202410729166.0 2024-06-05

Publications (1)

Publication Number Publication Date
WO2025251771A1 true WO2025251771A1 (zh) 2025-12-11

Family

ID=97839013

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2025/087426 Pending WO2025251771A1 (zh) 2024-06-05 2025-04-07 图像描述生成模型处理方法、装置、计算机设备及存储介质

Country Status (2)

Country Link
CN (1) CN121072613A (zh)
WO (1) WO2025251771A1 (zh)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111813978A (zh) * 2019-04-12 2020-10-23 北京京东尚科信息技术有限公司 一种图像描述模型的生成方法、生成装置和存储介质
US20220004719A1 (en) * 2019-04-24 2022-01-06 Fujifilm Corporation Learning method and device, program, learned model, and text generation device
CN115147585A (zh) * 2022-07-25 2022-10-04 济南浪潮数据技术有限公司 一种图像描述生成方法、装置、设备及存储介质
CN117494722A (zh) * 2023-10-09 2024-02-02 广发证券股份有限公司 一种信息处理方法、装置、设备及计算机可读存储介质
CN117671680A (zh) * 2023-11-14 2024-03-08 平安科技(上海)有限公司 基于图像的生成方法、装置、设备及介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111813978A (zh) * 2019-04-12 2020-10-23 北京京东尚科信息技术有限公司 一种图像描述模型的生成方法、生成装置和存储介质
US20220004719A1 (en) * 2019-04-24 2022-01-06 Fujifilm Corporation Learning method and device, program, learned model, and text generation device
CN115147585A (zh) * 2022-07-25 2022-10-04 济南浪潮数据技术有限公司 一种图像描述生成方法、装置、设备及存储介质
CN117494722A (zh) * 2023-10-09 2024-02-02 广发证券股份有限公司 一种信息处理方法、装置、设备及计算机可读存储介质
CN117671680A (zh) * 2023-11-14 2024-03-08 平安科技(上海)有限公司 基于图像的生成方法、装置、设备及介质

Also Published As

Publication number Publication date
CN121072613A (zh) 2025-12-05

Similar Documents

Publication Publication Date Title
CN111625635B (zh) 问答处理方法、装置、设备及存储介质
JP7430820B2 (ja) ソートモデルのトレーニング方法及び装置、電子機器、コンピュータ可読記憶媒体、コンピュータプログラム
US20200057811A1 (en) Hybrid Natural Language Understanding
CN112101042B (zh) 文本情绪识别方法、装置、终端设备和存储介质
CN118395957A (zh) 基于大语言模型的问答方法、装置、电子设备及存储介质
CN118568240A (zh) 基于大模型的问答信息生成方法、装置、电子设备及介质
CN114676241A (zh) 多轮对话的意图识别方法、装置、计算机设备及存储介质
CN118069813A (zh) 一种对话交互处理方法、装置、终端设备及存储介质
CN109165286A (zh) 自动问答方法、装置及计算机可读存储介质
CN115862035A (zh) 一种训练样本的扩充方法及装置
CN119783672B (zh) 语料扩展方法、设备及存储介质
CN114781477A (zh) 一种训练文本匹配模型的方法、装置以及存储介质
CN115858859A (zh) 视频搜索方法、装置及计算机可读存储介质
CN115391542A (zh) 分类模型的训练方法、文本分类方法、装置及设备
CN112347196A (zh) 基于神经网络的实体关系抽取方法及装置
CN110347807B (zh) 问题信息处理方法及装置
CN114218939A (zh) 文本分词方法、装置、设备及存储介质
CN120747672A (zh) 一种大模型的训练方法、装置、设备以及存储介质
CN118779533A (zh) 标签增强的实时推荐方法、装置、电子设备及存储介质
CN119293324A (zh) 基于大模型的内容推荐数据处理方法、装置、设备及介质
WO2024109597A1 (zh) 文本合并判断模型的训练方法和文本合并判断方法
CN114116996B (zh) 问题判定方法、装置、设备和存储介质
CN114328797B (zh) 内容搜索方法、装置、电子设备、存储介质及程序产品
CN115659998A (zh) 辅助翻译方法、装置、电子设备及存储介质
CN116680401A (zh) 文档处理方法、文档处理装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 25818831

Country of ref document: EP

Kind code of ref document: A1