CN116309135A - Diffusion model processing method and device and picture processing method and device - Google Patents

Abstract

The embodiment of this specification provides a diffusion model processing method and device, and a picture processing method and device, wherein the diffusion model processing method includes determining the time step set of the diffusion model and the time step interval corresponding to the time step set; determining from the time step set The first time step, and determine the target time step corresponding to the first time step according to the time step interval; input the noise-added picture corresponding to the first time step and the target time step into the diffusion model, and obtain the prediction noise corresponding to the noise-added picture; according to The target noise and prediction noise corresponding to the noised image are processed by the diffusion model. In this method, the time step set is divided into time step intervals. When the diffusion model is trained later, the diffusion model can share the time step condition in a time step interval, that is, the first time step shares the target in its corresponding time step interval. Time step reduces the time step condition, greatly reduces the training burden and improves the model training performance.

Description

Diffusion model processing method and device, image processing method and device

technical field

The embodiments of this specification relate to the field of computer technology, and in particular to a diffusion model processing method and device, an image processing method and device, a computing device, and a computer-readable storage medium.

Background technique

Diffusion model is a generative model, which can gradually destroy the picture distribution into Gaussian noise by constructing a Markov chain; then learn the reverse distribution through the network to gradually denoise and generate pictures. Diffusion models have achieved amazing results in a variety of tasks, including but not limited to multimodal generation tasks of text-to-image generation.

However, traditional diffusion models usually learn all single-step transition probabilities through a network, and control the learning of different probabilities through time-step conditions. This method will cause a heavy training burden, resulting in insufficient model capability and poor training effect.

Contents of the invention

In view of this, the embodiment of this specification provides a diffusion model processing method. One or more embodiments of this specification also relate to a diffusion model processing device, an image processing method, an image processing device, a computing device, a computer-readable storage medium, and a computer program to solve current problems. There are technical flaws in the technology.

According to the first aspect of the embodiments of this specification, a diffusion model processing method is provided, including:

Determining a time step set of the diffusion model and a time step interval corresponding to the time step set;

Determine a first time step from the time step set, and determine a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any in the time step set a time step;

Inputting the noise-added picture corresponding to the first time step and the target time step into a diffusion model to obtain the predicted noise corresponding to the noise-added picture;

Processing the diffusion model according to the target noise corresponding to the noised picture and the predicted noise.

According to the second aspect of the embodiments of this specification, a diffusion model processing device is provided, including:

An interval division module configured to determine a time step set of the diffusion model and a time step interval corresponding to the time step set;

A target time step determination module configured to determine a first time step from the set of time steps, and determine a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the set of time steps;

The first model prediction module is configured to input the noise-added picture corresponding to the first time step and the target time step into a diffusion model to obtain the predicted noise corresponding to the noise-added picture;

The model processing module is configured to process the diffusion model according to the target noise corresponding to the noise-added picture and the predicted noise.

According to a third aspect of the embodiments of this specification, there is provided an image processing method, including:

Determining the target noise-added picture, inputting the target noise-added picture into a diffusion model, and obtaining the predicted noise corresponding to the target noise-added picture;

determining a denoised target picture according to the target noise-added picture and the predicted noise corresponding to the target noise-added picture,

Wherein, the diffusion model is obtained through the above-mentioned diffusion model processing method.

According to a fourth aspect of the embodiments of this specification, an image processing device is provided, including:

The second model prediction module is configured to determine the target noise-added picture, input the target noise-added picture into the diffusion model, and obtain the predicted noise corresponding to the target noise-added picture;

The target picture determination module is configured to determine a denoised target picture according to the target noise-added picture and the predicted noise corresponding to the target noise-added picture,

Wherein, the diffusion model is obtained through the above-mentioned diffusion model processing method.

According to a fifth aspect of the embodiments of this specification, a computing device is provided, including:

memory and processor;

The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned diffusion model processing method or image processing method are implemented.

According to a sixth aspect of the embodiments of the present specification, there is provided a computer-readable storage medium, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned diffusion model processing method or image processing method are implemented.

According to a seventh aspect of the embodiments of the present specification, a computer program is provided, wherein, when the computer program is executed in a computer, the computer is instructed to execute the steps of the above-mentioned diffusion model processing method or image processing method.

An embodiment of this specification implements a diffusion model processing method, including determining a time step set of the diffusion model and a time step interval corresponding to the time step set; determining the first time step from the time step set, and according to The time step interval determines the target time step corresponding to the first time step, wherein the first time step is any time step in the time step set; the noise addition corresponding to the first time step The picture and the target time step are input into a diffusion model to obtain the predicted noise corresponding to the noise-added picture; and the diffusion model is processed according to the target noise corresponding to the noise-added picture and the predicted noise.

Specifically, the diffusion model processing method divides the time step set into time step intervals, and when the diffusion model is trained later, the diffusion model can share the time step conditions in a time step interval, that is, the first time step shares its corresponding The target time step within the time step interval reduces the time step conditions, greatly reduces the training burden, and improves the model training performance.

Description of drawings

FIG. 1 is a schematic structural diagram of a denoising diffusion probability model provided by an embodiment of this specification;

Fig. 2 is a schematic diagram of a specific implementation scenario of a diffusion model processing method provided by an embodiment of this specification;

Fig. 3 is a flowchart of a diffusion model processing method provided by an embodiment of this specification;

Fig. 4 is a flow chart of an image processing method provided by an embodiment of this specification;

Fig. 5 is a schematic structural diagram of a diffusion model processing device provided by an embodiment of this specification;

Fig. 6 is a schematic structural diagram of an image processing device provided by an embodiment of this specification;

Fig. 7 is a structural block diagram of a computing device provided by an embodiment of this specification.

Detailed ways

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the specification. However, this specification can be implemented in many other ways different from those described here, and those skilled in the art can make similar extensions without violating the connotation of this specification, so this specification is not limited by the specific implementations disclosed below.

Terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only, and are not intended to limit one or more embodiments of this specification. As used in one or more embodiments of this specification and the appended claims, the singular forms "a", "the", and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and/or" used in one or more embodiments of the present specification refers to and includes any or all possible combinations of one or more associated listed items.

It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present specification, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, the first may also be referred to as the second, and similarly, the second may also be referred to as the first without departing from the scope of one or more embodiments of the present specification. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

First, terms and terms involved in one or more embodiments of this specification are explained.

Diffusion Model: Diffusion Model, an image generation method for generating a variety of high-quality images.

Interval division: Divide the time steps in the Markov chain of the diffusion model into intervals.

Denoising Diffusion Probabilistic Model: DDPM, Denoising Diffusion Probabilistic Model.

In this specification, a diffusion model processing method is provided. One or more embodiments of this specification also relate to a diffusion model processing device, an image processing method, an image processing device, a computing device, a computer-readable storage medium, and a computer program. The following The detailed description will be given in detail one by one in the embodiment.

Referring to FIG. 1 , FIG. 1 shows a schematic structural diagram of a denoising diffusion probability model provided according to an embodiment of the present specification.

In practical applications, the basic idea of the denoising diffusion probability model is to construct a Markov chain, and uniformly model the transition probabilities of all time steps through a network, where the time step conditions are used as the conditional input of the network.

The x in Figure 1 can be understood as a picture, and t can be understood as a time step, then x _t can be understood as the picture of the t-th time step. If the denoising diffusion probability model is applied to the image denoising scene, the x _t can be understood as the noise-added image of the t-th time step, and x _t-1 can be understood as the noise-added image of the t-th time step After the image is denoised, the noise-added image of the t-1th time step is obtained; by analogy, the final x ₀ can be understood as the image obtained after complete denoising.

Referring to FIG. 2 , FIG. 2 shows a schematic diagram of a specific implementation scenario of a diffusion model processing method provided according to an embodiment of the present specification.

Figure 2 includes a cloud-side device 202 and a terminal-side device 204, wherein the cloud-side device 202 can be understood as a cloud server, of course, in another feasible solution, the cloud-side device 202 can also be replaced by a physical server; The side device 204 includes but is not limited to a desktop computer, a notebook computer, etc.; for ease of understanding, in the embodiments of this specification, the cloud side device 202 is a cloud server, and the terminal side device 204 is a notebook computer as an example for detailed introduction.

The application of the diffusion model processing method provided in the embodiment of this specification to the image denoising scene will be described in detail.

During specific implementation, the diffusion model training is performed on the cloud-side device 202, wherein the diffusion model can be understood as a diffusion model (TSDM) with reduced time step conditions.

As shown in FIG. 2 , for the specific structure of the diffusion model in FIG. 2 , please refer to the structure of the denoising diffusion probability model in FIG. 1 .

In the denoising diffusion probability model in Figure 1, it is necessary to use a network ∈ _θ (x _t ,t) to model T transition probabilities, where t is the time step condition, which is used to indicate where the network is currently A transition probability is modeled.

In practical applications, the denoising diffusion probability model requires a sufficiently large number of diffusion steps (ie t) to completely destroy the image signal (ie add noise), so the number of transition probabilities that need to be modeled by a single network is usually very large, which will lead to The training burden of the network (i.e., the denoising diffusion probability model) is large. In order to reduce the training burden of the network, it can be achieved by reducing the number of diffusion steps, but at the same time, the signal-to-noise ratio of x _T is not low enough, so that the final sampling quality is greatly reduced.

The diffusion model provided by the embodiment of this specification in Fig. 2 divides the number of diffusion steps into an interval by dividing the number of diffusion steps into intervals, and only reduces the time step condition ( That is, the number of t), the diffusion model is obtained by training to reduce the number of transition probabilities that the network needs to model, thereby reducing the burden on the network.

When the end-side device 204 needs to use the diffusion model, it can call the diffusion model obtained after the training of the cloud-side device 202 for functional use; The diffusion model trained in the cloud-side device 202 is deployed in the device-side device 204 . Deployment and implementation are performed according to actual applications, and no limitation is made here.

The diffusion model processing method provided by the embodiment of this specification divides the set of time steps into time step intervals, and when the diffusion model is trained later, the diffusion model can share the time step condition in a time step interval, that is, the average time step of the first time step is Sharing the target time step in its corresponding time step interval reduces the time step conditions, greatly reduces the training burden, and improves the model training performance.

Referring to FIG. 3 , FIG. 3 shows a flow chart of a method for processing a diffusion model according to an embodiment of the present specification, which specifically includes the following steps.

Step 302: Determine a time step set of the diffusion model and a time step interval corresponding to the time step set.

Wherein, the diffusion model can be understood as the diffusion model (TSDM) that reduces the time step condition of the above-mentioned embodiment; the time step set includes multiple time steps, and a single time step can be understood as T, T-1, etc. in the above-mentioned Fig. 1 , It can also be understood as the number of diffusion steps described in the above embodiments. For example, if the number of diffusion steps is 10, it can be understood that the set of time steps includes 10 time steps.

In order to reduce the number of time step conditions during the subsequent diffusion model training, it can be realized by dividing the time steps in the time step set into intervals. The specific implementation is as follows:

The time step set for determining the diffusion model and the time step interval corresponding to the time step set include:

A set of time steps of the diffusion model is determined, and time steps in the set of time steps are divided into intervals according to preset division conditions to obtain intervals of time steps corresponding to the set of time steps.

Wherein, the preset division condition can be set according to the actual application, which is not limited in the embodiment of this specification. For example, the preset division condition can be understood as dividing every 50 time steps into a time step interval.

For example, for example, the time step set of the diffusion model includes 1000 time steps, and the preset division condition is to divide every 50 time steps into a time step interval.

Then, determine the time step set of the diffusion model, and divide the time steps in the time step set according to the preset division conditions to obtain the time step interval corresponding to the time step set; it can be understood as determining the diffusion model The time step set [1000 time steps], according to the preset division conditions: divide every 50 time steps into a time step interval; perform interval division, and obtain the time step interval corresponding to the time step set: 20 time steps interval. Wherein, when dividing the time steps in the time step set into intervals, it needs to be divided according to the time sequence, so the time steps in each time step interval corresponding to the obtained time step set are also sorted according to the time sequence. For example, the first time step interval contains 0-50 time steps arranged in time series.

Step 304: Determine a first time step from the set of time steps, and determine a target time step corresponding to the first time step according to the time step interval, wherein the first time step is the set of time steps any time step in .

Specifically, the first time step may be understood as any time step in the time step set, such as the first time step, the second time step, the third time step, or the nth time step.

After the first time step is determined from the time step set, the target time step corresponding to the first time step can be determined according to the time step interval corresponding to the time step set.

In practical applications, since the time steps in the time step set are divided into intervals according to preset division conditions, each time step in the time step set has its corresponding time step interval. Then any time step in the time step set can determine the corresponding target time step from the corresponding time step interval.

In specific implementation, the time step corresponding to the interval endpoint of the time step interval corresponding to each time step can be used as the target time step corresponding to each time step, then when performing diffusion model training later, each time step can be used as The target time step corresponding to the interval endpoint of the corresponding time step interval is used as the time step condition, so as to reduce the time step condition during the diffusion model training, and improve the training efficiency and training effect of the diffusion model. The specific implementation is as follows:

The determining the target time step corresponding to the first time step according to the time step interval includes:

An interval endpoint of the time step interval is determined, and a target time step corresponding to the first time step is determined according to the interval endpoint.

Following the above example, if the first time step is the 20th time step, and the time step interval corresponding to the first time step is 0-50, then determine the target time step corresponding to the first time step according to the interval endpoint of the time step interval For: 0 time steps or 50 time steps.

Since each time step interval includes two interval endpoints, determining the target time step corresponding to the first time step according to the interval endpoints of the time step interval can include two situations, one is when the interval endpoint of the time step interval is the left endpoint of the interval In the case of , the target time step corresponding to the first time step is determined according to the interval endpoint of the time step interval, which is the left endpoint of the interval of the time step interval; similarly, the other is that the interval endpoint of the time step interval is the right endpoint of the interval In the case of , the target time step corresponding to the first time step is determined according to the interval endpoint of the time step interval, and it is the right endpoint of the interval of the time step interval.

First, taking the interval endpoint of the time step interval as the left endpoint of the interval as an example, the determination of the target time step corresponding to the first time step according to the interval endpoint is described in detail. The specific implementation method is as follows:

The determining the interval endpoint of the time step interval, and determining the target time step corresponding to the first time step according to the interval endpoint includes:

Determine the interval left endpoint of the time step interval, and determine the left endpoint of the interval as the target time step corresponding to the first time step, wherein the interval right endpoint of the time step interval is the next time step interval including the left endpoint of .

The first time step is the 53rd time step, and the time step interval corresponding to the first time step is 50-100 as an example for illustration.

The left endpoint of the time step interval 50-100 is 50, then the 50th time step in the time step interval can be used as the target time step corresponding to the first time step. That is, as long as the first time step belongs to the time step interval 50-100, then the target time step corresponding to the first time step is the left endpoint of the interval of the time step interval: the 50th time step.

Using the above example, if the time step set includes 1000 time steps and is divided into 20 time step intervals, the time step interval including 50-100 time steps can be represented by [50, 100), that is, the time step The right endpoint of the interval is the left endpoint included in the next time step interval, that is, the time step interval including 100-150 can be represented by [100, 150).

The diffusion model processing method provided by the embodiment of this specification can use the time step corresponding to the left end point of the time step interval corresponding to each time step as the target time step corresponding to each time step, then when performing diffusion model training later , the target time step corresponding to the left endpoint of the time step interval corresponding to each time step can be used as the time step condition, so as to reduce the time step condition during the diffusion model training and improve the training efficiency and training effect of the diffusion model.

Secondly, taking the interval endpoint of the time step interval as the right endpoint of the interval as an example, the determination of the target time step corresponding to the first time step according to the interval endpoint is described in detail. The specific implementation method is as follows:

The determining the interval endpoint of the time step interval, and determining the time step corresponding to the first time step according to the interval endpoint includes:

Determine the right endpoint of the interval of the time step interval, and determine the right endpoint of the interval as the target time step corresponding to the first time step, wherein the left endpoint of the interval of the time step interval is the last time step interval including the right endpoint of .

Following the above example, the first time step is still the 53rd time step, and the time step interval corresponding to the first time step is 50-100 as an example for illustration.

The right endpoint of the time step interval 50-100 is 100, then the 100th time step in the time step interval can be used as the target time step corresponding to the first time step. That is, as long as the first time step belongs to the time step interval 50-100, then the target time step corresponding to the first time step is the right endpoint of the interval of the time step interval: the 100th time step.

Still using the above example, if the time step set includes 1000 time steps and is divided into 20 time step intervals, the time step interval including 50-100 time steps can be represented by (50, 100], that is, the time The left endpoint of the step interval is the right endpoint included in the previous time step interval, that is, the time step interval including 100-150 can be represented by [50, 100).

The diffusion model processing method provided in the embodiment of this specification can use the time step corresponding to the right end point of the time step interval corresponding to each time step as the target time step corresponding to each time step, then when performing diffusion model training later , the target time step corresponding to the left endpoint of the time step interval corresponding to each time step can be used as the time step condition, so as to reduce the time step condition during the diffusion model training and improve the training efficiency and training effect of the diffusion model.

Of course, it does not rule out that an intermediate time step in the time step interval may be used as the target time step corresponding to the first time step. In practical applications, starting from an observation, it can be found that the network inputs different time steps t. If the value of the time step t is relatively close, the network prediction is also very close to the same input (this is caused by the continuity of the network) , so reducing the time step t can reduce the network burden; and the values of the time steps in the same time step interval are relatively close, so choosing any time step in each time step interval can be used as the first time step corresponding target time step.

Step 306: Input the noise-added picture corresponding to the first time step and the target time step into a diffusion model to obtain the predicted noise corresponding to the noise-added picture.

Wherein, the noise-added picture corresponding to the first time step can be understood as a noise-added picture that is noise-added at the first time step in the forward process of the diffusion model.

Specifically, the diffusion model is divided into two stages, including the forward process and the reverse process. The forward process is to construct a Markov chain, and gradually add noise to the picture signal to become a noise signal, that is, a picture with noise. Specifically, first construct a discrete Markov chain {x ₀ , x ₁ , .., x _N }, then the transition probability of the forward process can be expressed as Equation 1:

{β₀，β₁，...，β_N}是一个预先设计的噪声序列。in,

{β ₀ , β ₁ , ..., β _N } is a predesigned noise sequence.

According to formula 1, it can be seen that when it is forward to step T, its distribution is shown in formula 2:

This distribution is very close to the standard normal distribution, so the inverse generative process can sample directly from a Gaussian distribution.

Then, according to the above formula 1 and formula 2, the noise-added picture corresponding to each time step in the forward process of the diffusion model can be obtained; the specific implementation method is as follows:

Before inputting the noise-added picture corresponding to the first time step and the target time step into the diffusion model, and obtaining the predicted noise corresponding to the noise-added picture, it also includes:

determining the initial picture and the target noise corresponding to the first time step;

Determine a noise-added picture corresponding to the first time step and a target noise corresponding to the noise-added picture according to the initial picture and the target noise.

Among them, the initial picture can be understood as an unnoised picture of any size and any format, or a noisy picture output by the previous time step of the first time step.

Specifically, when the first time step is the original picture, determine the initial picture and the target noise corresponding to the first time step, add the target noise to the initial picture, and then generate the noise corresponding to the first time step The noise-added picture and the target noise corresponding to the noise-added picture. In the case that the first time step is not the original picture, the initial picture can be understood as the noised picture output by the previous time step of the first time step, then determine the initial picture and the target noise corresponding to the first time step , adding the target noise to the initial picture, the noise-added picture corresponding to the first time step and the target noise corresponding to the noise-added picture can be generated. Of course, in the process of adding noise to the initial image in the forward process of the diffusion model, it is necessary not only to add noise according to the target noise corresponding to the first time step, but also to consider the noise intensity corresponding to the first time step, etc. Noise process The embodiment of this specification does not make any limitation.

During specific implementation, after inputting the noise-added picture corresponding to the first time step and the target time step corresponding to the first time step into the diffusion model, the predicted noise corresponding to the noise-added picture can be obtained.

Step 308: Process the diffusion model according to the target noise corresponding to the noise-added picture and the predicted noise.

Specifically, processing the diffusion model according to the target noise corresponding to the noise-added picture and the predicted noise includes:

Calculate the noise loss function according to the target noise corresponding to the noise-added picture and the predicted noise, and adjust the network parameters of the diffusion model according to the noise loss function, and obtain the obtained results when the preset training end condition is met. The diffusion model described above.

In practical applications, after determining the target noise corresponding to the noise-added image and the predicted noise output by the diffusion model, the noise loss function can be calculated according to the difference between the target noise and the predicted noise, and then adjusted according to the noise loss function The network parameters of the diffusion model realize the training of the diffusion model, and obtain the trained diffusion model under the condition that the preset training end condition is satisfied. Wherein, the preset training end condition can be understood as that the number of training times meets the preset number of times threshold (such as 10,000 times or 20,000 times); or the model performance (such as precision, accuracy, etc.) of the diffusion model meets the preset performance threshold, etc. .

In specific implementation, the calculation process of the noise loss function can be referred to the following formula 3:

加噪的强度由t决定，具体的参数α_t是预先设计好的。将加噪后的图片x_t和/>

输入进扩散模型，将其输出(即预测噪声)和真实的噪声∈计算损失(训练的目标是让神经网络可以从一个加噪图片x_t和加噪的强度(由t决定)中预测加在图片上的噪声∈)。Among them, formula 3 represents the above-mentioned noise loss function for training. During the training process, t is first sampled, and t obeys a uniform distribution from 0 to T, which is equivalent to randomly selecting an integer from 0 to T as t. Next, sampling x ₀ is equivalent to sampling a real picture from the data set, and finally sampling a noise ∈ from the standard normal distribution. After sampling, add the noise ∈ to the real picture x ₀ to get the noised picture

The strength of adding noise is determined by t, and the specific parameter α _t is pre-designed. The noised image x _t and />

Input into the diffusion model, its output (ie predicted noise) and the real noise ∈ calculation loss (the goal of training is to allow the neural network to predict the added value from a noised image x _t and the intensity of the noise (determined by t) noise ∈ on the image).

in,

T is a sequence of time step intervals T={t ₀ ,t ₁ ,…,t _n }, and the elements inside are the left endpoints of the time step intervals. For example, if 1000 ts are divided into 20 intervals, T={0,50,100,...,1000}. f _T (t) is a function, the input is the current time step t, and the output is the left endpoint of the time step interval to which t belongs. For example, the time step interval corresponding to t=53 is 50-100, so f _T (53)=50.

In summary, the single-step transition probability of the reverse process of the diffusion model can be referred to the following formula 4, which is the reverse denoising process of the diffusion model:

方差为/>

的高斯分布，而该方差是预先设置的。均值/>

中β_t是一系列固定的参数，指示了前向过程中不同的t加噪的强度，α_t是用β_t算出来的参数：/>

是神经网络，输入是加噪的图片x_t，和t对应的左端点f_T(t)，输出即是加在x_t上的噪声。Among them, formula 3 represents the single-step transition probability of the reverse process of the diffusion model, that is, under the condition of given x _t , x _t-1 obeys a mean value

Variance is />

Gaussian distribution of , and the variance is preset. Mean />

Among them, β _t is a series of fixed parameters, indicating the strength of different t noise addition in the forward process, and α _t is a parameter calculated by β _t : />

is a neural network, the input is the noise-added image x _t , and the left endpoint f _T (t) corresponding to t, and the output is the noise added to x _t .

The diffusion model processing method provided in the embodiment of this specification divides the time step set into time step intervals, and when the diffusion model is trained subsequently, the diffusion model can share the time step condition in a time step interval, that is, the first time step Both share the target time step in the corresponding time step interval, which reduces the time step conditions, greatly reduces the training burden, and improves the model training performance.

In addition, after the diffusion model is obtained through training, the diffusion model can be used in practice. The specific implementation is as follows:

According to the target noise corresponding to the noise-added picture and the predicted noise, after processing the diffusion model, further include:

Determining the target noise-added picture, inputting the target noise-added picture into a diffusion model, and obtaining the predicted noise corresponding to the target noise-added picture;

A denoised target picture is determined according to the target noise-added picture and the predicted noise corresponding to the target noise-added picture.

Wherein, the target noise-added picture can be understood as a noise-added picture of any size and any format.

In another practicable manner, the target noise-added picture can be understood as a video frame, that is, the diffusion model can denoise the noise-added video frame to obtain a clear and accurate video frame. The specific implementation is as follows:

The noise-added picture of the determined target includes:

A set of noised video frames is determined, and any video frame in the set of video frames is determined as a target noised picture.

Wherein, the noise-added video frame set includes multiple video frames of any type added with various noises.

Specifically, after the noise-added video frame set is determined, any video frame in the video frame set may be used as a target noise-added picture for subsequent denoising processing.

After the target noise-added picture is determined, the target noise-added picture can be directly input into the diffusion model obtained through the above-mentioned diffusion model processing method training, and the diffusion model can output the corresponding prediction noise of the target noise-added picture; then By removing the noise in the target noise-added picture according to the predicted noise corresponding to the target noise-added picture, the denoised target picture can be accurately obtained.

In another realizable embodiment, a diffusion model processing method provided in this specification can also be applied to the field of image generation by AI (Artificial Intelligence, artificial intelligence) for text; that is, on the basis that the diffusion model is a picture generation model, the Text is used as a condition for AI image generation; in specific implementation, the text condition can use a pre-trained encoder to encode the text condition, and then combine the encoding with the diffusion model through the self-attention mechanism. In the diffusion model On the basis of image generation, the code is combined to generate an AI image regulated by the text condition.

For example, the text condition is: generate an AI image of a teddy bear skateboarding in Times Square, then input the text into the AI image generation model (ie text encoder + diffusion model), and the text encoder of the AI image generation model will The text is encoded, and then the encoding is combined with the diffusion model through self-attention to generate AI images. Finally, the AI image generation model can output an AI image of a teddy bear skateboarding in Times Square.

In practical applications, this method can generate not only AI images but also ordinary two-dimensional images, etc. The specific conditions are set according to actual applications, and this specification does not make any restrictions on this.

Referring to FIG. 4 , FIG. 4 shows a flowchart of an image processing method provided by an embodiment of this specification, which specifically includes the following steps.

Step 402: Determine the target noise-added picture, input the target noise-added picture into the diffusion model, and obtain the predicted noise corresponding to the target noise-added picture.

Step 404: According to the target noise-added picture and the predicted noise corresponding to the target noise-added picture, determine the denoised target picture,

Wherein, the diffusion model is obtained through the above-mentioned diffusion model processing method.

Specifically, for the specific implementation steps of the image processing method, refer to the detailed introduction of the diffusion model processing method in the foregoing embodiment, which will not be described in detail in the embodiment of this specification.

The image processing method provided by the embodiment of this specification can quickly and accurately denoise the image by reducing the time step condition training and high-performance diffusion model, obtain the denoised target image, and greatly improve the image denoising performance.

Corresponding to the foregoing method embodiments, this specification also provides an embodiment of a diffusion model processing device. FIG. 5 shows a schematic structural diagram of a diffusion model processing device provided by an embodiment of this specification. As shown in Figure 5, the device includes:

The interval division module 502 is configured to determine a time step set of the diffusion model and a time step interval corresponding to the time step set;

The target time step determining module 504 is configured to determine a first time step from the time step set, and determine a target time step corresponding to the first time step according to the time step interval, wherein the first time step Step is any time step in the set of time steps;

The first model prediction module 506 is configured to input the noise-added picture corresponding to the first time step and the target time step into a diffusion model to obtain the predicted noise corresponding to the noise-added picture;

The model processing module 508 is configured to process the diffusion model according to the target noise corresponding to the noise-added picture and the predicted noise.

Optionally, the device also includes:

A noise adding module configured to:

determining the initial picture and the target noise corresponding to the first time step;

Determine a noise-added picture corresponding to the first time step and a target noise corresponding to the noise-added picture according to the initial picture and the target noise.

Optionally, the interval division module 502 is further configured to:

A set of time steps of the diffusion model is determined, and time steps in the set of time steps are divided into intervals according to preset division conditions to obtain intervals of time steps corresponding to the set of time steps.

Optionally, the target time step determination module 504 is further configured to:

An interval endpoint of the time step interval is determined, and a target time step corresponding to the first time step is determined according to the interval endpoint.

Optionally, the target time step determination module 504 is further configured to:

Determine the interval left endpoint of the time step interval, and determine the left endpoint of the interval as the target time step corresponding to the first time step, wherein the interval right endpoint of the time step interval is the next time step interval including the left endpoint of .

Optionally, the target time step determination module 504 is further configured to:

Determine the right endpoint of the interval of the time step interval, and determine the right endpoint of the interval as the target time step corresponding to the first time step, wherein the left endpoint of the interval of the time step interval is the last time step interval including the right endpoint of .

Optionally, the model processing module 508 is further configured to:

Calculate the noise loss function according to the target noise corresponding to the noise-added picture and the predicted noise, and adjust the network parameters of the diffusion model according to the noise loss function, and obtain the obtained result when the preset training end condition is met. The diffusion model described above.

Optionally, the device also includes:

a denoising module configured to:

Determining the target noise-added picture, inputting the target noise-added picture into a diffusion model, and obtaining the predicted noise corresponding to the target noise-added picture;

A denoised target picture is determined according to the target noise-added picture and the predicted noise corresponding to the target noise-added picture.

Optionally, the denoising module is further configured to:

A set of noised video frames is determined, and any video frame in the set of video frames is determined as a target noised picture.

The diffusion model processing device provided in the embodiment of this specification divides the time step set into time step intervals, and when the diffusion model is trained subsequently, the diffusion model can share the time step condition in a time step interval, that is, the first time step Both share the target time step in the corresponding time step interval, which reduces the time step conditions, greatly reduces the training burden, and improves the model training performance.

The above is a schematic solution of a diffusion model processing device in this embodiment. It should be noted that the technical solution of the diffusion model processing device and the above-mentioned technical solution of the diffusion model processing method belong to the same concept, and details of the technical solution of the diffusion model processing device that are not described in detail can be found in the above-mentioned diffusion model processing method. Description of the technical solution.

Corresponding to the foregoing method embodiments, this specification also provides an embodiment of a picture processing device. FIG. 6 shows a schematic structural diagram of a picture processing device provided by an embodiment of this specification. As shown in Figure 6, the device includes:

The second model prediction module 602 is configured to determine the target noise-added picture, input the target noise-added picture into the diffusion model, and obtain the predicted noise corresponding to the target noise-added picture;

The target picture determining module 604 is configured to determine a denoised target picture according to the target noise-added picture and the predicted noise corresponding to the target noise-added picture,

Wherein, the diffusion model is obtained through the above-mentioned diffusion model processing method.

The image processing device provided in the embodiment of this specification can quickly and accurately perform image denoising by reducing the time step condition training and high-performance diffusion model, and obtain the denoised target image, which greatly improves image denoising performance.

The foregoing is a schematic solution of an image processing apparatus in this embodiment. It should be noted that the technical solution of the image processing device and the above-mentioned technical solution of the image processing method belong to the same concept, and details of the technical solution of the image processing device that are not described in detail can be found in the description of the technical solution of the above-mentioned image processing method .

FIG. 7 shows a structural block diagram of a computing device 700 provided according to an embodiment of this specification. Components of the computing device 700 include, but are not limited to, memory 710 and processor 720 . The processor 720 is connected to the memory 710 through the bus 730, and the database 750 is used for saving data.

Computing device 700 also includes an access device 740 that enables computing device 700 to communicate via one or more networks 760 . Examples of these networks include a public switched telephone network (PSTN, Public Switched Telephone Network), a local area network (LAN, Local Area Network), a wide area network (WAN, Wide Area Network), a personal area network (PAN, Personal Area Network) or a communication network such as the Internet The combination. The access device 740 may include one or more of wired or wireless network interfaces of any type (for example, a network interface card (NIC, network interface controller)), such as an IEEE802.11 wireless local area network (WLAN, Wireless Local Area Network) wireless interface , Worldwide Interoperability for Microwave Access (Wi-MAX, Worldwide Interoperability for Microwave Access) interface, Ethernet interface, Universal Serial Bus (USB, Universal Serial Bus) interface, cellular network interface, Bluetooth interface, near field communication (NFC, Near FieldCommunication ) interface, and so on.

In an embodiment of the present specification, the above-mentioned components of the computing device 700 and other components not shown in FIG. 7 may also be connected to each other, for example, through a bus. It should be understood that the structural block diagram of the computing device shown in FIG. 7 is only for the purpose of illustration, rather than limiting the scope of this description. Those skilled in the art can add or replace other components as needed.

Computing device 700 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile telephones (e.g., smartphones), ), wearable computing devices (eg, smart watches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PC, Personal Computer). Computing device 700 may also be a mobile or stationary server.

Wherein, the processor 720 is configured to execute the following computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the above-mentioned diffusion model processing method or image processing method are implemented. The foregoing is a schematic solution of a computing device in this embodiment. It should be noted that the technical solution of the computing device belongs to the same idea as the above-mentioned technical solution of the diffusion model processing method or the image processing method. A description of the technical solution of the image processing method.

An embodiment of the present specification further provides a computer-readable storage medium, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the above-mentioned diffusion model processing method or image processing method are implemented.

The foregoing is a schematic solution of a computer-readable storage medium in this embodiment. It should be noted that the technical solution of the storage medium belongs to the same idea as the above-mentioned technical solution of the diffusion model processing method or the image processing method, and details not described in detail in the technical solution of the storage medium can be referred to above A description of the technical solution of the image processing method.

An embodiment of the present specification also provides a computer program, wherein when the computer program is executed in a computer, the computer is instructed to execute the steps of the above-mentioned diffusion model processing method or image processing method.

The foregoing is a schematic solution of a computer program in this embodiment. It should be noted that the technical solution of the computer program belongs to the same idea as the technical solution of the above-mentioned diffusion model processing method or image processing method, and details not described in detail in the technical solution of the computer program can be found in the above-mentioned diffusion model processing method or A description of the technical solution of the image processing method.

The foregoing describes specific embodiments of this specification. Other implementations are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

The computer instructions include computer program code, which may be in source code form, object code form, executable file or some intermediate form or the like. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM, Read-Only Memory) , Random Access Memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, computer-readable media Excludes electrical carrier signals and telecommunication signals.

It should be noted that, for the sake of simplicity of description, the aforementioned method embodiments are expressed as a series of action combinations, but those skilled in the art should know that the embodiments of this specification are not limited by the described action sequences. Because according to the embodiment of the present specification, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the specification.

In the above-mentioned embodiments, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, reference may be made to relevant descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are only for helping to explain the present specification. Alternative embodiments are not exhaustive in all detail, nor are the inventions limited to specific implementations described. Obviously, many modifications and changes can be made according to the contents of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can well understand and use this specification. This specification is to be limited only by the claims, along with their full scope and equivalents.

Claims (14)

1. A diffusion model processing method comprising:

determining a time step set of a diffusion model and a time step interval corresponding to the time step set;

determining a first time step from the time step set, and determining a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the time step set;

inputting the noise adding picture corresponding to the first time step and the target time step into a diffusion model to obtain the prediction noise corresponding to the noise adding picture;

and processing the diffusion model according to the target noise corresponding to the noise-added picture and the prediction noise.

2. The diffusion model processing method according to claim 1, wherein before inputting the noise-added picture corresponding to the first time step and the target time step into a diffusion model to obtain the prediction noise corresponding to the noise-added picture, the method further comprises:

determining an initial picture and target noise corresponding to the first time step;

and determining a noise adding picture corresponding to the first time step and target noise corresponding to the noise adding picture according to the initial picture and the target noise.

3. The diffusion model processing method according to claim 1, wherein the determining the time step set of the diffusion model and the time step interval corresponding to the time step set includes:

determining a time step set of a diffusion model, and dividing time step intervals in the time step set according to preset dividing conditions to obtain time step intervals corresponding to the time step set.

4. The diffusion model processing method according to claim 1, wherein the determining the target time step corresponding to the first time step according to the time step interval includes:

and determining an interval endpoint of the time step interval, and determining a target time step corresponding to the first time step according to the interval endpoint.

5. The diffusion model processing method according to claim 4, wherein determining the interval end point of the time step interval, and determining the target time step corresponding to the first time step according to the interval end point, comprises:

and determining a section left end point of the time step section, and determining the section left end point as a target time step corresponding to the first time step, wherein a section right end point of the time step section is a left end point included in a next time step section.

6. The diffusion model processing method according to claim 4, wherein determining the interval end point of the time step interval, and determining the time step corresponding to the first time step according to the interval end point, comprises:

and determining a right end point of the interval of the time step interval, and determining the right end point of the interval as a target time step corresponding to the first time step, wherein a left end point of the interval of the time step interval is a right end point included in the interval of the last time step.

7. The diffusion model processing method according to claim 1, wherein the processing the diffusion model according to the target noise corresponding to the noisy picture and the prediction noise includes:

and calculating a noise loss function according to the target noise corresponding to the noise-added picture and the predicted noise, adjusting network parameters of the diffusion model according to the noise loss function, and obtaining the diffusion model under the condition that a preset training ending condition is met.

8. The method for processing a diffusion model according to any one of claims 1 to 7, wherein after the processing the diffusion model according to the target noise corresponding to the noisy picture and the prediction noise, the method further comprises:

Determining a target noise adding picture, and inputting the target noise adding picture into a diffusion model to obtain prediction noise corresponding to the target noise adding picture;

and determining the denoised target picture according to the target denoised picture and the prediction noise corresponding to the target denoised picture.

9. The diffusion model processing method according to claim 8, the determining a target noisy picture, comprising:

and determining a noisy video frame set, and determining any video frame in the video frame set as a target noisy picture.

10. A diffusion model processing apparatus comprising:

the interval dividing module is configured to determine a time step set of the diffusion model and a time step interval corresponding to the time step set;

the target time step determining module is configured to determine a first time step from the time step set, and determine a target time step corresponding to the first time step according to the time step interval, wherein the first time step is any time step in the time step set;

the first model prediction module is configured to input a noise-added picture corresponding to the first time step and the target time step into a diffusion model to obtain prediction noise corresponding to the noise-added picture;

And the model processing module is configured to process the diffusion model according to the target noise corresponding to the noise-added picture and the prediction noise.

11. A picture processing method, comprising:

determining a target noise adding picture, and inputting the target noise adding picture into a diffusion model to obtain prediction noise corresponding to the target noise adding picture;

determining a denoised target picture according to the target denoised picture and the prediction noise corresponding to the target denoised picture, wherein the diffusion model is obtained by the diffusion model processing method according to any one of the claims 1-9.

12. A picture processing apparatus comprising:

the second model prediction module is configured to determine a target noise-added picture, input the target noise-added picture into a diffusion model and obtain prediction noise corresponding to the target noise-added picture;

a target picture determining module configured to determine a denoised target picture according to the target denoised picture and a prediction noise corresponding to the target denoised picture,

wherein the diffusion model is obtained by the diffusion model processing method according to any one of the preceding claims 1 to 9.

13. A computing device, comprising:

A memory and a processor;

the memory is configured to store computer executable instructions that, when executed by a processor, implement the steps of the diffusion model processing method of any one of claims 1 to 9 or the picture processing method of claim 11.

14. A computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the steps of the diffusion model processing method of any one of claims 1 to 9 or the picture processing method of claim 11.

Images