CN116433501A

CN116433501A - Image processing method and device

Info

Publication number: CN116433501A
Application number: CN202310119971.7A
Authority: CN
Inventors: 杨涛; 任沛然; 谢宣松
Original assignee: Alibaba China Co Ltd
Current assignee: Alibaba China Co Ltd
Priority date: 2023-02-08
Filing date: 2023-02-08
Publication date: 2023-07-14
Anticipated expiration: 2043-02-08
Also published as: CN116433501B

Abstract

The embodiment of the specification provides an image processing method and device, wherein the image processing method comprises the following steps: determining an initial image, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image; inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical image to be processed; and inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed. The quality processing is carried out on the initial image, the quality processed image is input into a trained diffusion generation model, and the noise adding and the reverse denoising are carried out, so that the processed target image with reduced quality is obtained, and the target image is more in accordance with the real situation.

Description

Image processing method and device

Technical Field

The embodiment of the specification relates to the technical field of computers, in particular to an image processing method.

Background

And (5) processing, modifying and recovering the picture after the image is restored. The picture is usually subjected to color mixing, picture matting, synthesis, shading modification, chroma and chroma modification, special effect addition, editing, repair and the like through picture processing software. A similar concept to picture restoration is image restoration, a technique that analyzes, processes, and processes images to meet visual, psychological, and other requirements.

The effectiveness of deep learning based image restoration networks often depends heavily on the quality of the training data pairs, especially if the network is intended to be able to handle true degradation pictures, it is necessary to simulate as true degradation data as possible, where degradation refers to degradation. However, because the actual degradation is very complex and unknown, it is difficult to obtain ideal degradation data using the downsampling, classical degradation equations, etc. that are commonly used before, which severely limits the ability of the image restoration algorithm to process actual degraded pictures and videos. Thus, a better picture degradation scheme is needed.

Disclosure of Invention

In view of this, the present embodiment provides an image processing method. One or more embodiments of the present specification relate to an image processing apparatus, an image restoration method, a computing device, a computer-readable storage medium, and a computer program to solve the technical drawbacks of the related art.

According to a first aspect of embodiments of the present specification, there is provided an image processing method including:

determining an initial image, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image;

inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical image to be processed;

and inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed.

According to a second aspect of embodiments of the present specification, an image processing method, applied to a cloud-side device, includes:

acquiring an initial image from end side equipment, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image;

inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical target image;

And inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, and sending the target image to the end-side device, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed.

According to a third aspect of the embodiments of the present specification, there is provided an image processing apparatus comprising:

the image determining module is configured to determine an initial image, and perform quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image;

the image noise adding module is configured to input the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical image to be processed;

and the image denoising module is configured to input the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed.

According to a fourth aspect of embodiments of the present specification, there is provided an image processing apparatus applied to a cloud-side device, including:

The image determining module is configured to acquire an initial image from the terminal side equipment, and perform quality processing on the initial image to acquire an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image;

the image noise adding module is configured to input the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical target image;

and the image denoising module is configured to input the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, and send the target image to the end-side device, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed.

According to a fifth aspect of embodiments of the present specification, there is provided an image restoration method, comprising:

Inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed;

generating an image training sample set according to the target image and the initial image, and training to obtain an image restoration model according to the image training sample set;

inputting an image to be repaired into the image repair model to obtain a repair image, wherein the image quality of the image to be repaired is lower than that of the repair image.

According to a sixth aspect of embodiments of the present specification, there is provided a computing device comprising:

a memory and a processor;

the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which when executed by the processor, implement the steps of the image processing method described above.

According to a seventh aspect of the embodiments of the present specification, there is provided a computer-readable storage medium storing computer-executable instructions which, when executed by a processor, implement the steps of the above-described image processing method.

According to an eighth aspect of the embodiments of the present specification, there is provided a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the above-described image processing method.

The embodiment of the specification provides an image processing method and device, wherein the image processing method comprises the following steps: determining an initial image, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image; inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical image to be processed; and inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed. The quality processing is carried out on the initial image, the quality processed image is input into a trained diffusion generation model, noise adding and reverse denoising are carried out, the processed target image with reduced quality is obtained, and the diffusion image is denoised through the diffusion generation model, so that the target image is more in line with the real situation.

Drawings

Fig. 1 is a schematic view of a scenario of an image processing method according to an embodiment of the present disclosure;

FIG. 2a is a flow chart of an image processing method provided in one embodiment of the present disclosure;

FIG. 2b is a schematic diagram of an image processing method according to an embodiment of the present disclosure;

FIG. 3 is a flow chart of another image processing method provided by one embodiment of the present disclosure;

fig. 4 is a schematic structural view of an image processing apparatus according to an embodiment of the present specification;

fig. 5 is a schematic structural view of another image processing apparatus provided in one embodiment of the present specification;

FIG. 6 is a flow chart of an image restoration method according to one embodiment of the present disclosure;

FIG. 7 is a block diagram of a computing device provided in one embodiment of the present description.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present description. This description may be embodied in many other forms than described herein and similarly generalized by those skilled in the art to whom this disclosure pertains without departing from the spirit of the disclosure and, therefore, this disclosure is not limited by the specific implementations disclosed below.

The terminology used in the one or more embodiments of the specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of the specification. As used in this specification, one or more embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used in one or more embodiments of the present specification refers to and encompasses any or all possible combinations of one or more of the associated listed items.

It should be understood that, although the terms first, second, etc. may be used in one or more embodiments of this specification to describe various information, these information should not be limited by these terms. These terms are only used to distinguish one type of information from another. For example, a first may also be referred to as a second, and similarly, a second may also be referred to as a first, without departing from the scope of one or more embodiments of the present description. The word "if" as used herein may be interpreted as "at … …" or "at … …" or "responsive to a determination", depending on the context.

First, terms related to one or more embodiments of the present specification will be explained.

Image restoration: and inputting a degraded picture into the image restoration model, and restoring and enhancing the picture by using an algorithm so that the picture is clearer and sharper.

Diffusion model: a new model is generated, noise is continuously added by a Markov chain mode, and then a sample picture is recovered by a reverse denoising process.

Markov Chain (MC): is a stochastic process (stochastic process) in probability theory and physics statistics with Markov properties and existing within discrete index sets (index sets) and state space (state space).

The generation of reduced-prime data pairs is important for image restoration tasks, because deep learning-based network models tend to work well on the types of data that are seen in training. However, the degradation of data in the real world is very complex, that is, there may be a mixture of multiple degradation modes, and the degradation parameters are unknown, and although the previous modes try to expand the degradation space in multiple modes, the simulated degradation data and the real degradation data still have a large distribution gap.

At present, an image degradation method is a degradation method for randomly scrambling noise, blurring, downsampling, compressing and other degradation methods to increase a degradation space. The other image degradation mode is a method for generating degradation data by operating a classical degradation mode for multiple times, and mainly simulates the condition of multiple degradation of a real degradation picture in the transmission process. Both the two ways can greatly expand the degradation space, can simulate more complex degradation types, but cannot simulate data close to real degradation distribution, so that the simulated degradation data is still not real enough, and the effect of the model adopting the degradation methods in real scenes is severely limited.

It should be noted that, the user information (including, but not limited to, user equipment information, user personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) according to the embodiments of the present disclosure are information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data need to comply with the related laws and regulations and standards of the related country and region, and provide corresponding operation entries for the user to select authorization or rejection.

Based on this, in the present specification, an image processing method is provided, and the present specification relates to an image processing apparatus, a computing device, and a computer-readable storage medium, one by one, in the following embodiments.

Referring to fig. 1, fig. 1 shows a schematic view of a scene of an image processing method according to an embodiment of the present specification.

In the process of repairing an image, the image to be repaired is input into an image repairing model to obtain a repairing image. Before image restoration, the image restoration model needs to be trained so that the image restoration model can better perform image restoration. Before training of the image restoration model, in order to ensure the training effect, the training sample needs to be processed, so that the authenticity of the training sample is improved, and the restoration effect of the image restoration model is improved.

Specifically, the following steps may be performed based on the processing device 102, where the initial image 104 is determined, and the initial image 104 is subjected to quality processing to obtain a to-be-processed image 106 with an image quality smaller than that of the initial image 104, the to-be-processed image 106 is input into a noise-adding network layer of the diffusion generation model to obtain a diffusion image 108, the diffusion image 108 is input into a noise-removing network layer of the diffusion generation model to obtain a target image 110, where the image quality of the target image 110 is smaller than that of the initial image 104, and the image quality of the target image 110 is greater than that of the to-be-processed image 106. Through the steps, the obtained target image 110 is more in line with the real degraded image, the degraded image and the high-definition image can be used as training samples of the image restoration model, so that the image restoration model is trained, and the image restoration model with better restoration effect is obtained. And then repairing the image to be repaired by using the image repairing model to obtain a better repairing image.

Further, the training sample of the image restoration model may be a data pair composed of a high-definition image and a degraded image, where the image quality of the high-definition image is higher than that of the degraded image, for example, the image quality may be determined based on parameters such as resolution, image noise, and blur degree. The processing device may be a computer, a server, or the like capable of performing data processing. The initial image may be a high definition image.

According to the image processing method, the initial image is subjected to quality processing, the quality-processed image is input into the trained diffusion generation model, noise adding and reverse denoising are carried out, the processed target image with reduced quality is obtained, and the diffusion image is subjected to denoising through the diffusion generation model, so that the target image is more in line with the actual situation. And training the image restoration model through the target image so as to enable the restoration effect of the image restoration model to be better.

Referring to fig. 2a, fig. 2a shows a flowchart of an image processing method according to an embodiment of the present specification, which specifically includes the following steps.

Step 202: and determining an initial image, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image.

The initial image may be a high-definition image, the above-mentioned high-definition image may be determined by parameters of the image, and the parameters of the image may be parameters such as resolution, signal-to-noise ratio, etc., for example, the high-definition image is determined by the resolution of the image, and the initial image is an image with a resolution of 1280 pixels multiplied by 720 pixels or more. The quality processing may be processing that reduces the quality of an image, for example, downsampling the image. Correspondingly, the image to be processed is the image after quality processing.

In practical application, referring to fig. 2b, fig. 2b is a schematic diagram of an image processing method according to an embodiment of the present disclosure, for any given high-definition picture Y (belonging to the high-definition space Y), for example, the resolution of the high-definition picture Y is 1920 pixels by 1080 pixels, the resolution of the high-definition picture Y may be simply reduced by a conventional degradation method d to obtain a degraded picture X (belonging to the manual degradation space X), where the obtained (X, Y) degraded data pair is the previous method, and the space is a spatial concept in the index field: refers to those collections having special properties or additional structures. Mathematically, different kinds of spaces are sets of different kinds, each of which has its special properties or additional structure. For example, a three-dimensional Euclidean space, and other spaces are extensions, generalizations, developments, and abstractions of the three-dimensional Euclidean space in some special properties or additional structures.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to quality processing, and a to-be-processed image with an image quality smaller than that of the initial image is obtained, wherein the resolution of the to-be-processed image is 720×480, that is, 720 pixels by 480 pixels.

According to the embodiment of the specification, the image is subjected to primary degradation by using the preset degradation mode, so that the degradation space can be greatly expanded, more complex degradation types can be simulated, and a foundation is laid for the subsequent degradation method by using the diffusion model.

Specifically, the image may be degraded by some preset degradation processing methods, and specific implementation manners are as follows.

In one implementation, the performing quality processing on the initial image includes:

and carrying out quality processing on the initial image according to a preset quality processing rule.

The preset quality processing rule may be a preset image degradation processing rule, for example, a rule that processes an initial image by degradation methods such as noise, blurring, downsampling, compression, and the like.

In practical application, the initial image may be first processed by a preset degradation method. The degradation treatment can also be performed by a custom degradation mode. The self-defined degradation mode can be to limit the type of the initial image first, and the image is processed in different degradation modes according to the type of the image.

For example, the type of the initial image is first determined, and when the image is a text image, the initial image is processed in a first degradation manner, the first degradation manner may be a blurring process for the image, and when the image is another image, the initial image is processed in a second degradation manner, the second degradation manner may be a downsampling process for the image.

According to the embodiment of the specification, the initial image is processed through the preset quality processing rule, different degradation processes can be performed according to the first category of the initial image, and the individuation degree is improved.

Specifically, the performing quality processing on the initial image according to a preset quality processing rule includes:

and carrying out downsampling processing, blurring processing, noise adding processing and/or compression processing on the initial image according to a preset quality processing rule.

The preset quality processing rule may be a rule for processing the initial image by a degradation manner such as noise, blurring, downsampling, compression, or the like, or may be a quality processing rule for processing the image by a custom manner.

In practical applications, the initial image may be first processed by degradation such as noise, blurring, downsampling, compression, etc.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. And finally obtaining a to-be-processed image with image quality smaller than that of the initial image, wherein the resolution of the to-be-processed image is unchanged, namely 1280 pixels multiplied by 720 pixels.

For another example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is compressed to obtain a to-be-processed image with an image quality smaller than that of the initial image, and the resolution of the to-be-processed image is 720×480, that is, 720 pixels by 480 pixels.

It should be noted that one degradation processing method may be used for the initial image, or a combination of a plurality of degradation processing methods may be used in sequence.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels.

According to the embodiment of the specification, the initial image is processed in a plurality of or one degradation mode, so that the image after the classical degradation mode is obtained, and the degradation space can be greatly expanded.

Step 204: and inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to the historical image to be processed.

The noise adding network layer may be a network layer that performs noise adding on the image. Accordingly, the diffusion image may be a network layer that performs noise addition generation by the noise addition network layer.

In practical applications, referring to fig. 2b, after the de-prime data x is obtained in step 202, the de-prime data x still has a large distribution gap from the real de-prime space R. In order to get the de-prime data r as realistic as possible, we use a diffusion generation model. The specific mode is that the X is firstly diffused q, namely, the z is obtained by continuously adding noise, and the z has higher probability to be distributed in the Rt space according to the figure, wherein the Xt space and the Rt space are intermediate latent spaces in the diffusion process of the X space and the R space.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. And inputting the image to be processed into a noise adding network layer of the diffusion generation model to obtain a diffusion image.

According to the embodiment of the description, the image to be processed is input into the noise adding network layer of the diffusion generation model, the diffusion image is obtained, and the diffusion image is degraded in a noise adding mode, so that the degradation fidelity is improved.

Specifically, the noise adding network layer is used to add noise to the image, and the specific implementation is as follows.

In one implementation manner, the inputting the image to be processed into the noise adding network layer of the diffusion generation model to obtain a diffusion image includes:

inputting the image to be processed into a noise adding network layer of a diffusion generation model, and adding noise to the image to be processed at the noise adding network layer to obtain a diffusion image.

Where noise refers to image noise and unnecessary or redundant interference information present in the image data.

In practical applications, the noise-adding network layer in the diffusion generation model may first add noise to the image.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. Inputting the image to be processed into a noise adding network layer of the diffusion generation model, and adding noise to the image to be processed in the noise adding network layer to obtain a diffusion image.

According to the embodiment of the specification, the image to be processed is input into the noise adding network layer of the diffusion generation model, noise is added to the image to be processed in the noise adding network layer, the diffusion image is obtained, and the diffusion image is degraded in a noise adding mode, so that the fidelity of degradation is improved.

Specifically, adding noise to the image to be processed in the noise adding network layer may be based on the target table spreading time step, and the specific implementation manner is as follows.

In one implementation manner, the adding noise to the image to be processed at the noise adding network layer includes:

determining a target diffusion time step corresponding to the image to be processed and a noise processing rule corresponding to the image to be processed in the target diffusion time step at the noise adding network layer;

and adding noise to the image to be processed according to the noise processing rule corresponding to the target diffusion time step.

The target diffusion time step may be understood as the number of times of adding noise to the image to be processed in the noise adding network layer, for example, the target diffusion time step is 3, and then the noise is added to the image to be processed 3 times in the noise adding network layer.

In practical application, in the noise adding network layer, image processing is performed on the image to be processed according to the target diffusion time step, and noise is added to the image to be processed for multiple times.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. And inputting the image to be processed into a noise adding network layer of the diffusion generation model, wherein the target diffusion time step is 3, respectively determining a degradation mode corresponding to the target diffusion time step in the noise adding network layer, adding 3 times of noise according to the image to be processed corresponding to the target diffusion time step, and adding noise to the image to be processed in the noise adding network layer to obtain a diffusion image.

For another example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. Inputting the image to be processed into a noise adding network layer of a diffusion generation model, wherein the target diffusion time step is 3, and adding 3 times of noise to the image to be processed in the noise adding network layer to obtain a diffusion image.

According to the embodiment of the specification, the image to be processed is processed according to the target diffusion time step, and noise is added to the image to be processed for multiple times, so that the degradation effect is improved.

Specifically, before using the diffusion generation model, the diffusion generation model needs to be trained, and specific implementations are described below.

In one implementation manner, before the inputting the image to be processed into the noise adding network layer of the diffusion generation model, the method further includes:

acquiring a historical to-be-processed image and target noise corresponding to the historical to-be-processed image;

inputting the historical to-be-processed image and target noise corresponding to the historical to-be-processed image into a noise adding network layer of the diffusion generation model to obtain a sample diffusion image;

inputting the sample diffusion image into a noise removal network layer of the diffusion generation model to obtain prediction noise;

and training the diffusion generation model according to the prediction noise and the target noise.

The historical image to be processed can be an image sample to be trained.

In practical applications, before using the diffusion generating model, the diffusion generating model needs to be trained in order to ensure the effect of the diffusion generating model.

For example, a history to-be-processed image a is acquired, target noise to be added to the history to-be-processed image a is acquired, the history to-be-processed image a and the target noise are input into a noise adding network layer of a diffusion generation model, and a sample diffusion image is obtained. And inputting the sample diffusion image into a noise removal network layer of the diffusion generation model to obtain the prediction noise. And according to the comparison result of the prediction noise and the target noise, adjusting the model parameters of the diffusion generation model to complete the training of the diffusion generation model.

According to the embodiment of the specification, the diffusion generation model is trained, so that the effect of the degraded image generated by the diffusion generation model is improved.

Step 206: and inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed.

The noise-removing network layer may be a network layer that denoises the image in the diffusion generation model.

In practical application, after the diffusion image is input into the noise-removed network layer of the diffusion generation model, a target image with lower image quality than the initial image can be obtained.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. And inputting the image to be processed into a noise adding network layer of the diffusion generation model to obtain a diffusion image. And inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image.

According to the embodiment of the specification, the target image is obtained by inputting the diffusion image into the noise removal network layer of the diffusion generation model, so that a more real degradation image is generated.

In one implementation manner, the inputting the diffusion image into the noise-removing network layer of the diffusion generation model to obtain the target image includes:

and inputting the diffusion image into a noise removal network layer of the diffusion generation model, and removing noise from the diffusion image at the noise removal network layer to obtain a target image.

In practical application, referring to fig. 2b, the diffusion generation simulation has strong distribution fitting capability, so that the diffusion model p capable of generating real degradation data can be used for successfully denoising z reversely to obtain R, wherein the obtained R is located in the real degradation space R.

For example, an image with a resolution of 1280 pixels by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image of n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. And inputting the image to be processed into a noise adding network layer of the diffusion generation model, wherein the target diffusion time step is 3, respectively determining a degradation mode corresponding to the target diffusion time step in the noise adding network layer, adding 3 times of noise according to the image to be processed corresponding to the target diffusion time step, and adding noise to the image to be processed in the noise adding network layer to obtain a diffusion image. Inputting the diffusion image into a noise removing network layer of a diffusion generation model, and removing noise from the diffusion image at the noise adding network layer to obtain a target image.

In one implementation manner, after the inputting the diffusion image into the noise removal network layer of the diffusion generation model, obtaining a target image, the method further includes:

and generating an image training sample set according to the target image and the initial image, and training to obtain an image restoration model according to the image training sample set.

After the target image is obtained, the target image and the initial image can be used as a training set of an image restoration model, wherein the initial image can be used as an image obtained by restoring the target image by using the image restoration model.

In practical application, the target image obtained through the diffusion generation model is more in line with a real degradation image, and the degradation image and the high-definition image can be used as training samples of the image restoration model, so that the image restoration model is trained, and the image restoration model with better restoration effect is obtained.

For example, the degraded image is input into an image restoration model to obtain a restored image, and the image restoration model is trained based on the restored image and the high-definition image to obtain a trained image restoration model.

According to the embodiment of the specification, the training set of the image restoration model is generated through the obtained target image, so that the restoration image generated by the image restoration model is more real.

In one implementation, the generating an image training sample set from the target image and the initial image includes:

determining a first quality label of the target image, and determining a first training data set according to the first quality label and the target image;

determining a second quality label of the initial image, and determining a second training data set according to the second quality label and the initial image;

the image training sample set is determined from the first training data set and the second training data set.

The first quality label may be a high definition image label, and the second quality label may be a degraded image label. Accordingly, the first training data set may be a data set corresponding to a high-definition picture, and the second training data set may be a data set corresponding to a degraded picture.

In practical applications, the high-definition picture and the corresponding degraded picture can be used as a pair of training data for training the image restoration model.

For example, a high definition image is labeled with a high definition label, and a degraded image is labeled with a degraded label. And taking the high-definition image and the corresponding degraded image as a pair of training data.

The embodiments of the present description enable image restoration by tagging images to generate a training set of image restoration models.

In one implementation, after the obtaining the image restoration model, the method further includes:

The image to be repaired may be an image with lower quality, for example, in a manner that the degree of blurring exceeds a set blurring determination value,

in practical application, after the image restoration model is trained, the image restoration model can be used for image restoration.

For example, an image of 1280 pixels by 720 pixels is input into the image restoration model, so that a restoration image output by the image restoration model can be obtained, and the resolution of the restoration image can be 1920 pixels by 1080 pixels.

The image processing method provided in the present specification is further described below by taking an application of the image processing method to cloud-side devices as an example with reference to fig. 3. Fig. 3 is a flowchart of another image processing method according to an embodiment of the present disclosure, which specifically includes the following steps.

Step 302: acquiring an initial image from end side equipment, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image;

step 304: inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical target image;

step 306: and inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, and sending the target image to the end-side device, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed.

Specifically, the cloud-side device may be a processing device of the cloud, for example, a cloud server and other devices. The terminal device may be a terminal used by a user, such as a mobile phone terminal.

In one implementation manner, a user sends an image with a resolution of 1280 pixels and 720 pixels to a cloud device through a mobile phone, the cloud device obtains an image with a resolution of 1280 pixels and 720 pixels, takes the image as an initial image, performs blurring processing on the initial image, and gives gray values of all pixel points of the n rows and m columns of images, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. And inputting the image to be processed into a noise adding network layer of the diffusion generation model, wherein the target diffusion time step is 3, respectively determining a degradation mode corresponding to the target diffusion time step in the noise adding network layer, adding 3 times of noise according to the image to be processed corresponding to the target diffusion time step, and adding noise to the image to be processed in the noise adding network layer to obtain a diffusion image. Inputting the diffusion image into a noise removing network layer of a diffusion generation model, and removing noise from the diffusion image at the noise adding network layer to obtain a target image. The cloud device sends the target image to the mobile phone of the user, so that the user can utilize the target image.

It should be noted that, the cloud device may also be used to perform a training operation of the diffusion generation model or the image restoration model, which is not limited in the embodiment of the present specification.

According to the image processing method, the initial image is subjected to quality processing, the quality-processed image is input into the trained diffusion generation model, noise adding and reverse denoising are carried out, the processed target image with reduced quality is obtained, and the diffusion image is subjected to denoising through the diffusion generation model, so that the target image is more in line with the actual situation. And training the image restoration model through the target image so as to enable the restoration effect of the image restoration model to be better. And the processing capacity of cloud side equipment is utilized, so that the data processing efficiency is improved.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of an image processing apparatus, and fig. 4 shows a schematic structural diagram of an image processing apparatus according to one embodiment of the present disclosure. As shown in fig. 4, the apparatus includes:

an image determining module 402 configured to determine an initial image and perform quality processing on the initial image to obtain a to-be-processed image having an image quality smaller than the initial image;

The image noise adding module 404 is configured to input the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical image to be processed;

an image denoising module 406, configured to input the diffusion image into a noise removal network layer of the diffusion generation model, to obtain a target image, wherein the image quality of the target image is smaller than the initial image, and the image quality of the target image is larger than the image to be processed.

In one implementation, the image determination module 402 is configured to:

In one implementation, the image denoising module 404 is configured to:

In one implementation, the image denoising module 406 is configured to:

In one implementation, the image determination module 402 is configured to:

The embodiment of the specification provides an image processing method and device, wherein the image processing device comprises: determining an initial image, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image; inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical image to be processed; and inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed. The quality processing is carried out on the initial image, the quality processed image is input into a trained diffusion generation model, noise adding and reverse denoising are carried out, the processed target image with reduced quality is obtained, and the diffusion image is denoised through the diffusion generation model, so that the target image is more in line with the real situation.

The above is a schematic scheme of an image processing apparatus of the present embodiment. It should be noted that, the technical solution of the image processing apparatus and the technical solution of the image processing method belong to the same concept, and details of the technical solution of the image processing apparatus, which are not described in detail, can be referred to the description of the technical solution of the image processing method.

Corresponding to the above method embodiments, the present disclosure further provides an embodiment of an image processing apparatus, and fig. 5 shows a schematic structural diagram of an image processing apparatus according to one embodiment of the present disclosure. As shown in fig. 5, the apparatus is applied to cloud-side equipment, and includes:

an image determining module 502, configured to obtain an initial image from an end-side device, and perform quality processing on the initial image to obtain a to-be-processed image with image quality smaller than that of the initial image;

the image noise adding module 504 is configured to input the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical target image;

an image denoising module 506, configured to input the diffusion image into a noise removal network layer of the diffusion generation model, obtain a target image, and send the target image to the end-side device, wherein the image quality of the target image is smaller than the initial image, and the image quality of the target image is larger than the image to be processed.

According to the image processing device, the initial image is subjected to quality processing, the quality-processed image is input into the trained diffusion generation model, noise adding and reverse denoising are carried out, the processed target image with reduced quality is obtained, and the diffusion image is subjected to denoising through the diffusion generation model, so that the target image is more in line with the actual situation.

Further, referring to fig. 6, fig. 6 illustrates an image restoration method according to an embodiment of the present disclosure, based on the image processing method in the above embodiment, the embodiment of the present disclosure further provides an image restoration method, which specifically includes the following steps:

step 602: determining an initial image, and performing quality processing on the initial image to obtain an image to be processed, wherein the image quality of the image to be processed is smaller than that of the initial image;

Step 604: inputting the image to be processed into a noise adding network layer of a diffusion generation model to obtain a diffusion image, wherein the diffusion generation model is obtained by training according to a historical image to be processed;

step 606: inputting the diffusion image into a noise removal network layer of the diffusion generation model to obtain a target image, wherein the image quality of the target image is smaller than that of the initial image, and the image quality of the target image is larger than that of the image to be processed;

step 608: generating an image training sample set according to the target image and the initial image, and training to obtain an image restoration model according to the image training sample set;

step 610: inputting an image to be repaired into the image repair model to obtain a repair image, wherein the image quality of the image to be repaired is lower than that of the repair image.

In one implementation manner, an image with a resolution of 1280 pixels multiplied by 720 pixels is obtained, the image is taken as an initial image, the initial image is subjected to blurring processing, and the gray value of each pixel point of the image with n rows and m columns is given, specifically, the blurring processing can be performed by the following method: the gray value of the pixel point at the outermost side of the periphery of the initial image is unchanged, the new gray value of each pixel point in the middle is the average of the original gray values of the pixel point and four adjacent pixel points above, below, left and right, and if a decimal value exists, the nearest integer can be rounded. The image to be processed with the image quality smaller than that of the initial image after the blurring process can be obtained, and then the image to be processed is subjected to the downsampling process, so that the image to be processed with the image quality smaller than that of the initial image is obtained, and the resolution of the image to be processed is 720 multiplied by 480, namely 720 pixels multiplied by 480 pixels. And inputting the image to be processed into a noise adding network layer of the diffusion generation model to obtain a diffusion image. Inputting the diffusion image into a noise removing network layer of a diffusion generation model, and removing noise from the diffusion image at the noise adding network layer to obtain a target image.

Furthermore, the initial image and the target image can be used as training data of a pair of image restoration models, the degraded image is input into the image restoration models to obtain restored images, and the image restoration models are trained based on the restored images and the initial image to obtain trained image restoration models. The specific embodiments are described below.

For example, the degraded image is input into an image restoration model to obtain a restored image, a loss function value of the restored image and an initial image is calculated according to a loss function of the image restoration model, and parameter adjustment is performed on the image restoration model based on the loss function value, so that the image restoration model is trained.

Further, after training the image restoration model, the image restoration model can be used for restoring the picture.

Specifically, an image with 1280 pixels multiplied by 720 pixels is input into an image restoration model, so that a restoration image output by the image restoration model can be obtained, and the resolution of the restoration image can be 1920 pixels multiplied by 1080 pixels.

It should be noted that any image restoration model may be used as the image restoration model, and it is only necessary to ensure that the training data pair is an image data pair obtained by an image processing method.

According to the embodiment of the specification, the training set of the image restoration model is generated through the obtained target image, and the degraded image in the training set is more similar to the real degraded image, so that the trained image restoration model can have more accurate picture restoration capability, and the restoration image generated by the image restoration model is more real and has better effect.

Fig. 7 illustrates a block diagram of a computing device 700 provided in accordance with one embodiment of the present description. The components of computing device 700 include, but are not limited to, memory 710 and processor 720. Processor 720 is coupled to memory 710 via bus 730, and database 750 is used to store data.

Computing device 700 also includes access device 740, access device 740 enabling computing device 700 to communicate via one or more networks 760. Examples of such networks include public switched telephone networks (PSTN, public Switched Telephone Network), local area networks (LAN, local Area Network), wide area networks (WAN, wide Area Network), personal area networks (PAN, personal Area Network), or combinations of communication networks such as the internet. The access device 740 may include one or more of any type of network interface, wired or wireless, such as a network interface card (NIC, network interface controller), such as an IEEE802.11 wireless local area network (WLAN, wireless Local Area Network) wireless interface, a worldwide interoperability for microwave access (Wi-MAX, worldwide Interoperability for Microwave Access) interface, an ethernet interface, a universal serial bus (USB, universal Serial Bus) interface, a cellular network interface, a bluetooth interface, near field communication (NFC, near Field Communication).

In one embodiment of the present description, the above-described components of computing device 700, as well as other components not shown in FIG. 7, may also be connected to each other, such as by a bus. It should be understood that the block diagram of the computing device illustrated in FIG. 7 is for exemplary purposes only and is not intended to limit the scope of the present description. Those skilled in the art may add or replace other components as desired.

Computing device 700 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., tablet, personal digital assistant, laptop, notebook, netbook, etc.), mobile phone (e.g., smart phone), wearable computing device (e.g., smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or personal computer (PC, personal Computer). Computing device 600 may also be a mobile or stationary server.

Wherein the processor 620 is configured to execute computer-executable instructions that, when executed by the processor, perform the steps of the image processing method described above. The foregoing is a schematic illustration of a computing device of this embodiment. It should be noted that, the technical solution of the computing device and the technical solution of the image processing method belong to the same concept, and details of the technical solution of the computing device, which are not described in detail, can be referred to the description of the technical solution of the image processing method.

An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the image processing method described above.

The above is an exemplary version of a computer-readable storage medium of the present embodiment. It should be noted that, the technical solution of the storage medium and the technical solution of the image processing method belong to the same concept, and details of the technical solution of the storage medium which are not described in detail can be referred to the description of the technical solution of the image processing method.

An embodiment of the present specification also provides a computer program, wherein the computer program, when executed in a computer, causes the computer to perform the steps of the image processing method described above.

The above is an exemplary version of a computer program of the present embodiment. It should be noted that, the technical solution of the computer program and the technical solution of the image processing method belong to the same conception, and details of the technical solution of the computer program, which are not described in detail, can be referred to the description of the technical solution of the image processing method.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

The computer instructions include computer program code that may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

It should be noted that, for simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the embodiments are not limited by the order of actions described, as some steps may be performed in other order or simultaneously according to the embodiments of the present disclosure. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all required for the embodiments described in the specification.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.

The preferred embodiments of the present specification disclosed above are merely used to help clarify the present specification. Alternative embodiments are not intended to be exhaustive or to limit the invention to the precise form disclosed. Obviously, many modifications and variations are possible in light of the teaching of the embodiments. The embodiments were chosen and described in order to best explain the principles of the embodiments and the practical application, to thereby enable others skilled in the art to best understand and utilize the invention. This specification is to be limited only by the claims and the full scope and equivalents thereof.

Claims

1. An image processing method, comprising:

2. The method of claim 1, the quality processing of the initial image comprising:

3. The method of claim 2, the quality processing of the initial image according to a preset quality processing rule, comprising:

4. The method according to claim 1, wherein the inputting the image to be processed into the noise-adding network layer of the diffusion generation model, to obtain the diffusion image, includes:

5. The method of claim 4, the adding noise to the image to be processed at the noise-adding network layer, comprising:

6. The method of claim 1, the inputting the diffusion image into a noise-removing network layer of the diffusion generation model to obtain a target image, comprising:

7. The method of claim 1, further comprising, prior to said inputting the image to be processed into the noise-plus-network layer of the diffusion generation model:

8. The method of claim 1, further comprising, after said inputting the diffusion image into the noise-removing network layer of the diffusion generation model, obtaining a target image:

9. The method of claim 8, the generating an image training sample set from the target image and the initial image, comprising:

10. An image restoration method, comprising:

11. An image processing method applied to cloud side equipment comprises the following steps:

12. An image processing apparatus comprising:

13. A computing device, comprising:

a memory and a processor;

The memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions, which when executed by the processor, implement the steps of the image processing method of any one of claims 1 to 10.

14. A computer readable storage medium storing computer executable instructions which when executed by a processor implement the steps of the image processing method of any one of claims 1 to 10.