CN112419151A

CN112419151A - Image degradation processing method, device, storage medium and electronic equipment

Info

Publication number: CN112419151A
Application number: CN202011308390.0A
Authority: CN
Inventors: 王伟; 袁泽寰; 王长虎
Original assignee: Beijing Youzhuju Network Technology Co Ltd
Current assignee: Beijing Youzhuju Network Technology Co Ltd
Priority date: 2020-11-19
Filing date: 2020-11-19
Publication date: 2021-02-26
Anticipated expiration: 2040-11-19
Also published as: WO2022105638A1; CN112419151B

Abstract

The present disclosure relates to an image degradation processing method, apparatus, storage medium, and electronic device, which solve the problems of data distribution inconsistency and color shift between training data and actual data existing in acquiring low-quality images in the related art. The method comprises the following steps: acquiring a high-resolution and high-quality image; down-sampling the high-resolution and high-quality image to obtain a low-resolution and high-quality image; inputting the low-resolution high-quality image into an image degradation model to obtain a low-resolution low-quality image, wherein the image degradation model is obtained by training according to the brightness image characteristics of the sample low-resolution high-quality image and the brightness image characteristics of the sample low-resolution low-quality image, and the sample low-resolution high-quality image is obtained by down-sampling the sample high-resolution high-quality image; the low-resolution and low-quality images are used for training a super-resolution reconstruction network, and the super-resolution reconstruction network is used for repairing the input low-resolution and low-quality video to obtain a high-resolution and high-quality video.

Description

Image degradation processing method, device, storage medium and electronic equipment

Technical Field

The present disclosure relates to the field of video processing technologies, and in particular, to an image degradation processing method and apparatus, a storage medium, and an electronic device.

Background

With the continuous development of scientific technology, people can enjoy the experience of watching high-resolution and high-quality videos. However, the video is very susceptible to noise, blurring, compression and other factors in the acquisition stage and the transmission stage, which results in poor video quality, and this is especially a problem in some old movie sources.

The related art may perform video repair through a supervised restoration algorithm. In the training process of the supervised restoration algorithm, the low-quality image and the high-quality image need to be trained to learn the mapping between the low-quality image and the high-quality image, so as to realize the restoration of each frame of image in the video. However, in this process, the low-quality image is mainly obtained by artificially adding gaussian noise, gaussian blur, or other degradation operations to the high-quality image. If the noise, blur, etc. of the low-quality image used for training are not consistent with the real image to be restored, the problem that the real image cannot be effectively restored may be caused, thereby affecting the video restoration effect.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

In a first aspect, the present disclosure provides an image degradation processing method, including:

acquiring a high-resolution and high-quality image;

down-sampling the high-resolution high-quality image to obtain a low-resolution high-quality image;

inputting the low-resolution high-quality image into an image degradation model to obtain a low-resolution low-quality image, wherein the image degradation model is obtained by training according to the brightness image characteristics of the sample low-resolution high-quality image and the brightness image characteristics of the sample low-resolution low-quality image, and the sample low-resolution high-quality image is obtained by down-sampling the sample high-resolution high-quality image;

the low-resolution and low-quality images are used for training a super-resolution reconstruction network, and the super-resolution reconstruction network is used for repairing the input low-resolution and low-quality video to obtain a high-resolution and high-quality video.

In a second aspect, the present disclosure provides an image degradation processing apparatus, the apparatus comprising:

the acquisition module is used for acquiring a high-resolution and high-quality image;

the first processing module is used for performing down-sampling on the high-resolution and high-quality image to obtain a low-resolution and high-quality image;

the second processing module is used for inputting the low-resolution high-quality image into an image degradation model to obtain a low-resolution low-quality image, the image degradation model is obtained by training according to the brightness image characteristics of the sample low-resolution high-quality image and the brightness image characteristics of the sample low-resolution low-quality image, and the sample low-resolution high-quality image is obtained by down-sampling the sample high-resolution high-quality image;

In a third aspect, the present disclosure provides a computer readable medium having stored thereon a computer program which, when executed by a processing apparatus, performs the steps of the method of the first aspect.

In a fourth aspect, the present disclosure provides an electronic device comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to carry out the steps of the method of the first aspect.

By the technical scheme, the low-resolution low-quality images used for training the super-resolution reconstruction network can be obtained by processing a large number of high-resolution high-quality images through the image degradation model, and compared with a mode of obtaining the low-quality images by adding factors such as Gaussian noise, Gaussian blur and the like to the high-quality images in a manual control mode in the related art, various low-quality images which are more consistent with the actual situation can be obtained, the problem that video restoration cannot be effectively performed due to the fact that training data are inconsistent with actual data to be restored is solved, and the video restoration effect is improved. Moreover, in the process of training the image degradation model, the brightness characteristics of the image are extracted for model training, so that the problem of color shift of the subsequently obtained low-quality image can be avoided, the accuracy of the super-resolution reconstruction network obtained through the low-quality image training is improved, and the video restoration effect is further improved.

Additional features and advantages of the disclosure will be set forth in the detailed description which follows.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale. In the drawings:

FIG. 1 is a flow chart illustrating a method of image degradation processing according to an exemplary embodiment of the present disclosure;

fig. 2 is a schematic structural diagram illustrating an image degradation module and an image degradation removal module in an image degradation model in an image degradation processing method according to an exemplary embodiment of the present disclosure;

fig. 3 is a schematic structural diagram illustrating an image degradation module and an image degradation removal module in an image degradation model in an image degradation processing method according to another exemplary embodiment of the present disclosure;

FIG. 4 is a schematic diagram of the structure of each of the neural units in the image degradation module or the image degradation removal module shown in FIG. 3;

fig. 5 is a schematic structural diagram of a first discriminator and a second discriminator in an image degradation model in the case where the image degradation module or the image degradation removal module in the image degradation model is as shown in fig. 3;

fig. 6 is a schematic structural diagram illustrating an image degradation model in an image degradation processing method according to an exemplary embodiment of the present disclosure;

FIG. 7 is a schematic diagram illustrating the comparison of an output image of a method of image degradation processing with an actual low-resolution low-quality image according to an exemplary embodiment of the present disclosure;

fig. 8 is a block diagram illustrating an image degradation processing apparatus according to an exemplary embodiment of the present disclosure;

fig. 9 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description. It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units. It is further noted that references to "a", "an", and "the" modifications in the present disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.

As background, the related art employs a supervised restoration algorithm for video repair. In the training process of the supervised restoration algorithm, the low-quality image and the high-quality image need to be trained to learn the mapping between the low-quality image and the high-quality image, so as to realize the restoration of each frame of image in the video. However, in this process, the low-quality image is mainly obtained by artificially adding gaussian noise, gaussian blur, or other degradation operations to the high-quality image. If the noise, blur, etc. of the low-quality image used for training are not consistent with the real image to be restored, the problem that the real image cannot be effectively restored may be caused, thereby affecting the video restoration effect.

The inventor researches and discovers that the related art also carries out video restoration through an unsupervised image restoration algorithm. The method constructs image data pairs closer to the distribution of real image data through an unsupervised algorithm model so as to obtain a neural network model which can be generalized better on the real image data during testing. Although the method can align the data distribution between the training data and the test data to a certain extent, and reduce the problem of the supervised image restoration algorithm caused by the inconsistent data distribution between the training data and the test data, the method has the problem of color shift, which may lead to learning of color features irrelevant to image degradation features in the feature learning stage, thereby affecting the video restoration effect.

In view of the above, the present disclosure provides an image degradation processing method, an image degradation processing apparatus, a storage medium, and an electronic device, so as to solve a problem in the related art that effective video repair cannot be performed due to inconsistent data distribution between training data and test data or due to color shift, thereby improving a video repair effect.

Fig. 1 is a flowchart illustrating an image degradation processing method according to an exemplary embodiment of the present disclosure. Referring to fig. 1, the image degradation processing method includes the steps of:

step 101, obtaining a high-resolution and high-quality image.

Step 102, down-sampling the high resolution and high quality image to obtain a low resolution and high quality image.

And 103, inputting the low-resolution high-quality image into an image degradation model to obtain a low-resolution low-quality image, wherein the image degradation model is obtained by training according to the brightness image characteristics of the sample low-resolution high-quality image and the brightness image characteristics of the sample low-resolution low-quality image, and the sample low-resolution high-quality image is obtained by down-sampling the sample high-resolution high-quality image. The low-resolution low-quality images can be used for training a super-resolution reconstruction network, and the super-resolution reconstruction network is used for repairing input low-resolution low-quality videos to obtain high-resolution high-quality videos.

By the method, the low-resolution low-quality images used for training the super-resolution reconstruction network can be obtained by processing a large number of high-resolution high-quality images through the image degradation model, and compared with a method for obtaining the low-quality images by adding factors such as Gaussian noise, Gaussian blur and the like to the high-quality images in a manual control mode in the related art, various low-quality images which are more consistent with the actual situation can be obtained, the problem that video restoration cannot be effectively performed due to the fact that training data are inconsistent with actual data to be restored is solved, and the video restoration effect is improved. Moreover, in the process of training the image degradation model, the brightness characteristics of the image are extracted for model training, so that the problem of color shift of the subsequently obtained low-quality image can be avoided, the accuracy of the super-resolution reconstruction network obtained through the low-quality image training is improved, and the video restoration effect is further improved.

In order to make those skilled in the art understand the image degradation processing method provided by the embodiments of the present disclosure, the following describes the above steps in detail.

First, the training process of the image degradation model is explained.

Illustratively, a sample high resolution high quality image and a sample low resolution low quality image may be acquired first. It should be understood that resolution refers to the number of pixels contained per inch in an image. The higher the resolution, the more pixels an image unit inch contains, and the lower the resolution, the fewer pixels an image unit inch contains. The sample high resolution, high quality image unit inch includes more pixels than the sample low resolution, low quality image unit inch in the embodiments of the present disclosure. Image quality refers to the detail content stored in a pixel in an image, such as color, shading, contrast, etc. The higher the image quality, the richer the detail content contained in the pixel, and the lower the image quality, the less the detail content contained in the pixel. In the embodiment of the disclosure, the detail content of the pixel in the sample high-resolution high-quality image is richer than that of the pixel in the sample low-resolution low-quality image.

In a possible approach, the sample high resolution high quality image and the sample low resolution low quality image may be obtained by: a plurality of high resolution high quality videos and a plurality of low resolution low quality videos are captured. Then, for each high-resolution high-quality video, performing image segmentation on each frame image corresponding to the high-resolution high-quality video to obtain a plurality of first image blocks, and selecting a target first image block with a pixel value meeting a preset condition from the plurality of first image blocks as a sample high-resolution high-quality image. Similarly, for each low-resolution low-quality video, image segmentation may be performed on each frame image corresponding to the low-resolution low-quality video to obtain a plurality of second image blocks, and a target second pixel block having a pixel value meeting a preset condition is selected from the plurality of second image blocks as a sample low-resolution low-quality image.

For example, a plurality of high resolution high quality movie data and old movie data (low resolution low quality movie data) may be collected, and then image segmentation processing may be performed on each frame of image of the high resolution high quality movie data to obtain a plurality of first image blocks, for example, 3 × 3 image segmentation processing may be performed on each frame of image to obtain 9 first image blocks, and so on. Similarly, an image segmentation process may be performed on each frame of image of the old movie data, resulting in a plurality of second image blocks. For example, 3 × 3 image segmentation processing may be performed on each frame image to obtain 9 second image blocks, and so on.

After obtaining the plurality of first image blocks and the plurality of second image blocks, a target image block having a pixel value satisfying a preset condition may be selected for training the image degradation model. For example, the preset condition may be that a mean value of pixel values in the image block is greater than or equal to a preset threshold, or the preset condition may be that a variance of pixel values in the image block is greater than or equal to the preset threshold, or the preset condition may also be that a mean value or a variance corresponding to a low-frequency feature of a pixel point is greater than or equal to the preset threshold, and so on. The embodiment of the present disclosure is not limited to the specific content of the preset condition. In addition, the preset threshold in the above process may also be set according to an actual situation, which is not limited in the embodiment of the present disclosure.

The image blocks with rich details can be obtained by screening the first image block and the second image block through preset conditions, so that the content richness of the sample high-resolution high-quality image and the sample low-resolution low-quality image is increased, the robustness of the image degradation model is further increased, and the trained image degradation model outputs the low-resolution low-quality image which is more consistent with the actual situation. Among them, the detail enrichment is understood to mean that the difference in pixel values in the image block is large. For example, an image block with a small difference in pixel value, such as a sky image or a grass image, including a complex building, a person, and the like, is a detailed image block.

After the sample high-resolution high-quality image and the sample low-resolution low-quality image are obtained in the above manner, the image degradation model can be trained by the sample high-resolution high-quality image and the sample low-resolution low-quality image. For example, the high-resolution high-quality image of the sample may be down-sampled to obtain a low-resolution high-quality image of the sample, then the luminance image features of the low-resolution high-quality image of the sample and the luminance image features of the low-resolution low-quality image of the sample are extracted, and finally the image degradation model is trained according to the extracted luminance image features.

It should be appreciated that down-sampling is a technique for multi-rate digital signal processing or a process for reducing the signal sampling rate, typically used to reduce the data transfer rate or data size. In the embodiment of the disclosure, by down-sampling the sample high-resolution high-quality image, the number of pixel points included in a unit inch of the image can be reduced, so that the sample low-resolution high-quality image is obtained.

After obtaining the sample low-resolution high-quality image, the luminance image features of the sample low-resolution high-quality image can be extracted, and the luminance image features of the sample low-resolution low-quality image can be simultaneously extracted, so that the image degradation model is trained according to the extracted luminance image features.

It should be understood that the video repair method provided by the embodiment of the present disclosure can be applied to low-quality old movie repair. Considering that the low-quality old film is mostly black and white and has image characteristics such as noise and blur, the image degradation model needs to learn the image characteristics such as noise and blur of the low-quality old film without learning color characteristics in the image. In such a scenario, in the process of training the image degradation model, in order to avoid the color features in the sample high-resolution high-quality image from affecting the feature learning result of the image degradation model, i.e., to solve the color shift problem, the luminance image features of the sample low-resolution high-quality image and the luminance image features of the sample low-resolution low-quality image may be extracted for model training.

In a possible manner, the luminance image feature of the sample low-resolution high-quality image and the luminance image feature of the sample low-resolution low-quality image may be obtained by: and converting the sample low-resolution high-quality image from an RGB color space to a YCbCr color space to obtain a first target sample image, converting the sample low-resolution low-quality image from the RGB color space to the YCbCr color space to obtain a second target sample image, and extracting a brightness image characteristic corresponding to a Y channel of the first target sample image and a brightness image characteristic corresponding to a Y channel of the second target sample image.

Illustratively, the RGB color space is based on three basic colors of R (Red), G (Green ), and B (Blue), which are superimposed to different degrees, resulting in rich and wide colors. Y in the YCbCr color space is the luminance component of the color, and Cb and Cr are the density offsets of blue and red, respectively. In order to solve the problem of color shift in the embodiments of the present disclosure, an image may be converted from an RGB color space to a YCbCr color space, and then training of an image degradation model is performed in a Y channel, that is, a luminance image feature of the image is extracted to perform training of the image degradation model.

In a possible approach, the image degradation model may include an image degradation module, an image degradation removal module, a first discriminator, and a second discriminator. The image degradation module can be used for performing image degradation processing according to the brightness image characteristics of the input sample low-resolution high-quality image and outputting the brightness image characteristics of the simulated low-resolution low-quality image corresponding to the sample low-resolution high-quality image. The image degradation removing module can be used for carrying out image restoration processing according to the brightness image characteristics of the input sample low-resolution low-quality image and outputting the brightness image characteristics of the simulated low-resolution high-quality image corresponding to the sample low-resolution low-quality image. The first discriminator can be used for model training according to the input brightness image characteristics of the simulation low-resolution high-quality image and the brightness image characteristics of the sample low-resolution high-quality image. And the second discriminator is used for carrying out model training according to the brightness image characteristics of the input simulated low-resolution low-quality image and the brightness image characteristics of the sample low-resolution low-quality image.

Illustratively, the image degradation module and the image degradation removal module may employ a neural network structure including a plurality of residual modules (resblocks). For example, the neural network structure of the image degradation module and the image degradation removal module is shown in fig. 2. Wherein 3 × 3conv represents a convolution layer with a convolution kernel size of 3; the reduced Linear Unit (lu) represents an excitation function of the neural network structure, also called a modified Linear Unit,

representing a pixel-by-pixel addition operation.

Alternatively, in order to better utilize the context information of the image and enhance the neural network expression capability, the image degradation module and the image degradation removal module may adopt a neural network structure as shown in fig. 3. The neural network structure is described by taking a layer c4b as an example, and the layer c4b is obtained by laminating a layer c7 and a layer c4 a. Where the c7 layer is an deconvolution layer, its convolution step (stride) is 2, its input feature resolution is 4 x 4, and the up-sampled feature size is 8 x 8. The c4a layer is the same size as the c4 layer, and can be regarded as a "copy" of the c4 layer, which is twice the size of the c7 layer, and just the size of the c7 layer after upsampling, and the values can be directly added, so that the c4b layer is obtained. The data processing procedure of the other layers is similar to that of the layer c4b, and is not described in detail here. The structure of each neural network element in the neural network structure may be as shown in fig. 4, where concat represents a merge operation, and 1 × 1conv represents a convolution layer with a convolution kernel size of 1.

For example, in the case that the neural network structures of the image degradation module and the image degradation removal module are as shown in fig. 3, the first and second discriminators may adopt a block discriminator, and particularly, may include a full convolutional layer network of 7 convolutional layers. For example, the first and second discriminators have a structure as shown in fig. 5. Where 4 × 4conv denotes convolution layer with convolution kernel size 4, LeakyReLU denotes neural network activation function, also called leakage correction linear unit, similar to the common ReLU, SpectraLnorm (Spectral normalization) denotes Spectral normalization, Batchnorm (batch normalization) denotes batch normalization.

In one embodiment, a block diagram of the structure of the image degradation model may be as shown in fig. 6. Referring to fig. 6, after the luminance image feature Z of the sample low-resolution high-quality image is subjected to image degradation processing by the image degradation module G, the luminance image feature X' of the corresponding simulated low-resolution low-quality image can be obtained. After the brightness image feature X of the sample low-resolution low-quality image is subjected to image restoration processing by the image degradation removal module F, the brightness image feature Z' of the corresponding simulated low-resolution high-quality image can be obtained. First discriminator D_ZThe input of the method is a brightness image characteristic Z ' of the simulated low-resolution high-quality image and a brightness image characteristic Z of the sample low-resolution high-quality image, and the brightness image characteristic Z ' is used for judging that the brightness image characteristic Z ' is in accordance withProbability of luminance image feature Z. Second discriminator D_XThe input of (1) is a luminance image feature X ' of the simulated low-resolution low-quality image and a luminance image feature X of the sample low-resolution low-quality image, and the luminance image feature X ' is used for judging the probability that the luminance image feature X ' accords with the luminance image feature X.

In the process of training the image degradation model shown in fig. 6, the first discriminator D may be set in advance_ZIs a first probability value, and a second discriminator D_XIs the second probability value. Then, the first discriminator D can be made by adjusting the parameters of the image degradation module G and the image degradation removal module F_ZThe actual probability of the output is the first probability value, and the second discriminator D is used_XAnd outputting the actual probability as a second probability value, thereby realizing the training process of the image degradation model.

In a possible approach, the training process of the image degradation model may be: and extracting low-frequency signals aiming at the brightness image characteristics of the simulated low-resolution high-quality image, the brightness image characteristics of the sample low-resolution high-quality image, the brightness image characteristics of the simulated low-resolution low-quality image and the brightness image characteristics of the sample low-resolution low-quality image to obtain a target data set, and then training an image degradation model according to the image characteristics included in the target data set.

It should be understood that, in the embodiment of the present disclosure, in order to constrain the image degradation module and the image degradation removal module not to change the content of the input image, a low-frequency signal may be extracted from the luminance image feature of the image, and then model training may be performed according to the extracted low-frequency signal, so as to make the low-frequency content of the image output by the image degradation module and the image degradation removal module consistent.

In a possible way, the low frequency signal can be extracted by a gaussian low pass filter or by a haar wavelet transform. The Gaussian low-pass filter can make a compromise between excessive blurring (i.e. over smoothing) of image features and excessive abrupt variation (i.e. under smoothing) caused by noise and fine textures in an image, so that the extraction of low-frequency signals through the Gaussian filter for training of an image degradation model can achieve a good model training effect. In the image processing process, the haar wavelet transform can separate high-frequency and low-frequency information of an image, and in the embodiment of the disclosure, a low-frequency signal can be extracted from the brightness image characteristic of the image through the haar wavelet transform, so that model training is performed according to the extracted low-frequency signal, and the low-frequency content of the image output by the image degradation module and the image degradation removal module is consistent.

After extracting the low-frequency signal to obtain the target data set, the image degradation model may be trained according to image features included in the target data set. In a possible way, the loss function can be calculated first as follows:

L_cont＝λ₁||G(Z)_l-Z_l||₁+λ₂||F(X)_l-X_l||₁ (1)

wherein L is_contDenotes the loss function, λ₁And λ₂Representing a predetermined weight value, G (Z)_lRepresenting low-frequency signals, Z, extracted from luminance image features simulating low-resolution, low-quality images_lRepresenting low-frequency signals extracted from luminance image features of a sample low-resolution high-quality image, F (X)_lRepresenting low-frequency signals, X, extracted from luminance image features simulating a low-resolution, high-quality image_lRepresenting a low frequency signal extracted from luminance image features of a sample low resolution low quality image.

The parameters of the image degradation model may then be adjusted according to the above-mentioned loss function.

For example, referring to the image degradation model shown in FIG. 6, pass through a Gaussian low-pass filter ω_LExtracting the low frequency signal, then G (Z) in the loss function_lThen may be represented as ω_L*X1，Z_lCan be expressed as ω_L*Z，F(X)_lCan be expressed as ω_L*Z1，X_lCan be expressed as ω_LX. The loss function may be understood as a content consistency loss function of the image degradation model for characterizing the difference between the result processed by the image degradation module and the initial result input to the image degradation module, and through the imageAnd the difference between the result processed by the degradation removal module and the initial result input into the image degradation removal module is used for restraining the image degradation module and the image degradation removal module from changing the content of the input image.

In addition, referring to the image degradation model shown in fig. 6, the loss function of the image degradation model may further include a cyclic loss function and a domain alignment loss function, which are calculated in a manner similar to that of the related art, and will be briefly described below.

The cyclic loss function is used for representing the difference between the result obtained by inputting the image into the image degradation module and the result output by the image degradation module into the image degradation removal module and the initially output image, and the difference between the result obtained by inputting the image into the image degradation removal module and the result output by the image degradation removal module into the image degradation module and the initially output image. For example, referring to the image degradation model shown in fig. 6, the calculation formula of the cyclic loss function may be as follows:

L_cyc＝E_Z[||F(G(Z))-Z||₁]+E_X[||G(F(X))-X||₁] (2)

wherein L is_cycRepresenting the cyclic loss function, E_ZAnd E_XThe method is characterized in that mathematical expectation is shown, the calculation mode is similar to that of the related technology, Z shows a sample low-resolution high-quality image, G (Z) shows an output result obtained after the sample low-resolution high-quality image is input into an image degradation module, F (G (Z)) shows an output result obtained after the image degradation module is input into an image degradation removal module, X shows a sample low-resolution low-quality image, F (X) shows an output result obtained after the sample low-resolution low-quality image is input into the image degradation removal module, and G (F (X)) shows an output result obtained after the image degradation removal module is input into an image degradation module.

The domain alignment loss function is used for characterizing the difference between the simulated low-resolution low-quality image obtained by the image degradation module and the acquired actual low-resolution low-quality image, and the difference between the simulated low-resolution high-quality image obtained by the image degradation removal module and the acquired actual low-resolution high-quality image. It will be appreciated that the parameters of the first and second discriminators may be adjusted by a domain alignment loss function. For example, referring to the image degradation model shown in fig. 6, the calculation formula of the domain alignment loss function is as follows:

L(G,D_X)＝E_X[logD_X(X)]+E_Z[log(1-D_X(G(Z)))] (3)

L(F,D_Z)＝E_Z[logD_Z(Z)]+E_X[log(1-D_Z(F(X)))] (4)

wherein, L (G, D)_X) Represents D_XDomain alignment loss function of image domain, D_X(X) the low-resolution, low-quality image X representing the sample is input to a second discriminator D_XThe result obtained thereafter, D_X(G (Z)) represents the input of the analog low-resolution low-quality image into the second discriminator D_XThe result obtained is L (G, D)_Z) Represents D_ZDomain alignment loss function of image domain, D_Z(Z) A low resolution high quality image Z of the sample is input to a first discriminator D_ZThe result obtained thereafter, D_Z(F (X)) represents the simulation of the input of the low-resolution and high-quality image into the first discriminator D_ZThe result obtained is then.

Through the method, parameters of the image degradation model can be adjusted according to the cyclic loss function, the domain alignment loss function and the content consistency loss function, and the training process of the image degradation model is achieved. In the subsequent process, the low-resolution high-quality image can be input into the image degradation model to obtain the low-resolution low-quality image, and compared with a mode that the low-quality image is obtained by adding factors such as Gaussian noise, Gaussian blur and the like to the high-quality image in a manual control mode in the related technology, various low-quality images which are more consistent with the actual situation can be obtained, the problem that video restoration cannot be effectively performed due to the fact that training data is inconsistent with actual data to be restored is solved, and the video restoration effect is improved. Moreover, in the process of training the image degradation model, the brightness characteristics of the image are extracted for model training, so that the problem of color shift of subsequently obtained low-quality images can be avoided, the accuracy of the super-resolution reconstruction network obtained through the low-quality image training is improved, and the video restoration effect is improved

For example, referring to fig. 7, from left to right are: a true high definition high quality film image, a low definition old film image generated by an image degradation model, a true low definition old film image. As can be seen from fig. 7, the image degradation model in the video restoration method provided by the embodiment of the present disclosure can simulate the blur and compression of old movie data, so as to improve the subsequent video restoration effect.

For example, a low-resolution and low-quality image obtained by processing a high-resolution and high-quality image through the image degradation model can be used for training a super-resolution reconstruction network, so that the purpose of video restoration is achieved. For example, the high-resolution high-quality image P1 may be downsampled to obtain a low-resolution high-quality image P2, the low-resolution high-quality image P2 is input into a trained image degradation model to obtain a low-resolution low-quality image P3, and then the super-resolution reconstruction network is trained through paired training data { P3, P1}, so that each frame of image of the target video to be repaired is repaired according to the trained super-resolution reconstruction network to obtain a repaired video.

Alternatively, in order to increase the number of training samples, operations such as compression, blurring, noise and the like may be manually added to the high-resolution high-quality image P1 in the process of constructing the pair of training data to obtain a low-resolution low-quality image P4, and then the pair of training data { P4, P1} may be constructed. In this case, the super-resolution reconstruction network may be trained by the paired training data { P3, P1} and the paired training data { P4, P1 }. It should be understood that the specific structure of the super-resolution reconstruction network and the process of training by the pair of training data are similar to those of the related art, and will not be described herein.

Based on the same inventive concept, the embodiment of the disclosure also provides an image degradation processing device. Referring to fig. 8, the image degradation processing apparatus 800 may include:

an obtaining module 801, configured to obtain a high-resolution and high-quality image;

a first processing module 802, configured to down-sample the high-resolution and high-quality image to obtain a low-resolution and high-quality image;

a second processing module 803, configured to input the low-resolution high-quality image into an image degradation model to obtain a low-resolution low-quality image, where the image degradation model is trained according to luminance image features of a sample low-resolution high-quality image and luminance image features of a sample low-resolution low-quality image, and the sample low-resolution high-quality image is obtained by down-sampling the sample high-resolution high-quality image;

Optionally, the image degradation model includes an image degradation module, an image degradation removal module, a first discriminator and a second discriminator;

the image degradation module is used for carrying out image degradation processing according to the input brightness image characteristics of the sample low-resolution high-quality image and outputting the brightness image characteristics of the simulated low-resolution low-quality image corresponding to the sample low-resolution high-quality image;

the image degradation removal module is used for carrying out image restoration processing according to the input brightness image characteristics of the sample low-resolution low-quality image and outputting the brightness image characteristics of the simulated low-resolution high-quality image corresponding to the sample low-resolution low-quality image;

the first discriminator is used for carrying out model training according to the input brightness image characteristics of the simulated low-resolution high-quality image and the brightness image characteristics of the sample low-resolution high-quality image;

and the second discriminator is used for carrying out model training according to the input brightness image characteristics of the simulated low-resolution low-quality image and the brightness image characteristics of the sample low-resolution low-quality image.

Optionally, the apparatus 800 further comprises the following modules for training an image degradation model:

the extraction module is used for extracting a low-frequency signal aiming at the brightness image characteristic of the simulated low-resolution high-quality image, the brightness image characteristic of the sample low-resolution high-quality image, the brightness image characteristic of the simulated low-resolution low-quality image and the brightness image characteristic of the sample low-resolution low-quality image so as to obtain a target data set;

and the training module is used for training the image degradation model according to the image characteristics included in the target data set.

Optionally, the extraction module is configured to:

the low frequency signal is extracted by a gaussian low pass filter or by a haar wavelet transform.

Optionally, the training module is configured to:

the loss function is calculated as follows:

L_cont＝λ₁||G(Z)_l-Z_l||₁+λ₂||F(X)_l-X_l||₁ (1)

wherein L is_contDenotes the loss function, λ₁And λ₂Representing a predetermined weight value, G (Z)_lRepresenting low-frequency signals, Z, extracted from luminance image features simulating low-resolution, low-quality images_lRepresenting low-frequency signals extracted from luminance image features of a sample low-resolution high-quality image, F (X)_lRepresenting low-frequency signals, X, extracted from luminance image features simulating a low-resolution, high-quality image_lRepresenting a low frequency signal extracted from luminance image features of a sample low resolution low quality image;

and adjusting parameters of the image degradation model according to the loss function.

Optionally, the apparatus 800 further comprises the following modules for extracting luminance image features of the sample low-resolution high-quality image and luminance image features of the sample low-resolution low-quality image:

the first extraction module is used for converting the sample low-resolution high-quality image from an RGB color space to a YCbCr color space to obtain a first target sample image, and converting the sample low-resolution low-quality image from the RGB color space to the YCbCr color space to obtain a second target sample image;

and the second extraction module is used for extracting the brightness image characteristics corresponding to the Y channel of the first target sample image and the brightness image characteristics corresponding to the Y channel of the second target sample image.

Optionally, the apparatus 800 further comprises the following means for acquiring the sample high resolution high quality image and the sample low resolution low quality image:

the acquisition module is used for acquiring a plurality of high-resolution high-quality videos and a plurality of low-resolution low-quality videos;

the first selection module is used for carrying out image segmentation on each frame image corresponding to each high-resolution high-quality video aiming at each high-resolution high-quality video so as to obtain a plurality of first image blocks, and selecting a target first image block with a pixel value meeting a preset condition from the plurality of first image blocks as a sample high-resolution high-quality image;

and the second selection module is used for carrying out image segmentation on each frame of image corresponding to each low-resolution low-quality video aiming at each low-resolution low-quality video so as to obtain a plurality of second image blocks, and selecting a target second image block with a pixel value meeting the preset condition from the plurality of second image blocks as a sample low-resolution low-quality image.

With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Based on the same inventive concept, the disclosed embodiments also provide a computer readable medium, on which a computer program is stored, which when executed by a processing device, implements the steps of any of the image degradation processing methods described above.

Based on the same inventive concept, an embodiment of the present disclosure further provides an electronic device, including:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to implement the steps of any of the image degradation processing methods described above.

Referring now to FIG. 9, shown is a schematic diagram of an electronic device 900 suitable for use in implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a stationary terminal such as a digital TV, a desktop computer, and the like. The electronic device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 9, the electronic device 900 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 901 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage means 908 into a Random Access Memory (RAM) 903. In the RAM 903, various programs and data necessary for the operation of the electronic apparatus 900 are also stored. The processing apparatus 901, the ROM 902, and the RAM 903 are connected to each other through a bus 904. An input/output (I/O) interface 905 is also connected to bus 904.

Generally, the following devices may be connected to the I/O interface 905: input devices 906 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 907 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 908 including, for example, magnetic tape, hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic apparatus 900 to perform wireless or wired communication with other apparatuses to exchange data. While fig. 9 illustrates an electronic device 900 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer readable medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through the communication device 909, or installed from the storage device 908, or installed from the ROM 902. The computer program performs the above-described functions defined in the methods of the embodiments of the present disclosure when executed by the processing apparatus 901.

It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the communication may be performed using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.

The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a high-resolution and high-quality image; down-sampling the high-resolution high-quality image to obtain a low-resolution high-quality image; inputting the low-resolution high-quality image into an image degradation model to obtain a low-resolution low-quality image, wherein the image degradation model is obtained by training according to the brightness image characteristics of the sample low-resolution high-quality image and the brightness image characteristics of the sample low-resolution low-quality image, and the sample low-resolution high-quality image is obtained by down-sampling the sample high-resolution high-quality image; the low-resolution and low-quality images are used for training a super-resolution reconstruction network, and the super-resolution reconstruction network is used for repairing the input low-resolution and low-quality video to obtain a high-resolution and high-quality video.

Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The modules described in the embodiments of the present disclosure may be implemented by software or hardware. Wherein the name of a module in some cases does not constitute a limitation on the module itself.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

According to one or more embodiments of the present disclosure, an example provides an image degradation processing method including:

acquiring a high-resolution and high-quality image;

Example two provides the method of example one, the image degradation model comprising an image degradation module, an image degradation removal module, a first discriminator, and a second discriminator, in accordance with one or more embodiments of the present disclosure;

Example three provides the method of example two, the training process of the image degradation model including:

extracting a low-frequency signal aiming at the brightness image characteristic of the simulated low-resolution high-quality image, the brightness image characteristic of the sample low-resolution high-quality image, the brightness image characteristic of the simulated low-resolution low-quality image and the brightness image characteristic of the sample low-resolution low-quality image to obtain a target data set;

the image degradation model is trained on the image features included in the target data set.

Example four provides the method of example three, the extracting the low frequency signal comprising:

Example five provides the method of example three, the training the image degradation model according to image features included in the target dataset including:

the loss function is calculated as follows:

L_cont＝λ₁||G(Z)_l-Z_l||₁+λ₂||F(X)_l-X_l||₁

Example six provides the method of any one of examples one to five, the luminance image features of the sample low-resolution high-quality image and the luminance image features of the sample low-resolution low-quality image being obtained by:

converting the sample low-resolution high-quality image from an RGB color space to a YCbCr color space to obtain a first target sample image, and converting the sample low-resolution low-quality image from the RGB color space to the YCbCr color space to obtain a second target sample image;

and extracting the brightness image characteristic corresponding to the Y channel of the first target sample image and the brightness image characteristic corresponding to the Y channel of the second target sample image.

Example six provides the method of any one of examples one to five, the sample high resolution high quality image and the sample low resolution low quality image being obtained by:

collecting a plurality of high-resolution high-quality videos and a plurality of low-resolution low-quality videos;

aiming at each high-resolution high-quality video, carrying out image segmentation on each frame of image corresponding to the high-resolution high-quality video to obtain a plurality of first image blocks, and selecting a target first image block with a pixel value meeting a preset condition from the plurality of first image blocks as a sample high-resolution high-quality image;

and aiming at each low-resolution low-quality video, performing image segmentation on each frame image corresponding to the low-resolution low-quality video to obtain a plurality of second image blocks, and selecting a target second image block with a pixel value meeting the preset condition from the plurality of second image blocks as a sample low-resolution low-quality image.

Example eight provides, according to one or more embodiments of the present disclosure, an image degradation processing apparatus including:

Example nine provides the apparatus of example eight, the image degradation model comprising an image degradation module, an image degradation removal module, a first discriminator, and a second discriminator, in accordance with one or more embodiments of the present disclosure;

Example ten provides the apparatus of example nine, further comprising the following modules for training the image degradation model, in accordance with one or more embodiments of the present disclosure:

In accordance with one or more embodiments of the present disclosure, example eleven provides the apparatus of example ten, the extraction module to:

Example twelve provides the apparatus of example ten, the training module to:

the loss function is calculated as follows:

L_cont＝λ₁||G(Z)_l-Z_l||₁+λ₂||F(X)_l-X_l||₁

Example thirteen provides the apparatus of any one of examples eight to twelve, further comprising means for extracting luminance image features of the sample low-resolution high-quality image and luminance image features of the sample low-resolution low-quality image, as follows:

Example fourteen provides the apparatus of any one of examples eight to twelve, further comprising means for acquiring the sample high resolution high quality image and the sample low resolution low quality image as follows:

Example fifteen provides a computer-readable medium having stored thereon a computer program that, when executed by a processing device, implements the steps of the method of any one of examples one to seven in accordance with one or more embodiments of the present disclosure.

Example sixteen provides an electronic device, in accordance with one or more embodiments of the present disclosure, comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to carry out the steps of the method of any one of examples one to seven.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims. With regard to the apparatus in the above-described embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be elaborated here.

Claims

1. An image degradation processing method, characterized in that the method comprises:

acquiring a high-resolution and high-quality image;

2. The method of claim 1, wherein the image degradation model comprises an image degradation module, an image degradation removal module, a first discriminator, and a second discriminator;

3. The method of claim 2, wherein the training process of the image degradation model comprises:

4. The method of claim 3, wherein the extracting the low frequency signal comprises:

5. The method of claim 3, wherein the training of the image degradation model according to the image features included in the target data set comprises:

the loss function is calculated as follows:

L_cont＝λ₁||G(Z)_l-Z_l||₁+λ₂||F(X)_l-X_l||₁

6. The method according to any one of claims 1 to 5, wherein the luminance image characteristic of the sample low-resolution high-quality image and the luminance image characteristic of the sample low-resolution low-quality image are obtained by:

7. The method according to any of claims 1-5, wherein the sample high resolution high quality image and the sample low resolution low quality image are obtained by:

8. An image degradation processing apparatus characterized by comprising:

9. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the steps of the method of any one of claims 1 to 7.

10. An electronic device, comprising:

a storage device having a computer program stored thereon;

processing means for executing the computer program in the storage means to carry out the steps of the method according to any one of claims 1 to 7.