WO2022247568A1

WO2022247568A1 - Image restoration method and apparatus, and device

Info

Publication number: WO2022247568A1
Application number: PCT/CN2022/089429
Authority: WO
Inventors: 王伟; 袁泽寰; 王长虎
Original assignee: 北京有竹居网络技术有限公司
Priority date: 2021-05-28
Filing date: 2022-04-27
Publication date: 2022-12-01
Also published as: CN113222855B; CN113222855A

Abstract

Disclosed in the present application are an image restoration method and apparatus, and a device. The method comprises: determining a first degradation feature of a first image according to the first image to be restored and a target condition network; adjusting a parameter of a target super-resolution network according to the first degradation feature, and determining the adjusted target super-resolution network; and obtaining, according to the first image and the adjusted target super-resolution network, a second image after the first image is restored, the quality of the second image being higher than that of the first image. The target condition network is used for extracting a degradation feature of an image, and the target super-resolution network is used for restoring the quality of the image. In this way, the super-resolution network is adaptively adjusted by using a degradation feature describing the degradation situation of an image to be restored, and said image is restored by using the adjusted super-resolution network, such that low-quality images under degradation modes and degradation parameters can be restored, and an image restoration effect having relatively good generalization and practicability is realized, thereby providing a high-quality data source for computer vision tasks.

Description

Image restoration method, device and equipment

This application claims the priority of the Chinese patent application with the application number 202110594614.7 and the invention title "An Image Restoration Method, Device and Equipment" filed with the State Intellectual Property Office of China on May 28, 2021, the entire contents of which are incorporated by reference in this application.

technical field

The invention belongs to the technical field of image processing, and in particular relates to an image restoration method, device and equipment.

Background technique

In various computer vision tasks (such as video analysis, satellite monitoring, traffic supervision, criminal investigation, etc.), high-quality images (such as high-resolution images) have important application value and Research prospects. However, in actual situations, the process of image acquisition, storage, and transmission will inevitably be limited by external conditions or other interferences, resulting in varying degrees of quality degradation of high-quality images. Then, restoring the degraded low-quality image to a high-quality image is an important part of computer vision tasks.

At present, the methods used for image restoration can only realize image restoration for a specific degradation, but the degradation modes and degradation parameters that actually lead to low-quality images are various. Therefore, the current image restoration methods cannot universally realize the restoration effect of all low-quality images.

Based on this, it is urgent to provide an image restoration method capable of restoring low-quality images under various degradation modes and degradation parameters.

Contents of the invention

Embodiments of the present application provide an image restoration method, device, and equipment, which can restore various degraded low-quality images, and achieve image restoration effects with high generalization and practicability, so that it can be used for various computer vision tasks It is possible to provide high-quality data sources.

In the first aspect, the embodiment of the present application provides an image restoration method, including:

According to the first image to be restored and the target condition network, determine the first degradation feature of the first image, and the target condition network is used to extract the degradation feature of the image;

Adjust the parameters of the target super-resolution network according to the first degradation feature, determine the adjusted target super-resolution network, and the target super-resolution network is used to restore the quality of the image;

According to the first image and the adjusted target super-resolution network, a second image after restoration of the first image is obtained, and the quality of the second image is higher than that of the first image.

As an example, the target super-resolution network and the conditional network are obtained by alternately training the initial condition network and the initial super-resolution network using various samples in the sample database, wherein the sample database is based on high-quality samples The sample database includes multiple types of samples, and each type of sample includes images obtained by using the same degradation mode and degradation parameters to degrade the images in the sample image set.

Wherein, the degradation mode includes: at least one of resolution, noise, blur or compression.

As an example, the sample database includes the first type of samples and the second type of samples, and the alternate training of the initial condition network and the initial super-resolution network using the various types of samples in the sample database respectively includes:

Using the first type of samples, alternately train the initial condition network and the initial super-resolution network to obtain the intermediate condition network and the intermediate super-resolution network;

Based on the intermediate condition network and the intermediate super-resolution network, update the initial condition network and the initial super-resolution network, the updated initial condition network is the intermediate condition network, and the updated initial super-resolution network The sub-network is the intermediate super-divided network;

Using the second type of samples, alternately train the initial condition network and the initial super-resolution network to obtain the target condition network and the target super-resolution network.

As an example, using the first type of samples to alternately train the initial condition network and the initial super-resolution network to obtain the intermediate condition network and the intermediate super-resolution network, including:

determining a second degradation feature according to a plurality of third images in the first type of samples and the initial condition network;

adjusting the parameters of the initial super-resolution network according to the second degradation feature, and determining the adjusted initial super-resolution network;

Determine an output result according to the fourth image in the first type of samples and the adjusted initial super-resolution network;

Based on the output result, train the initial condition network to obtain the intermediate condition network;

Based on the intermediate condition network and the first type of samples, the initial super-resolution network is trained to obtain the intermediate super-resolution network.

Wherein, the target conditional network includes a convolutional layer and an average pooling layer, and the target super-resolution network includes a convolutional layer, a plurality of residual blocks and an upsampling function, and each residual block includes a convolutional layer.

As an example, the reconstruction loss function of the initial super-resolution network corresponding to the target super-resolution network is:

The comparative loss function in the initial condition network corresponding to the target condition network includes:

Among them, the Lres is the reconstruction loss function, I _LR is the input image of the initial super-resolution network Fsr, I _HR is the image before I _LR degradation, |||| ₁ is used to calculate the first-order norm, p(τ ) is a sampling function, E is used to calculate expectations, the Linner is an internal class loss function, the Lcross is a cross class loss function, Lcon is a contrastive loss function, _Xi , _Xi ' and X _j are the initial condition network The input image of Fc, Xi _i and Xi _' belong to the same class of samples, X _j and Xi _i belong to different classes of samples, p _x (τ) is the sampling function for the sample image set X, |||| ² is used to calculate 1 The square of the order norm.

In a second aspect, the embodiment of the present application further provides an image restoration apparatus, and the apparatus may include: a first determining unit, a second determining unit, and an obtaining unit. in:

A first determining unit, configured to determine a first degradation feature of the first image according to the first image to be restored and a target condition network, and the target condition network is used to extract the degradation feature of the image;

The second determination unit is configured to adjust the parameters of the target super-resolution network according to the first degradation feature, and determine the adjusted target super-resolution network, and the target super-resolution network is used to restore the quality of the image;

An obtaining unit, configured to obtain a second image restored from the first image according to the first image and the adjusted target super-resolution network, the quality of the second image is higher than that of the first image quality.

In a third aspect, the embodiment of the present application further provides an electronic device, where the electronic device includes: a processor and a memory;

said memory for storing instructions or computer programs;

The processor is configured to execute the instruction or the computer program in the memory, so that the electronic device executes the method provided in the first aspect above.

In a fourth aspect, the embodiment of the present application further provides a computer-readable storage medium, including instructions, which, when run on a computer, cause the computer to execute the method provided in the first aspect above.

It can be seen that the embodiment of the present application has the following beneficial effects:

The embodiment of the present application provides an image restoration method. The image restoration device that executes the method, when restoring the first image with poor quality, first determines the A first degenerate feature of the first image. The target conditional network is trained and used to extract the degenerated features of the image. Next, adjust the parameters of the target super-resolution network according to the first degradation feature, and determine the adjusted target super-resolution network. The target super-resolution network is trained and used to restore the quality of the image. Then, the device can obtain the second image restored from the first image according to the first image and the adjusted target super-resolution network. The quality of the second image is higher than that of the first image. It can be seen that through the method provided by the embodiment of the present application, the super-resolution network is adaptively adjusted by using the degradation characteristics describing the degradation of the image to be restored, and the image to be restored is restored by using the adjusted super-resolution network, which can correct various degradations. The low-quality images under the model and degradation parameters are restored, and the image restoration effect with better generalization and practicability is achieved, thus providing a high-quality data source for various computer vision tasks.

Description of drawings

The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the description, and are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention. In the attached picture:

FIG. 1 is a schematic flow chart of an image restoration method provided in an embodiment of the present application;

FIG. 2 is a schematic diagram of an example of image restoration performed by an image restoration method provided in an embodiment of the present application;

FIG. 3 is a schematic flow chart of a training process in an image restoration method provided in an embodiment of the present application;

FIG. 4 is a schematic structural diagram of an initial condition network and an initial super-resolution network in an embodiment of the present application;

FIG. 5 is a schematic flow diagram of a round of training for the initial condition network and the initial super-resolution network in the embodiment of the present application;

FIG. 6 is a schematic structural diagram of an image restoration device in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Detailed ways

In order to make the above objects, features and advantages of the present application more obvious and understandable, the embodiments of the present application will be further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It can be understood that the specific embodiments described here are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only parts relevant to the present application are shown in the drawings, not all structures.

Usually, high-quality images will degrade during the process of acquisition, storage, transmission, etc. The degradation modes include but are not limited to: resolution, blur, noise, and compression. However, many computer vision tasks (such as video analysis, traffic supervision) need to be completed based on the rich information in high-quality images. Therefore, restoring low-quality images to high-quality images is very important for most computer vision tasks.

Image super-resolution technology is used to restore the details of low-quality images and obtain high-quality images that reflect more abundant information. At present, methods for image restoration using image super-resolution technology include but are not limited to: Method 1, reconstructing low-quality images degraded by a fixed degradation mode (eg, resolution degradation mode of triple downsampling). Specifically, the neural network is used to learn the mapping relationship between the low-quality image and the high-quality image in the fixed degradation mode, so as to restore the low-quality image degraded by the fixed degradation mode by means of the neural network. However, this method 1 only supports the recovery of low-quality images degraded under a single degradation mode. Once a low-quality image is mixed with multiple degradation modes, the restoration performance will be greatly reduced, and high-quality images cannot be well restored. Method two, for low-quality images mixed with multiple degradation modes, a non-blind super-resolution algorithm is used for image restoration. The specific process includes: taking each low-quality image in the sample and the degradation of the low-quality image (such as blur kernel, noise coefficient, etc.) as the input of the model, using the output high-quality image and the known corresponding High-quality images to train the model. After the model training is completed, the degradation of the low-quality image to be restored is obtained by means of degradation estimation or manual adjustment, and the degradation and the low-quality image to be restored are input into the trained model, and the output is High-quality images recovered. Although the second method can restore low-quality images degraded by various degradation modes, the degradation of the low-quality images to be restored is often not accurate enough. The degradation situation is also inconsistent with the degradation situation of the sample image during the model training process, resulting in poor image restoration effect using the degradation situation and the trained model. Method three, for low-quality images mixed with multiple degradation modes, the blind super-resolution algorithm is used for image restoration. The specific process includes: first, preprocessing operations such as denoising, deblurring, and artifact removal are performed on the low-quality image to be restored. Next, use the neural network in Method 1 to reconstruct the preprocessed image. Although the third method can restore the low-quality images degraded by various degradation modes, because the degradation of the low-quality images to be restored is not completely consistent with the degradation of the sample images during the model training process, the third method cannot be applied to all low-quality images. Restoration of quality images. In summary, the current image restoration methods all have the problems of poor generalization and practicability.

Based on this, considering that the degradation mode and degradation parameters of the image to be restored are unknown and the degradation situation is complex, the embodiment of the present application provides an image restoration method, which can perform high-quality recovery on low-quality images under various degradation modes and degradation parameters. performance recovery. Specifically, the image restoration device implementing the method first determines the first degradation of the first image according to the first image to be restored and the target condition network when restoring the first image with poor quality feature. The target conditional network is trained and used to extract the degenerated features of the image. Next, adjust the parameters of the target super-resolution network according to the first degradation feature, and determine the adjusted target super-resolution network. The target super-resolution network is trained and used to restore the quality of the image. Then, the device can obtain the second image restored from the first image according to the first image and the adjusted target super-resolution network. The quality of the second image is higher than that of the first image.

In this way, through the method provided by the embodiment of the present application, the super-resolution network is adaptively adjusted by using the degradation characteristics describing the degradation of the picture to be restored, and the adjusted super-resolution network is used to restore the picture to be restored, and various degraded images can be recovered. The low-quality images under the model and degradation parameters are restored, and the image restoration effect with better generalization and practicability is achieved, which makes it possible to provide high-quality images as data sources for various computer vision tasks.

It should be noted that the subject implementing the embodiment of the present application may be a device with the image restoration function provided by the embodiment of the present application, and the device may be carried on a terminal, which may be existing, under development or future development, Any user device capable of interacting with each other through any form of wired and/or wireless connection, including but not limited to: smart wearable devices, smartphones, non-smartphones, tablets, laptops, existing, in development, or in the future Desktop PCs, desktop PCs, minicomputers, midrange computers, mainframes, etc. Wherein, the device implementing the embodiment of the present application may also include a target conditional network and a target super-resolution network.

In order to facilitate understanding of the specific implementation of the image restoration method provided by the embodiment of the present application, the following description will be made with reference to the accompanying drawings.

Referring to FIG. 1 , this figure is a schematic flow chart of an image restoration method provided by an embodiment of the present application. If it is necessary to restore the first image to be restored to obtain a high-quality second image, the method provided in the embodiment of the present application may be implemented. As shown in Figure 1, the method may include the following S101-S103:

S101. Determine a first degradation feature of the first image according to the first image to be restored and a target condition network, where the target condition network is used to extract the degradation feature of the image.

Wherein, the first image may be any low-quality image to be restored, and the first image may be an image obtained by degrading the high-quality image through at least one unknown degradation mode.

The target condition network is a model obtained by training the initial condition network and used to extract the degraded features of the image to be restored. The input of the target condition network is the image to be restored, and the output is the degraded feature of the image to be restored. A target-conditioned network can include, for example, convolutional layers and average pooling layers. Wherein, the structure of the initial condition network, the target condition network and the relevant description of the training to obtain the target condition network can refer to the introduction of the embodiments shown in FIG. 3 and FIG. 5 below.

The degradation feature of the image is used to describe the degradation condition of the image, and the degradation condition may include a degradation mode of the image and a degradation parameter corresponding to each degradation mode. Degenerate features can be represented as an array, for example: [128, 1, 1].

As an example, S101 may include, for example, inputting the first image into the target condition network, the target condition network outputs a degraded feature, and the degraded feature is recorded as the first degraded feature corresponding to the first image.

As another example, in order to reduce the calculation amount and time in the image restoration process, S101 may also include, for example: dividing the first image into blocks to obtain several image blocks. One or several image blocks among several image blocks are input into the target condition network. The target condition network outputs a degradation feature, and the degradation feature is used to describe the degradation of the image block input by the target condition network, and is also used to describe the degradation of the first image. Therefore, the degraded feature can be recorded as the first degraded feature corresponding to the first image.

After S101, the first degradation feature that can describe the degradation of the first image is obtained, which is ready for the subsequent adjustment of the target super-resolution network and the recovery of the first image by using the adjusted target super-resolution network.

S102. Adjust parameters of a target super-resolution network according to the first degradation feature, and determine an adjusted target super-resolution network, where the target super-resolution network is used to restore image quality.

The target super-resolution network is a model used to restore image quality obtained by training the initial super-resolution network. The input of the target super-resolution network is the image to be restored, and the output is the restored image. The target super-resolution network may include, for example, a convolutional layer, a plurality of residual blocks, and an upsampling function, each residual block including a convolutional layer. Wherein, the structure of the initial super-resolution network, the target super-resolution network, and the relevant description of the target super-resolution network after training can refer to the introduction of the embodiments shown in FIG. 3 and FIG. 5 below.

During specific implementation, the process of adjusting the parameters of the target super-resolution network by using the first degraded feature in S102 may include, for example: taking the first degraded feature as the conditional input of the target super-resolution network, performing linear layer transformation on the first degraded feature, and then Multiplying the transformed degraded features and the convolutional layer parameters in the target super-resolution network, using the calculated product to update the parameters of the corresponding convolutional layer, to obtain the adjusted target super-resolution network.

In this way, through S102, the adaptive adjustment of the target super-resolution network by using the degradation characteristics describing the degradation of the picture to be restored is realized, which provides a data basis for restoring the first picture based on the adjusted target super-resolution network in S103, so that the The method makes it possible to restore low-quality images under various degradation modes and degradation parameters.

S103. According to the first image and the adjusted target super-resolution network, obtain a second image after restoration of the first image, where the quality of the second image is higher than that of the first image.

In specific implementation, S103 may be, for example, inputting the first image into the target super-resolution network, and the image of the target super-resolution network is the second image in S103. The second image is a result obtained by restoring the first image through the method provided in the embodiment of the present application, that is, the second image is a high-quality image corresponding to the first image.

For example, using the image on the left side of FIG. 2 as the first image, the image on the right side of FIG. 2 (ie, the second image) can be obtained through the method provided in the embodiment of the present application. It can be seen from the comparison that the quality of the second image is higher than that of the first image.

It should be noted that the image quality mentioned in the embodiments of the present application is used to indicate the richness of information included in the image. For example, the quality of an image may be reflected by the resolution of the image. The higher the resolution of the image and the finer the details, the higher the quality of the image. Conversely, the lower the resolution of the image and the less details it reflects, the lower the quality of the image can be considered.

In some implementations, the target conditional network and the target super-resolution network can be used as two independent models in the image restoration device. Then, when executing the method, the image restoration device may first input the first image into the target condition network, and obtain the output of the target condition network—the first degraded feature. Next, the image restoration device inputs the first image and the first degraded feature into the target super-resolution network, and obtains the output of the target super-resolution network—the second image.

In other implementation manners, the target condition network and the target super-resolution network can be used as two units in an overall model in the image restoration device. Then, when executing the method, the image restoration device may input the first image into the overall model to obtain the output of the overall model—the second image. Wherein, the target condition network in the overall model first obtains the first degraded feature of the first image according to the first image. Next, the parameters of the target super-resolution network in the overall model are adjusted by using the first degenerate feature. Then, input the first image into the adjusted target super-resolution network in the overall model to obtain the second image.

It can be seen that, through the method provided by the embodiment of the present application, considering that the degradation mode and degradation parameters of the image to be restored are unknown and the degradation situation is complex, the conditional network is first used to obtain the degradation features of the low-quality image, and then the degradation characteristics describing the degradation of the image to be restored are used. The feature adaptively adjusts the super-resolution network, and then uses the adjusted super-resolution network to restore the picture to be restored. This enables high-performance restoration of low-quality images under various degradation modes and degradation parameters, and achieves image restoration effects with better generalization and practicability, thus making it possible to provide high-quality images for various computer vision tasks as Data sources are possible.

It can be understood that before the implementation of the embodiment shown in FIG. 1 , it is necessary to train the constructed initial condition network and initial super-resolution network to obtain the target condition network and target super-resolution network. Referring to Fig. 3, before performing the above S101-S103 using the target conditional network and the target super-resolution network, this embodiment of the present application may also include the following S301-S302:

S301. Construct a sample database according to the high-quality sample image set, degradation mode and degradation parameter, the sample database includes multiple types of samples, and each type of sample includes the samples in the sample image set using the same degradation mode and degradation parameters The image obtained after the image is degraded.

In order to make the trained target condition network and target super-resolution network applicable to the recovery of low-quality images under various degradation situations, in the embodiment of the present application, a sample database is first constructed based on S301. The sample database includes a wealth of samples to ensure the effectiveness and practicability of the trained target conditional network and target super-resolution network.

During specific implementation, the images in the high-quality sample image set are degraded according to different combinations of degradation modes and degradation parameters, and a set of low-quality sample images degraded by various degradation modes and degradation parameters are obtained. A set of low-quality sample images degraded by each degradation mode and degradation parameter is recorded as a class of samples, and multi-class samples are stored in the sample database to obtain the constructed sample database. The images in the sample database are the initial condition network and the training data of the initial super-resolution network.

Wherein, the degradation mode includes but not limited to: at least one of resolution, noise, blur or compression. Wherein, when the degradation mode includes resolution, the degradation parameters may correspond to different downsampling multiples, such as 2 times, 4 times, . . . . When the degradation mode includes noise, the degradation parameters can correspond to different Gaussian white noise coefficients, such as: 20, 30, . . . . When the degradation mode includes blur, the degradation parameters can correspond to different Gaussian blur kernels, such as: 0.5, 1.5, .... When the degradation mode includes compression, the degradation parameters can correspond to different compression algorithms.

It should be noted that when constructing the sample database, different degradation modes and combinations of degradation parameters can be preset. Under each combination, the images in the high-quality sample image set are degenerated to obtain a class of samples corresponding to the combination. This type of sample includes not only the low-quality image obtained after degrading by using the combination of the degradation mode and the degradation parameters, but also the combination of the degradation mode and the degradation parameters.

For example, assume that the high-quality sample image set Y includes 10 images: HR0, HR1, . . . , HR9. Combinations of degradation modes and degradation parameters include: combination 1 {Gaussian blur kernel G1, noise factor N1, downsampling multiple A1}, combination 2 {Gaussian blur kernel G2, downsampling multiple A2}, combination 3 {Gaussian blur kernel G1, noise Coefficient N1, downsampling multiple A3 and compression algorithm S}. Then, the sample database constructed through S301 may include: the first type sample X1, the second type sample X2 and the third type sample X3. Each type of sample includes 10 low-quality images, and each low-quality image is obtained by degrading an image in a sample image set Y through a combination corresponding to the type of sample. Wherein, the first type of sample X1 corresponds to combination 1, and the first type of sample X1 may include 10 images: LR10, LR11, . . . , LR19. The second type of sample X2 corresponds to combination 2, and the second type of sample X2 may include 10 images: LR20, LR21, . . . , LR29. The third type of sample X3 corresponds to combination 3, and the third type of sample X3 may include 10 images: LR30, LR31, . . . , LR39. Taking LR10 as an example, the LR10 may be an image obtained after HR0 is subjected to σG1 blur processing, σN1 noise processing and A1 downsampling. Among them, σ is the variance.

In this example, the sample database obtained through S301 includes: the first type of sample X1 {LR10, LR11, ..., LR19}-combination 1 {Gaussian blur kernel G1, noise factor N1, downsampling multiple A1}, the second Class sample X2{LR20, LR21,...,LR29}-combination 2{Gaussian blur kernel G2, downsampling multiple A2}, and third class sample X3{LR30, LR31,...,LR39}-combination 3{Gaussian blur Kernel G1, noise figure N1, downsampling multiple A3 and compression algorithm S}.

It should be noted that before S302, the initial condition network and the initial super-resolution network to be trained are also required. The initial condition network may include a convolutional layer and an average pooling layer, and the initial super-resolution network may include a convolutional layer, a plurality of residual blocks and an upsampling function, and each residual block includes a convolutional layer.

As an example, the initial condition network can adopt a structure of 4 convolutional layers and 2 average pooling layers. The initial super-resolution network can use 2 layers of convolutional layers, 10 residual blocks (such as SRResNet-10) and 1 upsampling function (English: Upsampling).

For example, as shown in FIG. 4 , the initial condition network 100 may include: a convolutional layer 1, a linear rectification function (English: ReLU) 1, a convolutional layer 2, a linear rectification function 2, an average pooling layer 1, and a convolutional layer 3 , linear rectification function 3, convolution layer 4, linear rectification function 4 and average pooling layer 2. Among them, the parameters of convolutional layer 1 and convolutional layer 2 can be K3n64s1. That is, the scale of the convolutional layer 1 and the convolutional layer 2 is: the convolution kernel is 3, the channel is 64, and the step size is 1. The parameters of average pooling layer 1 can be K2s2. That is, the size of the average pooling layer 1 is: the convolution kernel is 2, and the step size is 2. The parameters of convolutional layer 3 and convolutional layer 4 can be K3n128s1. That is, the scales of the convolutional layer 3 and the convolutional layer 4 are: 3 convolution kernels, 128 channels, and 1 step. The parameters of the average pooling layer 2 can be Kh/2sw/2. That is, the size of the average pooling layer 2 is: the convolution kernel is h/2, and the step size is w/2. Wherein, h and w are the height and width of the input image of the initial condition network 100 respectively. If the input of the initial condition network 100 is an image block with a height of h and a width of w in n images of a certain type of sample, and the channel is 3, then the output of the initial condition network 100 is the degraded features of these n images [128,1,1].

Still referring to FIG. 4 , the initial super-separation network 200 may include: fully connected layer 1, fully connected layer 2, ..., fully connected layer 20, residual block 1, residual block 2, ..., residual block 10, volume Product layer 5, convolutional layer 6 and upsampling function 1. Each residual block includes 2 convolutional layers and a linear rectification function, for example, residual block 1 includes: convolutional layer 7, linear rectification function 5, and convolutional layer 8. Among them, the inputs of the 20 fully connected layers are the degenerated features of the initial condition network 100 output, and the 20 outputs are respectively connected to the 20 convolutional layers of the 10 residual blocks, and the convolution in the fully connected layer and the residual block Layers correspond to each other. The input of the initial super-resolution network 200 includes: conditional input and super-resolution input. Among them, the conditional input is the degraded feature output by the initial condition network 100, and the super-resolution input is an image belonging to the same sample as the image input by the initial condition network 100. The super-resolution input passes through the convolutional layer 5, 10 residual blocks, the upper function 1 and the convolutional layer 6 in sequence to obtain the output of the initial super-resolution network, that is, the super-resolution input passes through the initial super-resolution network 200 for image restoration the resulting image.

In some implementations, the reconstruction loss function of the above-mentioned initial super-resolution network 200 can be expressed as the following formula (1):

Wherein, Lres is a reconstruction loss function, and the Lres can combine the output image I _SR of the initial super-resolution network 200 and the high-quality image I _HR corresponding to the input image I _LR to train the parameters of the initial super-resolution network 200 . In formula (1), I _LR is the input image of the initial super-resolution network Fsr 200, I _HR is the image before I _LR degradation, |||| ₁ is used to calculate the first-order norm, p(τ) is the sampling function, E is used to calculate expectations.

The comparative loss function in the initial condition network 100 may include the following formulas (2) to (4):

Among them, Linner is the inner class loss function, Lcross is the cross class loss function, and Lcon is the contrastive loss function. In the above formulas (2) and (3), Xi _, Xi _' and X _j are the input images of the initial condition network Fc100, Xi _and _Xi ' belong to the same class of samples, and X _j and _Xi belong to different classes of samples , p _x (τ) is the sampling function for the sample image set X, and |||| ² is used to calculate the square of the 1st order norm.

In this way, not only the sample database is constructed based on S301, but also the initial condition network and the initial super-resolution network to be trained are constructed, which is ready for S302 to obtain the target condition network and target super-resolution network.

S302. Using various samples in the sample database to alternately train an initial condition network and an initial super-resolution network to obtain the target condition network and the target super-resolution network.

For each type of sample in the multi-type samples in the sample database, it can be used to alternately train the initial condition network and the initial super-resolution network. The training process of each type of sample is similar to the initial condition network and the initial super-resolution network. Therefore, the model training process in the embodiment of the present application will be described below by taking two types of samples in the sample database as an example to train the initial condition network and the initial super-resolution network.

During specific implementation, it is assumed that the sample database includes samples of the first type and samples of the second type. Then, S302 may include, for example: S3021, using the first type of samples, alternately training the initial condition network and the initial super-resolution network to obtain the intermediate condition network and the intermediate super-resolution network. S3022. Based on the intermediate condition network and the intermediate super-resolution network, update the initial condition network and the initial super-resolution network, the updated initial condition network is the intermediate condition network, and the updated initial super-resolution network is the intermediate super-resolution network. S3023. Use the second type of samples to alternately train the initial condition network and the initial super-resolution network to obtain the target condition network and the target super-resolution network.

Among them, the alternate training process of each type of sample to the initial condition network and the initial super-resolution network is similar. Therefore, in order to illustrate the specific training process more clearly, you can refer to Figure 5, taking the use of the first type of samples to alternately train the initial condition network and the initial super-resolution network (S3021) as an example, and introduce the impact of each type of sample on the initial condition network and the initial super-resolution network. The training process of the sub-network, for example, may include:

S501. Determine a second degradation feature according to multiple third images in the first type of samples and the initial condition network.

S502. Adjust parameters of the initial super-resolution network according to the second degradation feature, and determine an adjusted initial super-resolution network.

S503. Determine an output result according to the fourth image in the first type of samples and the adjusted initial super-resolution network.

S504. Based on the output result, train the initial condition network to obtain the intermediate condition network.

S505. Based on the intermediate condition network and the first type of samples, train the initial super-resolution network to obtain the intermediate super-resolution network.

During specific implementation, S501 may, for example, be that the image restoration apparatus first selects several (for example, 5) third images from the first type of samples. The selected third image is input into the initial condition network, and the output of the initial condition network is the second degraded feature corresponding to the input multiple third images. Wherein, selecting the third image from the first type of samples may be selected randomly, or may be selected according to other possible preset rules, which is not limited in this embodiment of the present application. Optionally, in order to save computing resources, the selected third images can also be divided into blocks, and one or more image blocks of each third image can be input into the initial condition network to obtain the second degradation feature. It should be noted that whether the input of the initial condition network is the third image or the image block of the third image can be determined according to the structure when the initial condition network is constructed. accomplish.

Next, in S502, for example, the image restoration device may use the second degraded feature output by the initial condition network as the input condition of the initial super-resolution network, output it to the initial super-resolution network, and adjust the convolutional layer in each residual block in the initial super-resolution network. parameters to get the adjusted initial super-resolution network. Taking the initial condition network 100 and the initial super-resolution network 200 shown in Fig. 4 as an example, the second degradation features are respectively input into the fully connected layer 1 to the fully connected layer 20, and the fully connected layer 1 to the fully connected layer 20 degenerate the second After the features are transformed by the linear layer, the transformed results are respectively input to the convolutional layer in the residual block, multiplied by the parameters of the corresponding convolutional layer, and the product is used as the updated parameter of the convolutional layer. For example, the transformed result output by the fully connected layer 1 is input to the convolutional layer 7 in the residual block 1, and the updated parameters of the convolutional layer 7 are equal to the original parameters of the convolutional layer 7 and the output result of the fully connected layer 1. product. In this way, after updating the parameters of the convolutional layers in each residual block based on the second degradation feature, the obtained initial super-resolution network is the "adjusted initial super-resolution network" in S502.

Then, the image restoration device can select at least one fourth image from the first type of samples, and then use the fourth image as a super-resolution input and input it to the adjusted initial super-resolution network, and the output result of the initial super-resolution network is Fifth image. Then, between S503 and S504, the initial super-resolution network can also adjust the parameters in the initial super-resolution network based on the output result, the image corresponding to the fourth image in the high-quality sample image set and the reconstruction loss function, to obtain Updated initial super-resolution network.

Next, in S504, for example, the initial condition network may be trained based on the output result in S503 to obtain an intermediate condition network. As an example, S504 may include, for example: S1, input several third images in the first type of samples into the initial condition network to obtain the second degradation feature. S2. Input several sixth images in the first type of samples into the initial condition network to obtain the third degenerated features. S3. Input several seventh images in the second type of samples into the initial condition network to obtain the fourth degraded feature. S4. Determine the first result according to the second degenerate feature, the third degenerate feature and the internal class loss function, and determine the second result according to the second degenerate feature (or the third degenerate feature), the fourth degenerate feature and the cross-class loss function. Thus, the parameters of the initial condition network are adjusted according to the first result, the second result and the comparative loss function to obtain the intermediate condition network.

At this time, the intermediate conditional network can be regarded as a trained conditional network, and several images in the first type of samples are input into the intermediate conditional network, and the initial super-resolution network is adjusted by using the degraded features output by the intermediate conditional network. And input any image in the first sample into the adjusted initial super-resolution network, and use the output image of the initial super-resolution network and the input image of the initial super-resolution network to correspond to each other in the high-quality sample image set Image and reconstruction loss function, adjust the parameters in the initial super-resolution network to obtain the intermediate super-resolution network.

It can be seen that through the implementation of S501-S505 above, the intermediate condition network and the intermediate super-resolution network obtained by using the first type of samples to train the initial condition network and the initial super-resolution network are obtained. Then, the intermediate condition network can be recorded as the initial condition network for the next update, and the intermediate super-resolution network can be recorded as the initial super-resolution network for the next update. Then, enter the next round of training. For example, use the next class of samples that have not participated in the training to alternately train the initial condition network and the initial super-resolution network to obtain the intermediate condition network and the intermediate super-resolution network, and then return to execute "record the intermediate condition network as the next update The initial condition network, the intermediate super-resolution network is recorded as the initial super-resolution network for the next update" and "using the next class of samples that have not participated in the training, alternately train the initial condition network and the initial super-resolution network, Obtain the intermediate conditional network and the intermediate super-resolution network". Until all the class samples in the sample database participate in the training, the training of the initial super-resolution network and the initial condition network is ended, and the intermediate condition network and the intermediate super-resolution network obtained after the training of the last class of samples are recorded as the target condition network and Target hyperresolution network.

It should be noted that the above S3021 can be regarded as a round of training based on samples of the first type, and S3023 can be regarded as another round of training based on samples of the second type. The above S3021 to S3023 are specifically expressed by taking the example that the sample database only includes two types of samples (namely, the first type of samples and the second type of samples).

It should be noted that if the input of the initial condition network is an image, then the input of the target condition network can also be an image. If the input of the initial condition network is an image block in the image, then the input of the target condition network can also be an image block in the image.

It should be noted that the initial condition network and the initial super-resolution network can be used as two independent models in the image restoration device. Alternatively, the initial condition network and the initial super-resolution network can also be used as two units in an overall model in the image restoration device, which are not specifically limited in this embodiment of the present application.

It can be seen that through the method provided by the embodiment of the present application, the sample database, the initial condition network and the initial super-resolution network can be reasonably constructed, so that a reasonable network can be trained based on as many samples as possible, and the target applicable to various degradation situations can be obtained. Conditional network and target super-resolution network. The target condition network and the target super-resolution network have good generalization and practicability, which provide a basis for the recovery of complex images with unknown degradation conditions in the embodiment of the present application, and achieve better generalization and practicability. Image restoration effects, thus making it possible to provide high-quality images as data sources for various computer vision tasks.

Correspondingly, the embodiment of the present application also provides an image restoration apparatus 600, as shown in FIG. 6 . The apparatus 900 may include: a first determining unit 601 , a second determining unit 602 and an obtaining unit 603 . in:

The first determining unit 601 is configured to determine a first degradation feature of the first image according to the first image to be restored and a target condition network, and the target condition network is used to extract the degradation feature of the image;

The second determination unit 602 is used to adjust the parameters of the target super-resolution network according to the first degradation feature, and determine the adjusted target super-resolution network, and the target super-resolution network is used to restore the quality of the image;

An obtaining unit 603, configured to obtain a second image restored from the first image according to the first image and the adjusted target super-resolution network, and the quality of the second image is higher than that of the first image the quality of.

Based on the intermediate condition network and the first type of samples, train the initial super-resolution network to obtain the intermediate super-resolution network.

It should be noted that the device 600 corresponds to the method shown in the above-mentioned Fig. 1, Fig. 3 and Fig. 5, and the implementation of the device 600 and the effect achieved can refer to the implementation shown in the above-mentioned Fig. 1, Fig. 3 and Fig. 5 Description of the example.

In addition, the embodiment of the present application also provides an electronic device 700, as shown in FIG. 7 . The electronic device 700 includes: a processor 701 and a memory 702; wherein:

The memory 702 is used to store instructions or computer programs;

The processor 701 is configured to execute the instructions or computer programs in the memory 702, so that the electronic device executes the methods provided in the embodiments shown in FIG. 1 , FIG. 3 and FIG. 5 .

In addition, an embodiment of the present application also provides a computer-readable storage medium, including instructions, which, when run on a computer, cause the computer to execute the methods provided in the above-mentioned embodiments shown in FIG. 1 , FIG. 3 and FIG. 5 .

The "first" in the names such as "first image" and "first type sample" mentioned in the embodiment of the present application is only used for name identification, and does not mean the first in order. The same rule applies to "second" etc.

From the above description of the implementation manners, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of software products, and the computer software products can be stored in storage media, such as read-only memory (English: read-only memory, ROM)/RAM, disk, CDs, etc., include several instructions to make a computer device (which may be a personal computer, a server, or a network communication device such as a router) execute the methods described in various embodiments or some parts of the embodiments of this application.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment and the device embodiment, because they are basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to the part of the description of the method embodiment. The device and system embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

The above descriptions are only preferred implementations of the present application, and are not intended to limit the protection scope of the present application. It should be pointed out that those skilled in the art can make some improvements and modifications without departing from the present application, and these improvements and modifications should also be regarded as the protection scope of the present application.

Claims

An image restoration method, characterized in that, comprising:

According to the first image to be restored and the target condition network, determine the first degradation feature of the first image, and the target condition network is used to extract the degradation feature of the image;

Adjust the parameters of the target super-resolution network according to the first degradation feature, determine the adjusted target super-resolution network, and the target super-resolution network is used to restore the quality of the image;

According to the first image and the adjusted target super-resolution network, a second image after restoration of the first image is obtained, and the quality of the second image is higher than that of the first image.
The method according to claim 1, wherein the target super-resolution network and the conditional network are obtained by alternately training the initial condition network and the initial super-resolution network using various samples in the sample database, wherein the The sample database is constructed based on high-quality sample image collections, degradation modes and degradation parameters. The sample database includes multiple types of samples, and each type of sample includes the same degradation mode and degradation parameters for the sample image collection. The image obtained after the image is degraded.
The method according to claim 2, wherein the degradation mode comprises: at least one of resolution, noise, blur or compression.
The method according to claim 2, wherein the sample database includes samples of the first type and samples of the second type, and the initial condition network and the initial superstructure are alternately trained using various types of samples in the sample database. network, including:

Using the first type of samples, alternately train the initial condition network and the initial super-resolution network to obtain the intermediate condition network and the intermediate super-resolution network;

Based on the intermediate condition network and the intermediate super-resolution network, update the initial condition network and the initial super-resolution network, the updated initial condition network is the intermediate condition network, and the updated initial super-resolution network The sub-network is the intermediate super-divided network;

Using the second type of samples, alternately train the initial condition network and the initial super-resolution network to obtain the target condition network and the target super-resolution network.
The method according to claim 4, wherein said utilizing said first type of samples alternately trains an initial condition network and an initial super-resolution network to obtain an intermediate condition network and an intermediate super-resolution network, comprising:

Determining a second degradation feature based on a plurality of third images in the first type of samples and the initial condition network;

adjusting the parameters of the initial super-resolution network according to the second degradation feature, and determining the adjusted initial super-resolution network;

Determine an output result according to the fourth image in the first type of samples and the adjusted initial super-resolution network;

Based on the output result, train the initial condition network to obtain the intermediate condition network;

Based on the intermediate condition network and the first type of samples, the initial super-resolution network is trained to obtain the intermediate super-resolution network.
The method according to any one of claims 1 to 5, wherein the target conditional network includes a convolutional layer and an average pooling layer, and the target super-resolution network includes a convolutional layer, a plurality of residual blocks and Upsampling function, each residual block includes a convolutional layer.
The method according to any one of claims 1 to 5, wherein the reconstruction loss function of the initial super-resolution network corresponding to the target super-resolution network is:

The comparative loss function in the initial condition network corresponding to the target condition network includes:

Among them, the Lres is the reconstruction loss function, I LR is the input image of the initial super-resolution network Fsr, I HR is the image before I LR degradation, |||| 1 is used to calculate the first-order norm, p(τ ) is a sampling function, E is used to calculate expectations, the Linner is an internal class loss function, the Lcross is a cross class loss function, Lcon is a contrastive loss function, Xi , Xi ' and X j are the initial condition network The input image of Fc, Xi i and Xi ' belong to the same class of samples, X j and Xi i belong to different classes of samples, p x (τ) is the sampling function for the sample image set X, |||| 2 is used to calculate 1 The square of the order norm.
An image restoration device, characterized in that the device comprises:

A first determining unit, configured to determine a first degradation feature of the first image according to the first image to be restored and a target condition network, and the target condition network is used to extract the degradation feature of the image;

The second determination unit is configured to adjust the parameters of the target super-resolution network according to the first degradation feature, and determine the adjusted target super-resolution network, and the target super-resolution network is used to restore the quality of the image;

An obtaining unit, configured to obtain a second image restored from the first image according to the first image and the adjusted target super-resolution network, the quality of the second image is higher than that of the first image quality.
An electronic device, characterized in that the electronic device includes: a processor and a memory;

said memory for storing instructions or computer programs;

The processor is configured to execute the instructions or computer programs in the memory, so that the electronic device executes the method according to any one of claims 1 to 7.
A computer-readable storage medium is characterized by comprising instructions, which, when run on a computer, cause the computer to perform the method described in any one of claims 1 to 7 above.