CN111260558A - Image super-resolution network model with variable magnification - Google Patents
Image super-resolution network model with variable magnification Download PDFInfo
- Publication number
- CN111260558A CN111260558A CN202010075845.2A CN202010075845A CN111260558A CN 111260558 A CN111260558 A CN 111260558A CN 202010075845 A CN202010075845 A CN 202010075845A CN 111260558 A CN111260558 A CN 111260558A
- Authority
- CN
- China
- Prior art keywords
- image
- resolution
- residual
- learning
- magnification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4046—Scaling the whole image or part thereof using neural networks
Abstract
The invention discloses a variable-magnification image super-resolution network model, which comprises a parameterized residual learning network PRNet, a residual refined learning network RRNet and a superposition network; a parameterized residual learning network PRNet for learning a mapping between a low resolution LR image to a high resolution HR image; the residual error refinement learning network RRNet is used for learning and reconstructing mapping from the high-resolution image to the residual error image; and the superposition network is used for superposing the high-resolution HR image and the residual image to form a final super-resolution SR image and outputting the final super-resolution SR image. According to the method, the magnification factor parameter is explicitly expressed, and a parameterized residual learning network model is established, so that the model can accept input of any scale, and the requirement of a variable-magnification super-resolution task is met; the invention provides a residual error refinement learning network, and the mapping relation between the reconstructed high-resolution image and the reconstructed residual error is further learned, so that residual error compensation is carried out on the reconstructed image, and the super-resolution reconstruction quality is improved.
Description
Technical Field
The invention belongs to the technical field of digital image processing, and relates to an image super-resolution network model, in particular to a variable-magnification image super-resolution network model.
Technical Field
The deep learning technology promotes the huge leap of the super-resolution (SR) reconstruction performance of the image, but the deep learning model facing the super-resolution reconstruction can only aim at one fixed magnification at a time, and if the super-resolution tasks of different magnifications are to be executed at the same time, a plurality of deep learning network models corresponding to different magnifications need to be trained. For applications that require arbitrary uncertain magnification to be performed, this limitation of the super-resolution network limits its practical application range.
The super-resolution application scene with uncertain magnification is ubiquitous in reality, and a common scene is face super-resolution recognition of a surveillance video, namely the resolution of a face is improved through a super-resolution technology so as to improve the precision of a face recognition system. Due to the fact that the distance between the monitoring target and the camera is different, the resolution ratios of the collected human faces are different. The method is a typical variable-magnification super-resolution processing task for improving the low-resolution face images with different input sizes to the uniform high-resolution size required by a face recognizer.
The method for training the respective models for different magnifications is complicated and impractical, and one alternative scheme is to firstly use the traditional image interpolation method to sample the input images with indefinite sizes to the uniform final size and then perform super-resolution reconstruction with constant magnification. In this way, the deep learning super-resolution model no longer expands the spatial resolution, but supplements the detailed information of the image. Because original detail abundance degrees of input images with uniform sizes are different, and detail increment needing super-resolution model supplement is not fixed, the super-resolution network is required to have strong learning capacity of detail with unequal increment, and challenges are provided for diversity of training samples and feature pattern expression capacity of the network.
As can be seen, for the super-resolution task with variable magnification, the two current processing modes have serious defects, and the mode of training the corresponding network model for each magnification is complicated and is impractical; the mode of interpolation to uniform resolution and then over-resolution reconstruction ignores the difference of input images with different resolutions in content components, and is difficult to obtain ideal reconstruction effect. The reason why the network trained for one magnification (such as X2) is not suitable for image magnification of another magnification (such as X3) is that the existing super-resolution reconstruction network does not explicitly express the variable of magnification, so that a super-resolution network model with variable magnification is necessary to meet the requirement of any scale input.
Disclosure of Invention
In order to solve the technical problem, the invention provides a variable-magnification image super-resolution network model.
The technical scheme adopted by the invention is as follows: a variable-magnification image super-resolution network model is characterized in that: the model comprises a parameterized residual error learning network PRNet, a residual error refined learning network RRNet and a superposition network;
the parameterized residual learning network PRNet is used for learning the mapping from the low-resolution LR image to the high-resolution HR image;
the residual error refinement learning network RRNet is used for learning and reconstructing mapping from a high-resolution image to a residual error image;
and the superposition network is used for superposing the high-resolution HR image and the residual image to form a final super-resolution SR image output.
Preferably, the parameterized residual learning network PRNet has a basic computing unit that is a parameterized residual learning unit;
the parameterized residual learning unit regulates and controls residual obtained by residual image learning network learning, adjusts the contribution proportion of the residual to super-resolution reconstruction, greatly contributes at high magnification, and slightly contributes at low magnification, and directly outputs the original image without the residual participating in reconstruction when the magnification is 1, so that the information quantity of the residual participating in super-resolution reconstruction is positively related to the magnification;
the parameterized residual learning unit is formally expressed as:
y＝f·r(x)+x；
wherein x is an input low-resolution image, r (x) is residual error information obtained by learning, f is a residual error regulation factor, and y is an output image, namely a reconstructed high-resolution image.
Preferably, in the parameterized residual learning unit, the input image x is upsampled to the target magnification resolution by an image interpolation method so as to unify the inputs with different sizes; the input image x is overlapped on the residual information by skipping Long Skip connection, and the residual information r (x) is obtained by Conv learning of a plurality of cascaded convolutional layers.
Preferably, in the parameterized residual learning unit, the residual regulation factor f is calculated according to the magnification and is in a direct proportion relationship with the magnification, and the calculation formula is as follows:
where St, Si are the target magnification size and the input original low resolution image size, respectively.
Preferably, the basic computing unit of the residual refinement learning network RRNet is an existing residual learning unit.
Preferably, the optimal parameterized residual error learning network PRNEt and the residual error refined learning network RRNet are obtained through training;
the training sample set of the parameterized residual learning network PRNet is composed of original low-resolution LR image and original high-resolution HR image sample pairs and is constructed by the existing commonly adopted Gaussian down-sampling mode;
the training sample set of the residual error refined learning network RRNet is composed of a pair of a reconstructed high-resolution HR image and a residual error image sample; in order to generate a residual image sample required by training, subtracting a high-resolution HR image reconstructed by a parameterized residual learning network (PRNet) from an original high-resolution HR image to obtain a residual image sample; and generating a residual image sample for each image in the training sample set, and summing to form a residual image sample set.
Compared with the existing super-resolution reconstruction network based on deep learning, the super-resolution reconstruction network has the following advantages and positive effects:
1) by explicitly expressing the magnification factor parameters, a parameterized residual learning network model is established, so that the model can accept the input of any scale and meet the requirement of a variable-magnification super-resolution task.
2) And a residual refinement learning network is provided, and the mapping relation between the reconstructed high-resolution image and the reconstructed residual is further learned, so that residual compensation is performed on the reconstructed image, and the super-resolution reconstruction quality is improved.
Drawings
FIG. 1 is a structural diagram of a variable-magnification image super-resolution network model according to an embodiment of the present invention;
fig. 2 is a diagram of a parameterized residual unit according to an embodiment of the present invention.
Detailed Description
In order to facilitate the understanding and implementation of the present invention for those of ordinary skill in the art, the present invention is further described in detail with reference to the accompanying drawings and examples, it is to be understood that the embodiments described herein are merely illustrative and explanatory of the present invention and are not restrictive thereof.
Existing super-resolution networks can only be trained at one magnification at a time, and networks trained at one magnification (such as X2) cannot be applied to image magnifications at another magnification (such as X3). If any magnification is to be achieved, a network model must be trained for each magnification, which is impractical in real-world applications. The root of the problem is that the existing super-resolution reconstruction network does not explicitly express the variable of the magnification, and therefore, a parameterized residual error learning network model should be designed to meet the requirement of any scale input.
In addition, the super-resolution reconstructed image is not completely the same as the real high-resolution image, and usually has deviation, and if the deviation is compensated into the reconstruction result, the fineness, the fidelity and the naturalness of the super-resolution image can be improved. For this reason, a residual refinement network needs to be further designed to implement the compensation of the prediction residual.
Based on the above principle, please refer to fig. 1, the structure of the super-resolution image network model with variable magnification provided by the present invention is as follows: including parameterized residual learning networks (prnets), residual refined learning networks (rrnets), and overlay networks. The former learns the mapping between Low Resolution (LR) images to High Resolution (HR) images, and the latter learns the mapping between reconstructed high resolution images to residuals. The superposition network superposes the learning results of the two to form a final super-resolution (SR) image output.
In the parameterized residual learning network in fig. 1, the basic computing unit is a parameterized residual learning unit. In the residual refinement learning network in fig. 1, the basic computing unit is an existing residual learning unit.
An example of the design of the key module parameterized residual learning unit is further described below.
1) Parameterized residual learning unit
The closer the resolution of the input image is to the reconstructed target resolution, the less detailed information needs to be compensated by the super-resolution reconstruction algorithm, and when the input resolution is the same as the target resolution, the super-resolution algorithm does not even make any changes to the image. Therefore, the image information that needs to be supplemented by the super-resolution algorithm has a positive correlation with the magnification. The high-frequency content information to be supplemented can be measured by the residual error (corresponding to high-frequency detail components) between the low-resolution image and the real high-resolution image, so that the problem is converted into the relation between the residual error and the magnification factor.
In order to realize the idea, parameterized residual learning is introduced into the existing super-resolution network, the structure of a traditional residual learning unit is modified, the residual obtained by the residual network learning is regulated, the contribution proportion of the residual to super-resolution reconstruction is adjusted, the contribution is large at high magnification, the contribution is small at low magnification, and the residual does not participate in the reconstruction when the magnification is 1, and the original image is directly output, so that the information quantity of the residual participating in the super-resolution reconstruction is positively related to the magnification. Based on the idea, the parameterized residual unit structure with residual regulation and control capability designed by the invention is shown in fig. 2.
As can be seen from fig. 2, the formalized expression of the parameterized residual learning unit is:
y＝f·r(x)+x；
wherein x is an input low-resolution image, r (x) is residual error information obtained by learning, f is a residual error regulation factor, and y is an output image, namely a reconstructed high-resolution image. As can be seen from the formula, the proportion of the reconstructed image to supplement the residual depends on the residual regulation factor f.
In addition, the constructed parameterized residual learning unit also has the following design points:
(1) the input image x is upsampled to a target upscaling resolution by conventional image interpolation methods (e.g., Bicubic, Lanczos, etc.) to unify the inputs of varying sizes.
(2) The input image x is superimposed on the residual information by Long-range Skip (Long Skip) concatenation, and the residual information r (x) is obtained by conv.
(3) The residual regulation factor f is calculated according to the magnification and is in direct proportion with the magnification, and the calculation formula is as follows:
where St, Si are the target magnification size and the input original low resolution image size, respectively. The magnification minus 1 when calculating f is to output the original image directly at an identical resolution.
(4) Since f is calculated from the magnification, it ultimately depends on the magnification. When the resolution of input and output is not changed, f is equal to 0, and no residual information needs to be supplemented.
2) Network training
The training of the variable magnification image super-resolution network comprises the training of a parameterized residual learning network and the training of a residual refinement learning network.
The training sample set of the parameterized residual learning network is composed of original LR image and HR image sample pairs, and can be constructed by the existing commonly adopted Gaussian down-sampling mode.
The training sample set of the residual refined learning network is composed of reconstructed HR images and residual image sample pairs. In order to generate residual samples required by training, subtracting the HR image reconstructed by the parameterized residual learning network from the original HR image to obtain residual image samples. A residual sample map is generated for each image in the training sample set, and the residual sample maps are summed to form a residual sample set.
It should be understood that parts of the specification not set forth in detail are well within the prior art.
It should be understood that the above description of the preferred embodiments is given for clarity and not for any purpose of limitation, and that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (6)
1. A variable-magnification image super-resolution network model is characterized in that: the model comprises a parameterized residual error learning network PRNet, a residual error refined learning network RRNet and a superposition network;
the parameterized residual learning network PRNet is used for learning the mapping from the low-resolution LR image to the high-resolution HR image;
the residual error refinement learning network RRNet is used for learning and reconstructing mapping from a high-resolution image to a residual error image;
and the superposition network is used for superposing the high-resolution HR image and the residual image to form a final super-resolution SR image output.
2. The variable-magnification image super-resolution network model according to claim 1, wherein: the parameterized residual error learning network PRNet has a basic calculation unit of a parameterized residual error learning unit;
the parameterized residual learning unit regulates and controls residual obtained by residual image learning network learning, adjusts the contribution proportion of the residual to super-resolution reconstruction, greatly contributes at high magnification, and slightly contributes at low magnification, and directly outputs the original image without the residual participating in reconstruction when the magnification is 1, so that the information quantity of the residual participating in super-resolution reconstruction is positively related to the magnification;
the parameterized residual learning unit is formally expressed as:
y＝f·r(x)+x；
wherein x is an input low-resolution image, r (x) is residual error information obtained by learning, f is a residual error regulation factor, and y is an output image, namely a reconstructed high-resolution image.
3. The variable-magnification image super-resolution network model according to claim 2, wherein: in the parameterized residual error learning unit, an input image x is up-sampled to a target amplification resolution by an image interpolation method so as to unify the input of indefinite sizes; the input image x is overlapped on the residual information by skipping Long Skip connection, and the residual information r (x) is obtained by Conv learning of a plurality of cascaded convolutional layers.
4. The variable-magnification image super-resolution network model according to claim 2, wherein: in the parameterized residual error learning unit, a residual error regulation factor f is calculated according to the amplification factor and is in a direct proportion relation with the amplification factor, and the calculation formula is as follows:
where St, Si are the target magnification size and the input original low resolution image size, respectively.
5. The variable-magnification image super-resolution network model according to claim 1, wherein: and a basic calculation unit of the residual error refinement learning network RRNet is a residual error learning unit.
6. The variable-magnification image super-resolution network model according to any one of claims 1 to 5, wherein: the optimal parameterized residual learning network PRNEt and the residual refined learning network RRNet are obtained through training;
the training sample set of the parameterized residual learning network PRNet is composed of original low-resolution LR image and original high-resolution HR image sample pairs and is constructed by the existing commonly adopted Gaussian down-sampling mode;
the training sample set of the residual error refined learning network RRNet is composed of a pair of a reconstructed high-resolution HR image and a residual error image sample; in order to generate a residual image sample required by training, subtracting a high-resolution HR image reconstructed by a parameterized residual learning network (PRNet) from an original high-resolution HR image to obtain a residual image sample; and generating a residual image sample for each image in the training sample set, and summing to form a residual image sample set.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010075845.2A CN111260558B (en) | 2020-01-22 | 2020-01-22 | Image super-resolution network model with variable magnification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010075845.2A CN111260558B (en) | 2020-01-22 | 2020-01-22 | Image super-resolution network model with variable magnification |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111260558A true CN111260558A (en) | 2020-06-09 |
| CN111260558B CN111260558B (en) | 2021-10-22 |
Family
ID=70952635
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010075845.2A Active CN111260558B (en) | 2020-01-22 | 2020-01-22 | Image super-resolution network model with variable magnification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111260558B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023184525A1 (en) * | 2022-04-02 | 2023-10-05 | Covidien Lp | System and method for deep learning based hybrid image enlargement |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216889A (en) * | 2008-01-14 | 2008-07-09 | 浙江大学 | A face image super-resolution method with the amalgamation of global characteristics and local details information |
| CN107292821A (en) * | 2017-06-23 | 2017-10-24 | 武汉大学 | A kind of super-resolution image reconstruction method and system |
| CN108288251A (en) * | 2018-02-11 | 2018-07-17 | 深圳创维－Rgb电子有限公司 | Image super-resolution method, device and computer readable storage medium |
| CN108428212A (en) * | 2018-01-30 | 2018-08-21 | 中山大学 | A kind of image magnification method based on double laplacian pyramid convolutional neural networks |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109389552A (en) * | 2017-08-02 | 2019-02-26 | 中山大学 | A kind of Image Super-resolution based on context-sensitive multitask deep learning |
| CN109509152A (en) * | 2018-12-29 | 2019-03-22 | 大连海事大学 | A kind of image super-resolution rebuilding method of the generation confrontation network based on Fusion Features |
| US20190095795A1 (en) * | 2017-03-15 | 2019-03-28 | Samsung Electronics Co., Ltd. | System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions |
| RU2698649C1 (en) * | 2018-01-16 | 2019-08-29 | Акционерное общество "Федеральный научно-производственный центр "Нижегородский научно-исследовательский институт радиотехники" | Method of detecting and classifying small objects on images obtained by synthetic aperture radar stations |
| CN110211038A (en) * | 2019-04-29 | 2019-09-06 | 南京航空航天大学 | Super resolution ratio reconstruction method based on dirac residual error deep neural network |
| CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
-
2020
- 2020-01-22 CN CN202010075845.2A patent/CN111260558B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101216889A (en) * | 2008-01-14 | 2008-07-09 | 浙江大学 | A face image super-resolution method with the amalgamation of global characteristics and local details information |
| US20190095795A1 (en) * | 2017-03-15 | 2019-03-28 | Samsung Electronics Co., Ltd. | System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions |
| CN107292821A (en) * | 2017-06-23 | 2017-10-24 | 武汉大学 | A kind of super-resolution image reconstruction method and system |
| CN109389552A (en) * | 2017-08-02 | 2019-02-26 | 中山大学 | A kind of Image Super-resolution based on context-sensitive multitask deep learning |
| RU2698649C1 (en) * | 2018-01-16 | 2019-08-29 | Акционерное общество "Федеральный научно-производственный центр "Нижегородский научно-исследовательский институт радиотехники" | Method of detecting and classifying small objects on images obtained by synthetic aperture radar stations |
| CN108428212A (en) * | 2018-01-30 | 2018-08-21 | 中山大学 | A kind of image magnification method based on double laplacian pyramid convolutional neural networks |
| CN108288251A (en) * | 2018-02-11 | 2018-07-17 | 深圳创维－Rgb电子有限公司 | Image super-resolution method, device and computer readable storage medium |
| CN108734660A (en) * | 2018-05-25 | 2018-11-02 | 上海通途半导体科技有限公司 | A kind of image super-resolution rebuilding method and device based on deep learning |
| CN109509152A (en) * | 2018-12-29 | 2019-03-22 | 大连海事大学 | A kind of image super-resolution rebuilding method of the generation confrontation network based on Fusion Features |
| CN110276721A (en) * | 2019-04-28 | 2019-09-24 | 天津大学 | Image super-resolution rebuilding method based on cascade residual error convolutional neural networks |
| CN110211038A (en) * | 2019-04-29 | 2019-09-06 | 南京航空航天大学 | Super resolution ratio reconstruction method based on dirac residual error deep neural network |
Non-Patent Citations (2)
| Title |
|---|
| GUOSHENG LIN 等: "RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation", 《 2017 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》 * |
| 李梦溪: "基于特征融合和困难样例挖掘的图像语义分割", 《中国优秀博硕士学位论文全文数据库(硕士)信息科技辑》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023184525A1 (en) * | 2022-04-02 | 2023-10-05 | Covidien Lp | System and method for deep learning based hybrid image enlargement |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111260558B (en) | 2021-10-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Sun et al. | Learned image downscaling for upscaling using content adaptive resampler | |
| Gao et al. | Implicit diffusion models for continuous super-resolution | |
| Zheng et al. | Crossnet: An end-to-end reference-based super resolution network using cross-scale warping | |
| CN109949221B (en) | Image processing method and electronic equipment | |
| CN114549731A (en) | Method and device for generating visual angle image, electronic equipment and storage medium | |
| Li et al. | Dual-stage approach toward hyperspectral image super-resolution | |
| Chadha et al. | iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks | |
| CN110335330B (en) | Image simulation generation method and system, deep learning algorithm training method and electronic equipment | |
| CN103985085A (en) | Image super-resolution amplifying method and device | |
| Liu et al. | Iterative network for image super-resolution | |
| Dai et al. | Image super-resolution via residual block attention networks | |
| Cai et al. | TDPN: Texture and detail-preserving network for single image super-resolution | |
| Zhang et al. | Implicit neural representation learning for hyperspectral image super-resolution | |
| CN111260558B (en) | Image super-resolution network model with variable magnification | |
| CN113570536A (en) | Panchromatic and multispectral image real-time fusion method based on CPU and GPU cooperative processing | |
| Huang et al. | Refsr-nerf: Towards high fidelity and super resolution view synthesis | |
| CN115526777A (en) | Blind over-separation network establishing method, blind over-separation method and storage medium | |
| Wang et al. | DDistill-SR: Reparameterized dynamic distillation network for lightweight image super-resolution | |
| CN113628115A (en) | Image reconstruction processing method and device, electronic equipment and storage medium | |
| CN113724134A (en) | Aerial image blind super-resolution reconstruction method based on residual distillation network | |
| CN113487482A (en) | Self-adaptive super-resolution method based on meta-migration learning | |
| Liu et al. | From coarse to fine: Hierarchical pixel integration for lightweight image super-resolution | |
| CN113658046B (en) | Super-resolution image generation method, device, equipment and medium based on feature separation | |
| CN116309066A (en) | Super-resolution imaging method and device | |
| Yuan et al. | Unsupervised real image super-resolution via knowledge distillation network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |