CN115564653A - Multi-factor fusion image super-resolution method - Google Patents
Multi-factor fusion image super-resolution method Download PDFInfo
- Publication number
- CN115564653A CN115564653A CN202211209153.8A CN202211209153A CN115564653A CN 115564653 A CN115564653 A CN 115564653A CN 202211209153 A CN202211209153 A CN 202211209153A CN 115564653 A CN115564653 A CN 115564653A
- Authority
- CN
- China
- Prior art keywords
- image
- convolution
- super
- data
- resolution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 27
- 230000004927 fusion Effects 0.000 title description 6
- 238000005070 sampling Methods 0.000 claims abstract description 21
- 238000003780 insertion Methods 0.000 claims abstract description 6
- 230000037431 insertion Effects 0.000 claims abstract description 5
- 230000008521 reorganization Effects 0.000 claims abstract description 5
- 238000013135 deep learning Methods 0.000 abstract description 4
- 238000005215 recombination Methods 0.000 abstract description 3
- 230000006798 recombination Effects 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 5
- 230000003321 amplification Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a multi-factor fused image super-resolution method, which comprises the following steps: 1) Acquiring an image to be processed; 2) Performing up-sampling on the image through an interpolation algorithm, and recording the operation as BI; 3) Performing upsampling on an image through a convolution operation, wherein the operation is marked as ZI; 4) Upsampling the image by a pixel reorganization operation, which is denoted as MI; 5) And inputting the results subjected to the up-sampling of the three operations into the weight convolution layer, and outputting a final super-resolution image. The invention discloses a deep learning network integrating a plurality of upsampling modes, which integrates common interpolation, zero insertion convolution and pixel recombination, and can form a better output result by utilizing the advantages of different methods.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a multi-factor fusion image super-resolution method.
Background
The image super-resolution is a very important research problem in the fields of computer vision and image processing, and is widely applied to practical scenes such as medical image analysis, biological feature recognition, video monitoring, safety and the like. The traditional image interpolation amplification method can only improve the resolution of an image through the content of the image, does not bring more information, and has the side effects of noise amplification, increased calculation complexity, fuzzy result and the like. With the development of deep learning technology, the image hyper-segmentation method based on deep learning obtains the currently optimal performance and effect on a plurality of test tasks.
Disclosure of Invention
The purpose of the invention is as follows: aiming at the prior art, a multi-factor fusion image super-resolution method is provided.
A multi-factor fused image super-resolution method comprises the following steps:
1) Acquiring an image to be processed;
2) Performing up-sampling on the image through an interpolation algorithm, and recording the operation as BI;
3) Performing up-sampling on an image through convolution operation, wherein the operation is marked as ZI;
4) Upsampling the image by a pixel reorganization operation, which is denoted as MI;
5) And inputting the results subjected to the up-sampling of the three operations into the weight convolution layer, and outputting a final super-resolution image.
Preferably, in step 2), the image I is obtained by a bicubic interpolation algorithm, the operation is denoted as BI, and the interpolated data is denoted as OBI (I).
Preferably, in step 3), the convolutional network is denoted as O, the operation is denoted as ZI, and the output data of the convolutional network is denoted as O (ZI (I)).
Preferably, in step 4), 4 sets of convolution are performed on the image I to generate 4 channels of data c1, c2, c3, c4, and if the size of the data is consistent with that of the image to be processed, the output upsampling data c is:
c(2*i,2*j)＝c1(i,j)
c(2*i+1,2*j)＝c2(i,j)
c(2*i+1,2*j+1)＝c3(i,j)
c(2*i,2*j+1)＝c4(i,j)
wherein i, j are pixel coordinates; the pixel reorganization operation is denoted as MI, and the output data is denoted as O (MI (I)).
Preferably, the final result SR (I) is obtained by weighting and fusing the plurality of results by the dynamic weights α, β, γ:
SR(I)＝αO(BI(I))+βO(ZI(I))+γO(MI(I))
preferably, in step 5), a 1 × 1 weight convolution layer is added, the final result SR (I) is input, and the final super-resolution image is output.
Has the advantages that: the invention designs a deep learning network integrating a plurality of upsampling modes, integrates common interpolation, zero insertion convolution and pixel recombination, and can utilize the advantages of different methods to form a better output result.
Detailed Description
The present invention is further explained below.
The method is explained based on the structure of the front-end up-sampling super-resolution network, but the method is applicable to all super-resolution networks with up-sampling structure layers.
For the up-sampling layer in the network, in order to obtain the image with the same resolution by the following up-sampling mode, an interpolation method of respectively enlarging the length and the width by one time is adopted. And calculating the interpolated image by adopting various interpolation strategies.
And respectively adopting a nearest neighbor interpolation method, bilinear interpolation, bicubic interpolation of a 4x4 pixel neighborhood and bicubic interpolation of an 8x8 pixel neighborhood to obtain 4 groups of up-sampled images, including three rgb channels, and obtaining a data block of 12 channels in total. Then, 64 convolution filters of 3x3x12 are connected to obtain data of 64 channels; and then a convolution filter of 1x1x64 is carried out to obtain an image up-sampling result integrating a plurality of interpolation methods. The operation is recorded as BI, the image is recorded as I, the data after interpolation is recorded as BI (I), the subsequent convolution network is recorded as O1, and the output data is recorded as O 1 (BI (I)). The loss function uses pixel difference loss to optimally compute the network model.
The zero insertion convolution also considers the condition that the length and the width of the image are doubled, and after the original image is doubled by zero insertion, the original image passes through 64 3x3 convolution filters and a 1x1x64 channel transformation filter to output an up-sampling result. The operation is denoted as ZI, the same subsequent network is adopted, and the output data is denoted as O 2 (ZI (I)). The loss function uses pixel difference loss to optimally compute the network model.
And (4) performing pixel recombination interpolation, and performing 4 groups of convolution on the image I by using 4 groups of convolution, wherein the convolution comprises 64 3x3 convolution filters and a 1x1x64 channel transformation filter to generate 4 channels of data c1, c2, c3 and c4, and the size of the data is consistent with that of the original image. The output up-sampled data c is:
c(2*i,2*j)＝c1(i,j)
c(2*i+1,2*j)＝c2(i,j)
c(2*i+1,2*j+1)＝c3(i,j)
c(2*i,2*j+1)＝c4(i,j)
where i, j are pixel coordinates. When the pixel rearrangement operation is recorded as MI, the output data is recorded as O 3 The (MI (I)) loss function optimally computes the network model using pixel difference loss.
For the network O, the same parameter structure exists for the above three super-resolution methods, and the network parameters are not equal due to the respective individual optimization. In order to improve the operation efficiency, a mode of fusing three groups of network parameters for optimizing again is adopted, and O = (O) 1 +O 2 +O 3 ) The parameters of/3,O are the average values of the O1, O2, O3 parameters. The output of the new network is noted as: o (BI (I)) + O (ZI (I)) + O (MI (I)), which is the 3-group rgb data after ultrafractionation.
Because of different up-sampling means, the recovery results of the details are different and have advantages, in order to synthesize the results of different up-sampling, 64 3x3x9 convolution filters are connected in sequence to obtain data of a 64-channel; and then a convolution filter of 1x1x64 is used to obtain an image up-sampling result integrating three super-resolution methods. The loss function uses pixel difference loss to optimally compute the entire network model.
The last convolution filter, 1x1x64, is equivalent to performing a weighted fusion calculation on the differently up-sampled outputs of the 3 channels using dynamic weights. Thus, a final super-resolution image with better comprehensive effect than that of a plurality of up-sampling modes is obtained. The loss function can adopt the traditional pixel difference loss and the like, and the network optimization method is also unchanged.
For the Unet and its variant networks, such as DRN networks, the method is also applicable, at the key up-sampling layer, a plurality of up-sampling operations are used for replacement, and a weight fusion operation is added to the final output.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (6)
1. A multi-factor fused image super-resolution method is characterized by comprising the following steps:
1) Acquiring an image to be processed;
2) Performing up-sampling on the image through an interpolation algorithm, and recording the operation as BI;
3) Performing upsampling on the image through zero insertion convolution operation, wherein the operation is marked as ZI;
4) Upsampling the image by a pixel reorganization operation denoted as MI;
5) And inputting the results subjected to the up-sampling of the three operations into the weight convolution layer, and outputting a final super-resolution image.
2. The multi-factor fused image super-resolution method according to claim 1, wherein in step 2), the images to be processed are up-sampled by using a nearest neighbor interpolation algorithm, a bilinear interpolation algorithm, a bicubic interpolation algorithm of a 4x4 pixel neighborhood, and a bicubic interpolation algorithm of an 8x8 pixel neighborhood, respectively, to obtain 4 groups of up-sampled images, each group of up-sampled images includes three rgb channels, a data block of 12 channels is obtained in total, and 64 convolution filters of 3x3x12 are connected behind the data block to obtain a data block of 64 channels; and obtaining an image up-sampling result integrating multiple interpolation methods through a convolution filter of 1x1x64, wherein the operation is marked as BI, the interpolated data is marked as BI (I), and the convolution network is marked as O 1 And the output data is recorded as O 1 (BI(I))。
3. The method for super-resolution of multi-factor fused images as claimed in claim 2, wherein in step 3), after the original image is multiplied by zero-insertion convolution, the up-sampled result is output through 64 3x3 convolution filters and a 1x1x64 channel transform filter, the operation is denoted as ZI, and the convolution network is denoted as O 2 And the output data is marked as O 2 (ZI(I))。
4. The multi-factor fused image super-resolution method as claimed in claim 3, wherein in step 4), the image I is convolved by 4 sets to generate 4 channels of data c1, c2, c3, c4, and if the size of the data is consistent with that of the image to be processed, the upsampled data c is output as:
c(2*i,2*j)＝c1(i,j)
c(2*i+1,2*j)＝c2(i,j)
c(2*i+1,2*j+1)＝c3(i,j)
c(2*i,2*j+1)＝c4(i,j)
wherein i, j are pixel coordinates; record the pixel reorganization operation as MI and the output data as O 3 (MI(I))。
5. The multi-factor fused image super resolution method according to claim 4, wherein the final output result SR (I) is obtained by weighting and fusing a plurality of results by dynamic weights α, β, γ:
SR(I)＝αO 1 (BI(I))+βO 2 (ZI(I))+γO 3 (MI(I))。
6. the method for super-resolution of multi-factor fused images as claimed in claim 5, wherein in step 5), in order to synthesize the results of different upsampling, the final output result SR (I) is followed by 64 convolution filters of 3x3x9 to obtain 64 channels of data; and then obtaining an image up-sampling result integrating three super-resolution methods through a convolution filter of 1x1x 64.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211209153.8A CN115564653A (en) | 2022-09-30 | 2022-09-30 | Multi-factor fusion image super-resolution method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211209153.8A CN115564653A (en) | 2022-09-30 | 2022-09-30 | Multi-factor fusion image super-resolution method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115564653A true CN115564653A (en) | 2023-01-03 |
Family
ID=84743884
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211209153.8A Pending CN115564653A (en) | 2022-09-30 | 2022-09-30 | Multi-factor fusion image super-resolution method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115564653A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150104116A1 (en) * | 2012-03-05 | 2015-04-16 | Thomason Licensing | Method and apparatus for performing super-resolution |
| CN110942424A (en) * | 2019-11-07 | 2020-03-31 | 昆明理工大学 | Composite network single image super-resolution reconstruction method based on deep learning |
| CN111161150A (en) * | 2019-12-30 | 2020-05-15 | 北京工业大学 | Image super-resolution reconstruction method based on multi-scale attention cascade network |
| CN111951167A (en) * | 2020-08-25 | 2020-11-17 | 深圳思谋信息科技有限公司 | Super-resolution image reconstruction method, super-resolution image reconstruction device, computer equipment and storage medium |
| CN113822803A (en) * | 2021-07-22 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Image super-resolution processing method, device, equipment and computer readable storage medium |
| CN114219738A (en) * | 2021-12-30 | 2022-03-22 | 中国电建集团中南勘测设计研究院有限公司 | Single-image multi-scale super-resolution reconstruction network structure and method |
| WO2022057868A1 (en) * | 2020-09-21 | 2022-03-24 | 华为技术有限公司 | Image super-resolution method and electronic device |
| CN114943643A (en) * | 2022-04-12 | 2022-08-26 | 浙江大华技术股份有限公司 | Image reconstruction method, image coding and decoding method and related equipment |
| CN114998101A (en) * | 2022-05-26 | 2022-09-02 | 电子科技大学 | Satellite image super-resolution method based on deep learning |
-
2022
- 2022-09-30 CN CN202211209153.8A patent/CN115564653A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150104116A1 (en) * | 2012-03-05 | 2015-04-16 | Thomason Licensing | Method and apparatus for performing super-resolution |
| CN110942424A (en) * | 2019-11-07 | 2020-03-31 | 昆明理工大学 | Composite network single image super-resolution reconstruction method based on deep learning |
| CN111161150A (en) * | 2019-12-30 | 2020-05-15 | 北京工业大学 | Image super-resolution reconstruction method based on multi-scale attention cascade network |
| CN111951167A (en) * | 2020-08-25 | 2020-11-17 | 深圳思谋信息科技有限公司 | Super-resolution image reconstruction method, super-resolution image reconstruction device, computer equipment and storage medium |
| WO2022057868A1 (en) * | 2020-09-21 | 2022-03-24 | 华为技术有限公司 | Image super-resolution method and electronic device |
| CN113822803A (en) * | 2021-07-22 | 2021-12-21 | 腾讯科技（深圳）有限公司 | Image super-resolution processing method, device, equipment and computer readable storage medium |
| CN114219738A (en) * | 2021-12-30 | 2022-03-22 | 中国电建集团中南勘测设计研究院有限公司 | Single-image multi-scale super-resolution reconstruction network structure and method |
| CN114943643A (en) * | 2022-04-12 | 2022-08-26 | 浙江大华技术股份有限公司 | Image reconstruction method, image coding and decoding method and related equipment |
| CN114998101A (en) * | 2022-05-26 | 2022-09-02 | 电子科技大学 | Satellite image super-resolution method based on deep learning |
Non-Patent Citations (1)
| Title |
|---|
| 韩亮;李婵飞;蒲秀娟;: "图像分割与平稳小波变换法融合红外与可见光图像", 重庆大学学报, no. 06 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111161150B (en) | Image super-resolution reconstruction method based on multi-scale attention cascade network | |
| CN108765296B (en) | Image super-resolution reconstruction method based on recursive residual attention network | |
| CN107123089B (en) | Remote sensing image super-resolution reconstruction method and system based on depth convolution network | |
| CN111028150B (en) | Rapid space-time residual attention video super-resolution reconstruction method | |
| CN109903223B (en) | Image super-resolution method based on dense connection network and generation type countermeasure network | |
| CN108537733A (en) | Super resolution ratio reconstruction method based on multipath depth convolutional neural networks | |
| CN109934771B (en) | Unsupervised remote sensing image super-resolution reconstruction method based on recurrent neural network | |
| CN112308803B (en) | Self-supervision low-illumination image enhancement and denoising method based on deep learning | |
| CN110889895A (en) | Face video super-resolution reconstruction method fusing single-frame reconstruction network | |
| CN112508794B (en) | Medical image super-resolution reconstruction method and system | |
| CN112017116A (en) | Image super-resolution reconstruction network based on asymmetric convolution and construction method thereof | |
| CN115358929B (en) | Compressed image super-resolution method, image compression method and system | |
| CN117372564B (en) | Method, system and storage medium for reconstructing multispectral image | |
| Guo et al. | Joint demosaicking and denoising benefits from a two-stage training strategy | |
| JP2014042176A (en) | Image processing device and method, program and solid image pickup device | |
| CN104853059A (en) | Super-resolution image processing method and device | |
| CN115564653A (en) | Multi-factor fusion image super-resolution method | |
| CN116823662A (en) | Image denoising and deblurring method fused with original features | |
| Chung et al. | New joint demosaicing and arbitrary-ratio resizing algorithm for color filter array based on DCT approach | |
| CN115760638A (en) | End-to-end deblurring super-resolution method based on deep learning | |
| Saito et al. | Demosaicing approach based on extended color total-variation regularization | |
| Wang et al. | Multi-scale detail enhancement network for image super-resolution | |
| KR20070119482A (en) | Image resampling method | |
| CN111242087A (en) | Object recognition method and device | |
| CN115937018B (en) | Method for restoring image by using multichannel feedback network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230103 |