CN114359069A - Image recovery method, system, device and storage medium - Google Patents
Image recovery method, system, device and storage medium Download PDFInfo
- Publication number
- CN114359069A CN114359069A CN202111492256.5A CN202111492256A CN114359069A CN 114359069 A CN114359069 A CN 114359069A CN 202111492256 A CN202111492256 A CN 202111492256A CN 114359069 A CN114359069 A CN 114359069A
- Authority
- CN
- China
- Prior art keywords
- feature map
- image
- low
- feature
- global
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 52
- 238000011084 recovery Methods 0.000 title claims abstract description 27
- 238000005286 illumination Methods 0.000 claims abstract description 62
- 230000004927 fusion Effects 0.000 claims abstract description 42
- 230000004913 activation Effects 0.000 claims abstract description 31
- 238000012549 training Methods 0.000 claims abstract description 29
- 238000010586 diagram Methods 0.000 claims abstract description 27
- 238000005070 sampling Methods 0.000 claims abstract description 23
- 238000000605 extraction Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 26
- 230000006870 function Effects 0.000 claims description 21
- 230000009466 transformation Effects 0.000 claims description 11
- 238000011176 pooling Methods 0.000 claims description 6
- 238000012935 Averaging Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 2
- 230000001965 increasing effect Effects 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 7
- 230000009286 beneficial effect Effects 0.000 description 4
- 239000000284 extract Substances 0.000 description 3
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Landscapes
- Image Processing (AREA)
Abstract
The invention discloses an image recovery method, a system, a device and a storage medium, wherein the method comprises the following steps: acquiring a low-illumination image pair, and selecting a training set according to the low-illumination image pair; training an image recovery model by adopting the training set; acquiring a low-illumination image needing to be restored, inputting the low-illumination image into the trained image restoration model, and outputting the restored low-illumination image; after the low-illumination image is input into the image recovery model, the low-illumination image is processed by a convolution layer and an activation layer to obtain a main feature map with the channel number of 32, then downsampling is performed for three times in sequence to obtain three feature maps with different sizes, and feature extraction and feature fusion are performed on the three feature maps. According to the method, the characteristic diagram is firstly subjected to down sampling for multiple times, and then the characteristic extraction and the up sampling fusion are carried out on different characteristic diagrams, so that the depth of the network is reduced by increasing the width of the network. The invention can be widely applied to the technical field of image restoration.
Description
Technical Field
The present invention relates to the field of image restoration technologies, and in particular, to an image restoration method, system, apparatus, and storage medium.
Background
Pictures taken during nighttime low light conditions tend to exhibit low visual perception, low contrast, and poor color contrast. In the existing image enhancement method, the deep convolutional neural network performs better in the aspect of image enhancement, but the deep convolutional neural network performs poor in the aspects of maintaining the spatial characteristics of the image and denoising. Particularly, the conventional U-net network structure has high cpu occupancy rate and low running speed, and cannot effectively extract the structural features of the image; with the deepening of the U-net layers, noise in the image is often more obvious, regularization parameters need to be carefully selected, and the regularization parameters are difficult to automatically adapt to various scenes.
Disclosure of Invention
To solve at least some of the technical problems in the prior art, the present invention provides an image restoration method, system, device and storage medium.
The technical scheme adopted by the invention is as follows:
an image restoration method comprising the steps of:
acquiring a low-illumination image pair, and selecting a training set according to the low-illumination image pair;
training an image recovery model by adopting the training set;
acquiring a low-illumination image needing to be restored, inputting the low-illumination image into the trained image restoration model, and outputting the restored low-illumination image;
wherein the image restoration model performs the following operations on the low-illumination picture:
processing the low-illumination picture by a convolution layer and an activation layer to obtain a main feature map with the channel number of 32, and sequentially performing down-sampling for three times to obtain a first feature map, a second feature map and a third feature map under different sizes;
extracting features of the first feature map to obtain a first global feature map and a first local feature map, and fusing the first global feature map and the first local feature map to obtain a first fused feature map;
extracting features of the second feature map to obtain a second global feature map and a second local feature map, and fusing the second global feature map, the second local feature map and the first fusion feature map to obtain a second fusion feature map;
extracting features of the third feature map to obtain a third global feature map and a third local feature map, and fusing the third global feature map, the third local feature map and the second fusion feature map to obtain a third fusion feature map;
and fusing the main feature graph and the third fused feature graph, and outputting a recovered picture.
Further, the downsampling of the main feature map for three times to obtain a first feature map, a second feature map and a third feature map under different sizes includes:
performing discrete wavelet transformation on the main feature graph with the channel number of 32 to obtain a feature graph with the channel number of 32 × 4 — 128, performing processing on the feature graph by a convolution layer and an activation layer, and halving the channel number to obtain a first feature graph with the channel number of 64;
performing discrete wavelet transformation on the first feature map to obtain a feature map with the channel number of 64 × 4-256, performing processing on the feature map by a convolution layer and an activation layer, and halving the channel number to obtain a second feature map with the channel number of 128;
and performing discrete wavelet transformation on the second feature map to obtain a feature map with the channel number of 128 × 4-512, performing processing on the feature map by one convolution layer and an activation layer, and halving the channel number to obtain a third feature map with the channel number of 256.
Further, feature extraction is carried out on feature maps of different sizes in the following mode:
acquiring a maximum pixel value and an average pixel value of the feature map through a global pooling layer and an average pooling layer, averaging the average pixel value and the global pixel value to serve as a standard pixel value of the feature map, performing fine adjustment through a convolution layer and an activation layer to serve as an adjustment pixel value of the feature map, performing addition fusion on the adjustment pixel value and the feature map, and performing processing through a convolution layer and an activation layer to serve as global features of the feature map;
processing the feature map by two residual error layers (residual error layers) to obtain local information of the feature map, and adding and fusing the obtained local information and the feature map to obtain local features of the feature map;
splicing and fusing the global features and the local features to obtain a fused feature map;
and processing the fused feature map by inverse discrete wavelet transform to obtain a feature map with the channel number of 1/4 of the fused feature map, and converting the feature map into a feature map with the channel number of 1/2 of the fused feature map by a convolutional layer and an activation layer for the next fusion operation.
Further, the fusing the main feature map and the third fused feature map and outputting the restored picture includes:
and splicing the main feature map and the third fusion feature map, then performing ResBlock residual error layer fusion processing, and finally obtaining a result feature map through a convolutional layer and an activation layer to serve as an output recovered picture.
Further, the image restoration method further includes the step of training the image restoration model:
inputting the low exposure image into the image recovery model, and outputting a characteristic image;
and constructing a multi-scale structure similarity loss function and a mean absolute error loss function according to the high exposure image and the feature image output by the model, and performing parameter optimization on the image recovery model.
The other technical scheme adopted by the invention is as follows:
an image restoration system comprising:
the data acquisition module is used for acquiring a low-illumination image pair and selecting a training set according to the low-illumination image pair;
the model training module is used for training an image recovery model by adopting the training set;
the image recovery module is used for acquiring a low-illumination image needing to be recovered, inputting the low-illumination image into the trained image recovery model, and outputting the recovered low-illumination image;
wherein the image restoration model performs the following operations on the low-illumination picture:
processing the low-illumination picture by a convolution layer and an activation layer to obtain a main feature map with the channel number of 32, and sequentially performing down-sampling for three times to obtain a first feature map, a second feature map and a third feature map under different sizes;
extracting features of the first feature map to obtain a first global feature map and a first local feature map, and fusing the first global feature map and the first local feature map to obtain a first fused feature map;
extracting features of the second feature map to obtain a second global feature map and a second local feature map, and fusing the second global feature map, the second local feature map and the first fusion feature map to obtain a second fusion feature map;
extracting features of the third feature map to obtain a third global feature map and a third local feature map, and fusing the third global feature map, the third local feature map and the second fusion feature map to obtain a third fusion feature map;
and fusing the main feature graph and the third fused feature graph, and outputting a recovered picture.
The other technical scheme adopted by the invention is as follows:
an image restoration apparatus comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method described above.
The other technical scheme adopted by the invention is as follows:
a computer readable storage medium in which a processor executable program is stored, which when executed by a processor is for performing the method as described above.
The invention has the beneficial effects that: according to the method, the characteristic diagram is firstly subjected to down sampling for multiple times, and then the characteristic extraction and the up sampling fusion are carried out on different characteristic diagrams, so that the depth of the network is reduced by increasing the width of the network.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description is made on the drawings of the embodiments of the present invention or the related technical solutions in the prior art, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solutions of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic structural diagram of a self-directed network model in an embodiment of the invention;
FIG. 2 is a schematic structural diagram of a feature extraction module in an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a global attention network in an embodiment of the present invention;
FIG. 4 is a schematic structural diagram of ResBlock in an embodiment of the present invention;
FIG. 5 is a schematic diagram of a conventional U-net architecture in an embodiment of the present invention;
FIG. 6 is a first diagram illustrating recovery via a self-directed network model in an embodiment of the invention;
fig. 7 is a second schematic diagram of recovery by the self-guided network model in the embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided only for convenience of illustration, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
In the description of the present invention, it should be understood that the orientation or positional relationship referred to in the description of the orientation, such as the upper, lower, front, rear, left, right, etc., is based on the orientation or positional relationship shown in the drawings, and is only for convenience of description and simplification of description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus, should not be construed as limiting the present invention.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and larger, smaller, larger, etc. are understood as excluding the number, and larger, smaller, inner, etc. are understood as including the number. If the first and second are described for the purpose of distinguishing technical features, they are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
In the description of the present invention, unless otherwise explicitly limited, terms such as arrangement, installation, connection and the like should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meanings of the above terms in the present invention in combination with the specific contents of the technical solutions.
The present embodiment provides an image restoration model (hereinafter referred to as a bootstrap network model) and a conventional U-net structure will be described before describing the bootstrap network model. Fig. 5 is a schematic diagram of a conventional U-net structure, and as shown in fig. 5, the conventional U-net structure adopts a process of down-sampling and up-sampling, and the process of down-sampling and up-sampling is performed after several layers of convolutional layers are passed in the down-sampling process, so that the number of network layers is increased with the increase of up-and-down sampling times, the time overhead of network training is increased with the increase of the number of convolutional network layers, and more original information of the input feature map is lost.
In order to reduce the depth of the network and fully utilize the parallelization capability of the system, the depth of the network is reduced by enhancing the width of the network, as shown in fig. 1, the self-guided network model of the embodiment first directly performs a plurality of times of feature maps on an original input feature map, changes the process of downsampling convolution and then downsampling in the past, performs feature extraction on an original image to obtain a main feature map with the number of channels being 32, performs downsampling on the main feature map for three times, and then extracts corresponding feature map information respectively on feature maps with different sizes.
Referring to fig. 1, the self-steering network model adopts a top-down guiding strategy structure, firstly performs down-sampling on the feature map, and then performs feature extraction and up-sampling fusion on different feature maps, so that the depth of the network is reduced by increasing the width of the network, and meanwhile, the context information of the top sub-network can be conducted to a sub-network with higher resolution by using a top-down strategy, so as to guide branches with higher resolution, and the network depth is reduced by increasing the width of the network as a whole. Wherein, the terms in FIG. 1 are defined as follows, MS-SSIM: multi-scale structural similarity function, IDWT: inverse discrete wavelet transform, DWT: discrete wavelet transform, GAM: global Attention module Global Attention network.
Referring to fig. 1, after a picture is input into the self-guided network model, the model performs the following processing on the picture: firstly, an input picture is cut into 256 × 256 feature maps, the 256 × 256 feature maps pass through a convolution layer and an activation layer to obtain 32-channel-number main feature maps, then, DWT (discrete wavelet transform) is used for obtaining low-pass coefficients yl and band-pass coefficients yh corresponding to the 256 × 256 feature maps, then, 2x down-sampling is carried out on the yl and the yh to obtain 128 × 128 feature maps, then, a convolution layer is used for obtaining compressed feature maps which are half of the former 64 × 128, and similar operations are carried out, so that 32 × 64, 64 × 128, 128 × 64, and 32 × 256 feature maps are obtained from bottom to top. The feature map obtained from each layer fully extracts effective information in the feature map through an FEM (feature extraction module), then obtains a corresponding 2x up-sampling feature map through IDWT (inverse discrete wavelet transform), sequentially splices the feature maps with low resolution from top to bottom, and then obtains a restored feature map through the lowest branch (main line branch). In the training process, an L1loss (mean absolute value error function) and an MS-SSIM loss (multi-scale structure similarity loss function) are used as the overall loss function of the network, and an Adam optimizer is used for carrying out multiple iterations to obtain an ideal model. In the actual use process, a better picture (high-quality picture) can be recovered by loading the ideal model, and carrying out operations such as convolution, scaling, splicing and the like in the image with low illumination through the self-guided network.
As shown in fig. 2, in the feature extraction module, local features are obtained through two sequentially connected resblocks, and global features are obtained through GAM (global attention network). Fig. 3 is a schematic diagram of a global attention network, fig. 4 is a schematic diagram of ResBlock, and ResBlock includes two blocks, each of which is composed of a convolution layer with a convolution kernel of 3 × 3 and an activation layer with prilu as an activation function.
Based on the above automatic network model, this embodiment provides an image restoration method, including the following steps:
s101, acquiring a low-illumination image pair, wherein the low-illumination image pair comprises a low exposure image and a high exposure image, and selecting a training set according to the low-illumination image pair;
s102, training an image recovery model by adopting a training set;
s103, acquiring a low-illumination picture to be restored, inputting the low-illumination picture into the trained image restoration model, and outputting the restored low-illumination picture;
the image recovery model performs the following operations on the low-illumination picture:
processing the low-illumination picture by a convolution layer and an activation layer to obtain a main feature map with the channel number of 32, and sequentially performing down-sampling for three times to obtain a first feature map, a second feature map and a third feature map under different sizes;
extracting features of the first feature map to obtain a first global feature map and a first local feature map, and fusing the first global feature map and the first local feature map to obtain a first fused feature map;
extracting features of the second feature map to obtain a second global feature map and a second local feature map, and fusing the second global feature map, the second local feature map and the first fusion feature map to obtain a second fusion feature map;
extracting features of the third feature map to obtain a third global feature map and a third local feature map, and fusing the third global feature map, the third local feature map and the second fusion feature map to obtain a third fusion feature map;
and fusing the main feature graph and the third fused feature graph, and outputting the recovered picture.
In this embodiment, the low-light pictures include a low-light picture taken at night and a low-light picture taken in an area with poor light in the daytime. The low exposure image is an image with low exposure rate and obvious image noise, and the high exposure image is an image with sufficient exposure and low noise. After the picture to be restored is input into the self-guided network model, the model firstly carries out down-sampling on the input picture for three times to obtain a plurality of feature maps with different sizes. The feature images can be simultaneously extracted, and the extracted features are fused, so that the efficiency of recovering image output is improved.
As a further optional implementation, the downsampling is performed three times on the main feature map to obtain the first feature map, the second feature map and the third feature map under different sizes, and the downsampling includes:
performing discrete wavelet transformation on the main feature graph with the channel number of 32 to obtain a feature graph with the channel number of 32 × 4 — 128, performing processing on the feature graph by a convolution layer and an activation layer, and halving the channel number to obtain a first feature graph with the channel number of 64;
performing discrete wavelet transformation on the first feature map to obtain a feature map with the channel number of 64 × 4-256, performing processing on the feature map by a convolution layer and an activation layer, and halving the channel number to obtain a second feature map with the channel number of 128;
and performing discrete wavelet transformation on the second feature map to obtain a feature map with the channel number of 128 × 4-512, performing processing on the feature map by one convolution layer and an activation layer, and halving the channel number to obtain a third feature map with the channel number of 256.
Further as an optional implementation manner, the feature extraction is performed on the feature map in the following manner:
acquiring a maximum pixel value and an average pixel value of the feature map through a global pooling layer and an average pooling layer, averaging the average pixel value and the global pixel value to serve as a standard pixel value of the feature map, performing fine adjustment through a convolution layer and an activation layer to serve as an adjustment pixel value of the feature map, performing addition fusion on the adjustment pixel value and the feature map, and performing processing through a convolution layer and an activation layer to serve as global features of the feature map;
processing the feature map by two layers of residual error layers to obtain local information of the feature map, and adding and fusing the obtained local information and the feature map to obtain local features of the feature map;
splicing and fusing the global features and the local features to obtain a fused feature map;
and processing the fused feature map by inverse discrete wavelet transform to obtain a feature map with the channel number of 1/4, and converting the feature map into a feature map with the channel number of 1/2 through a convolutional layer and an activation layer for the next fusion operation.
As a further optional implementation, fusing the main feature map and the third fused feature map, and outputting the recovered picture, including:
and splicing the main characteristic diagram and the third fusion characteristic diagram, then performing ResBlock (residual error layer) fusion processing, and finally obtaining a result characteristic diagram through a convolutional layer and an activation layer to serve as an output recovered image.
The above method is explained in detail below with reference to specific examples.
An image enhancement method for nighttime low-light image restoration, comprising the steps of:
and S1, acquiring an image with low illumination at night.
And S2, collecting the night pictures by adopting different exposure degrees and exposure times, and screening out the low-exposure night pictures and the high-exposure night pictures.
S3, constructing 400 pairs of training sets of night low-exposure pictures and high-exposure pictures, and selecting 15 additional pairs of night pictures as test sets and target reference pictures for comparison.
And S4, inputting the night picture into an SGN (self-guided network) to obtain feature maps with different feature map sizes, and extracting image feature information of different region points.
Wherein, the step S4 includes steps S41-S43:
and S41, performing one-layer convolution on the input low-illumination feature map to obtain a feature map with the channel number of 32, and then performing discrete wavelet transform to compress the size of the feature map.
And S42, after the characteristic diagram size is compressed by wavelet transformation, the number of channels is changed to half of the number of channels by a layer of convolution layer of 3x 3. Approximate details are often used in wavelet analysis, where the approximation represents the high-scale, i.e. low-frequency, information of the signal, and the detail represents the high-scale, i.e. high-frequency, information of the signal, and for noise-containing signals, the main energy of the noise components is concentrated in the detail components of the wavelet decomposition.
S43, following the above operations, the feature map is sequentially down-sampled by 2x,4x, and 8x to obtain a feature map of a corresponding size.
And S5, respectively importing feature maps with different sizes into the global branch and the local branch for learning.
And S6, using a GAM module in the global branch to carry out global learning to obtain global information, and using two layers of ResBlock in the local branch to obtain local information.
Wherein, the step S6 includes steps S61-S63:
s61, here, the maximum pixel value and the average pixel value in the size of the feature map are taken as global information.
And S62, taking the characteristic diagram as local information of the characteristic diagram through two ResNet residual layers.
And S63, splicing the global information obtained in S61 and S62, and obtaining the same as the input feature map by using one convolution layer.
And S7, fusing the global branch and the local branch under different sizes to obtain fused feature maps under different sizes.
S8, up-sampling the fusion characteristic diagram by IDWT, and continuously fusing with the fusion characteristic diagram of the lower layer to obtain an output characteristic diagram with the same size as the original low-exposure picture as the final output high-quality picture
Wherein, the step S8 includes steps S81-S83:
s81, after obtaining the fused feature maps with different sizes, the top fused feature map is subjected to IDWT conversion from top to bottom, then spliced with the feature map of the next layer in the channel direction, and then fused with the convolution layer of 3x3 and the activation layer.
And S82, finally obtaining the feature map with the same size and channel number as the main feature map through continuous IDWT transformation and splicing fusion.
And S83, in order to fully acquire local features of the main feature map, splicing the feature map obtained in the step S82 with the main feature map, and fusing two layers of ResBlock (residual error layer) to obtain a final result feature map.
And S9, taking a night picture under high exposure as a reference, introducing MS-SSIM and L1loss as an objective function, and performing iterative learning by using an Adam optimizer to optimize network parameters.
And S10, finally obtaining an image recovery model capable of processing the nighttime low-illumination picture through 200 times of iteration process.
Referring to fig. 6 and 7, fig. 6(a) and 7(a) are input night low-light photographs, fig. 6(b) and 7(b) are high-quality pictures output after model restoration, and it can be clearly seen from fig. 6 and 7 that the picture restoration effect is good.
In summary, the image recovery method of the present embodiment has the following beneficial effects compared with the prior art:
(1) the nighttime low-illumination image recovery method provided by this embodiment adopts a novel SGN self-directed network structure, which not only accelerates the running speed of the model, but also reduces the CPU memory occupied by the conventional U-net structure.
(2) The nighttime low-illumination image restoration method provided by this embodiment refers to discrete wavelet transform DWT and inverse discrete wavelet transform IDWT to perform down-sampling and up-sampling of the feature map, aiming at the problems that nighttime low-illumination image noise is obvious and noise is further expanded due to deepening of U-net structural hierarchy. Better noise processing is carried out on the low-illumination image at night, and the problem of contradiction between brightness detail increase and noise increase is better solved.
(3) According to the nighttime low-illumination image restoration method, aiming at the situation that local details and global structural features of a nighttime low-illumination image are difficult to capture simultaneously, a mode of capturing global information by using GMANT and capturing local information by using two layers of ResNet is adopted for feature extraction, and the problem of contradiction between the global structural information and the local structural details of the nighttime low-illumination image is well solved.
The present embodiment also provides an image restoration system, including:
the data acquisition module is used for acquiring a low-illumination image pair which comprises a low-exposure image and a high-exposure image, and selecting a training set according to the low-illumination image pair;
the model training module is used for training an image recovery model by adopting the training set;
the image recovery module is used for acquiring a low-illumination image needing to be recovered, inputting the low-illumination image into the trained image recovery model, and outputting the recovered low-illumination image;
wherein the image restoration model performs the following operations on the low-illumination picture:
sequentially carrying out downsampling on the low-illumination pictures for three times to obtain a first feature map, a second feature map and a third feature map under different sizes;
extracting features of the first feature map to obtain a first global feature and a first local feature, and fusing the first global feature and the first local feature to obtain a first fused feature map;
extracting features of the second feature graph to obtain a second global feature and a second local feature, and fusing the second global feature, the second local feature and the first fusion feature graph to obtain a second fusion feature graph;
extracting features of the third feature map to obtain a third global feature and a third local feature, and fusing the third global feature, the third local feature and the second fusion feature map to obtain a third fusion feature map;
and fusing the low-illumination image to be recovered and the third fusion characteristic image, and outputting the recovered image.
The image restoration system of the embodiment can execute the image restoration method provided by the embodiment of the method of the invention, can execute any combination of the implementation steps of the embodiment of the method, and has corresponding functions and beneficial effects of the method.
The present embodiment also provides an image restoration apparatus including:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method described above.
The image restoration device of the embodiment can execute the image restoration method provided by the method embodiment of the invention, can execute any combination of the implementation steps of the method embodiment, and has corresponding functions and beneficial effects of the method.
The embodiment also provides a storage medium, which stores instructions or programs capable of executing the image restoration method provided by the embodiment of the method of the invention, and when the instructions or the programs are executed, the steps can be implemented in any combination of the embodiment of the method, and the corresponding functions and advantages of the method are achieved.
In alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flow charts of the present invention are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are performed independently.
Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the described functions and/or features may be integrated in a single physical device and/or software module, or one or more functions and/or features may be implemented in a separate physical device or software module. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary for an understanding of the present invention. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be understood within the ordinary skill of an engineer, given the nature, function, and internal relationship of the modules. Accordingly, those skilled in the art can, using ordinary skill, practice the invention as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative of and not intended to limit the scope of the invention, which is defined by the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The logic and/or steps represented in the flowcharts or otherwise described herein, e.g., an ordered listing of executable instructions that can be considered to implement logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, via for instance optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner if necessary, and then stored in a computer memory.
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the foregoing description of the specification, reference to the description of "one embodiment/example," "another embodiment/example," or "certain embodiments/examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that: various changes, modifications, substitutions and alterations can be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (8)
1. An image restoration method, characterized by comprising the steps of:
acquiring a low-illumination image pair, and selecting a training set according to the low-illumination image pair;
training an image recovery model by adopting the training set;
acquiring a low-illumination image needing to be restored, inputting the low-illumination image into the trained image restoration model, and outputting the restored low-illumination image;
wherein the image restoration model performs the following operations on the low-illumination picture:
processing the low-illumination picture by a convolution layer and an activation layer to obtain a main feature map with the channel number of 32, and sequentially performing down-sampling for three times to obtain a first feature map, a second feature map and a third feature map under different sizes;
extracting features of the first feature map to obtain a first global feature map and a first local feature map, and fusing the first global feature map and the first local feature map to obtain a first fused feature map;
extracting features of the second feature map to obtain a second global feature map and a second local feature map, and fusing the second global feature map, the second local feature map and the first fusion feature map to obtain a second fusion feature map;
extracting features of the third feature map to obtain a third global feature map and a third local feature map, and fusing the third global feature map, the third local feature map and the second fusion feature map to obtain a third fusion feature map;
and fusing the main feature graph and the third fused feature graph, and outputting a recovered picture.
2. The image restoration method according to claim 1, wherein downsampling the main feature map three times to obtain a first feature map, a second feature map and a third feature map at different sizes comprises:
performing discrete wavelet transformation on the main feature graph with the channel number of 32 to obtain a feature graph with the channel number of 32 × 4 — 128, performing processing on the feature graph by a convolution layer and an activation layer, and halving the channel number to obtain a first feature graph with the channel number of 64;
performing discrete wavelet transformation on the first feature map to obtain a feature map with the channel number of 64 × 4-256, performing processing on the feature map by a convolution layer and an activation layer, and halving the channel number to obtain a second feature map with the channel number of 128;
and performing discrete wavelet transformation on the second feature map to obtain a feature map with the channel number of 128 × 4-512, performing processing on the feature map by one convolution layer and an activation layer, and halving the channel number to obtain a third feature map with the channel number of 256.
3. An image restoration method according to claim 1, characterized in that the feature extraction is performed on the feature map by adopting the following method:
acquiring a maximum pixel value and an average pixel value of the feature map through a global pooling layer and an average pooling layer, averaging the average pixel value and the global pixel value to be used as a standard pixel value of the feature map, performing fine tuning through a convolution layer and an activation layer to be used as an adjustment pixel value of the feature map, performing addition fusion on the adjustment pixel value and the feature map, and performing processing through a convolution layer and an activation layer to be used as a global feature of the feature map;
processing the feature map by two layers of residual error layers to obtain local information of the feature map, and adding and fusing the obtained local information and the feature map to obtain local features of the feature map;
splicing and fusing the global features and the local features to obtain a fused feature map;
and processing the fused feature map by inverse discrete wavelet transform to obtain a feature map with the channel number of 1/4 of the fused feature map, and converting the feature map into a feature map with the channel number of 1/2 of the fused feature map by a convolutional layer and an activation layer for the next fusion operation.
4. The image restoration method according to claim 1, wherein the fusing the main feature map and the third fused feature map and outputting a restored picture comprises:
and splicing the main characteristic diagram and the third fusion characteristic diagram, then carrying out residual layer fusion treatment, and finally obtaining a result characteristic diagram through a convolution layer and an activation layer to be used as an output recovered picture.
5. An image restoration method according to claim 1, further comprising the step of training an image restoration model:
inputting the low exposure image into the image recovery model, and outputting a characteristic image;
and constructing a multi-scale structure similarity loss function and a mean absolute error loss function according to the high exposure image and the feature image output by the model, and performing parameter optimization on the image recovery model.
6. An image restoration system, comprising:
the data acquisition module is used for acquiring a low-illumination image pair and selecting a training set according to the low-illumination image pair;
the model training module is used for training an image recovery model by adopting the training set;
the image recovery module is used for acquiring a low-illumination image needing to be recovered, inputting the low-illumination image into the trained image recovery model, and outputting the recovered low-illumination image;
wherein the image restoration model performs the following operations on the low-illumination picture:
processing the low-illumination picture by a convolution layer and an activation layer to obtain a main feature map with the channel number of 32, and sequentially performing down-sampling for three times to obtain a first feature map, a second feature map and a third feature map under different sizes;
extracting features of the first feature map to obtain a first global feature map and a first local feature map, and fusing the first global feature map and the first local feature map to obtain a first fused feature map;
extracting features of the second feature map to obtain a second global feature map and a second local feature map, and fusing the second global feature map, the second local feature map and the first fusion feature map to obtain a second fusion feature map;
extracting features of the third feature map to obtain a third global feature map and a third local feature map, and fusing the third global feature map, the third local feature map and the second fusion feature map to obtain a third fusion feature map;
and fusing the main feature graph and the third fused feature graph, and outputting a recovered picture.
7. An image restoration apparatus, comprising:
at least one processor;
at least one memory for storing at least one program;
when executed by the at least one processor, cause the at least one processor to implement the method of any one of claims 1-5.
8. A computer-readable storage medium, in which a program executable by a processor is stored, wherein the program executable by the processor is adapted to perform the method according to any one of claims 1 to 5 when executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111492256.5A CN114359069B (en) | 2021-12-08 | Image recovery method, system, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111492256.5A CN114359069B (en) | 2021-12-08 | Image recovery method, system, device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114359069A true CN114359069A (en) | 2022-04-15 |
| CN114359069B CN114359069B (en) | 2024-07-02 |
Family
ID=
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108629753A (en) * | 2018-05-22 | 2018-10-09 | 广州洪森科技有限公司 | A kind of face image restoration method and device based on Recognition with Recurrent Neural Network |
| CN112001863A (en) * | 2020-08-28 | 2020-11-27 | 太原科技大学 | Under-exposure image recovery method based on deep learning |
| CN112634276A (en) * | 2020-12-08 | 2021-04-09 | 西安理工大学 | Lightweight semantic segmentation method based on multi-scale visual feature extraction |
| CN113628123A (en) * | 2020-05-09 | 2021-11-09 | 深圳市中兴微电子技术有限公司 | Training method and device of image recovery model, electronic equipment and readable medium |
| CN113689517A (en) * | 2021-09-08 | 2021-11-23 | 云南大学 | Image texture synthesis method and system of multi-scale channel attention network |
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108629753A (en) * | 2018-05-22 | 2018-10-09 | 广州洪森科技有限公司 | A kind of face image restoration method and device based on Recognition with Recurrent Neural Network |
| CN113628123A (en) * | 2020-05-09 | 2021-11-09 | 深圳市中兴微电子技术有限公司 | Training method and device of image recovery model, electronic equipment and readable medium |
| CN112001863A (en) * | 2020-08-28 | 2020-11-27 | 太原科技大学 | Under-exposure image recovery method based on deep learning |
| CN112634276A (en) * | 2020-12-08 | 2021-04-09 | 西安理工大学 | Lightweight semantic segmentation method based on multi-scale visual feature extraction |
| CN113689517A (en) * | 2021-09-08 | 2021-11-23 | 云南大学 | Image texture synthesis method and system of multi-scale channel attention network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | Real-world single image super-resolution: A brief review | |
| Zou et al. | Sdwnet: A straight dilated network with wavelet transformation for image deblurring | |
| Yang et al. | Scale-free single image deraining via visibility-enhanced recurrent wavelet learning | |
| CN111062892B (en) | Single image rain removing method based on composite residual error network and deep supervision | |
| CN111402146B (en) | Image processing method and image processing apparatus | |
| CN111091503A (en) | Image out-of-focus blur removing method based on deep learning | |
| CN105678728A (en) | High-efficiency super-resolution imaging device and method with regional management | |
| CN111951195A (en) | Image enhancement method and device | |
| CN112164011A (en) | Motion image deblurring method based on self-adaptive residual error and recursive cross attention | |
| CN114936605A (en) | Knowledge distillation-based neural network training method, device and storage medium | |
| CN111754531A (en) | Image instance segmentation method and device | |
| CN114219722A (en) | Low-illumination image enhancement method by utilizing time-frequency domain hierarchical processing | |
| CN116152120A (en) | Low-light image enhancement method and device integrating high-low frequency characteristic information | |
| CN114663309A (en) | Image defogging method and system based on multi-scale information selection attention mechanism | |
| CN112163994A (en) | Multi-scale medical image fusion method based on convolutional neural network | |
| CN114359073A (en) | Low-illumination image enhancement method, system, device and medium | |
| CN112330548A (en) | Underwater image restoration method, device, equipment and medium based on multi-scale fusion | |
| CN115965559A (en) | Integrated aerial image enhancement method for forest scene | |
| CN116386023B (en) | High-phase locomotive brand recognition method and system based on space-time diffusion and electronic equipment | |
| CN116128768B (en) | Unsupervised image low-illumination enhancement method with denoising module | |
| CN111598841B (en) | Example significance detection method based on regularized dense connection feature pyramid | |
| CN110400270B (en) | License plate defogging method utilizing image decomposition and multiple correction fusion | |
| CN114359069B (en) | Image recovery method, system, device and storage medium | |
| CN114359069A (en) | Image recovery method, system, device and storage medium | |
| CN115358962B (en) | End-to-end visual odometer method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |