CN113689356A - Image restoration method and device - Google Patents
Image restoration method and device Download PDFInfo
- Publication number
- CN113689356A CN113689356A CN202111072986.XA CN202111072986A CN113689356A CN 113689356 A CN113689356 A CN 113689356A CN 202111072986 A CN202111072986 A CN 202111072986A CN 113689356 A CN113689356 A CN 113689356A
- Authority
- CN
- China
- Prior art keywords
- image
- stage
- processing
- current stage
- input
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 134
- 238000011084 recovery Methods 0.000 claims abstract description 62
- 238000005457 optimization Methods 0.000 claims abstract description 48
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 36
- 230000003044 adaptive effect Effects 0.000 claims abstract description 21
- 230000001629 suppression Effects 0.000 claims abstract description 12
- 230000000007 visual effect Effects 0.000 claims description 21
- 238000000605 extraction Methods 0.000 claims description 16
- 230000008439 repair process Effects 0.000 claims description 13
- 230000005284 excitation Effects 0.000 claims description 11
- 238000012549 training Methods 0.000 abstract description 22
- 230000000694 effects Effects 0.000 abstract description 8
- 230000005764 inhibitory process Effects 0.000 abstract description 5
- 238000010586 diagram Methods 0.000 description 16
- 238000004364 calculation method Methods 0.000 description 11
- 230000008569 process Effects 0.000 description 10
- 238000011176 pooling Methods 0.000 description 8
- 238000012360 testing method Methods 0.000 description 5
- 238000013461 design Methods 0.000 description 4
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 230000002411 adverse Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 238000012113 quantitative test Methods 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 238000005067 remediation Methods 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Processing (AREA)
Abstract
The application discloses a method for repairing an image, which comprises the following steps: and extracting the characteristics of the image to be repaired, inputting the extracted image characteristics into a plurality of cascaded convolutional neural networks, and recovering the mixed distorted image. In the processing of each convolutional neural network, selecting strategies according to the input image characteristics and the joint training results of a plurality of cascaded convolutional neural networks, matching the attention weights of the selected strategies, and performing adaptive optimization; generating a residual error recovery image of the current stage according to the output characteristics obtained by self-adaptive optimization, and generating a staged recovery image of the current stage by using the residual error recovery image; processing the staged recovered image to obtain a low information characteristic inhibition mask; and obtaining the output characteristics of the current stage by using the low information characteristic suppression mask and the self-adaptive optimization characteristic processing. The high-efficiency and automatic restoration can be realized through the photos with mixed distortion, and the restoration effect can be effectively improved.
Description
Technical Field
The present application relates to image processing technologies, and in particular, to a method and an apparatus for image restoration.
Background
The method is characterized in that a common user uses a smart phone to take a photo, the photo is limited by the limitation of a smart phone camera and the influence of various adverse environmental factors, the photo generally contains combined distortion of unknown mixing ratio and intensity, such as low illumination, high light interference, shadow, Gaussian noise and the like, so that the quality of the photo is reduced, and the effective repair of the photo with the mixed distortion can only depend on professional technicians to carry out combined manual correction by using various professional software.
Specifically, for the combined distortion of unknown mixing ratio and intensity contained in the photo taken by the general user using the smart phone, for example, the photo contains the whole underexposure/overexposure, the subject object to be taken is covered by shadow, moire, gaussian noise, etc., it can only be repaired by manual means, which requires the professional technician to perform complicated technical processing with professional equipment and software, so as to ensure that the combined distortion affecting the quality of the photo can be correctly eliminated, and the high-quality photo can be really restored (as shown in fig. 1). For the automatic repair mode, training and repairing of a convolutional neural network which is popular in recent years is a mainstream method, but most of the existing Convolutional Neural Networks (CNN) can only carry out targeted design training and repairing on a single environmental negative factor and achieve good recovery precision, such as a recovery network for singly removing rainwater in a picture, a recovery network for singly removing shadow in a picture, and the like. The single repair network described above does not allow correct repair of photographs containing combined distortions of unknown mix ratios and intensities.
Recently, there have been studies that propose to combine the above-mentioned single repair tools to form a repair tool chain to repair the photos of mixed distortions, and this solution has proved to be effective, but it is inefficient and the accuracy of the final result is relatively limited, since it requires serial use of multiple CNN mini-networks, each of which requires pre-training, for a specific degree/type of distortion. Based on this, the current effective repair of mixed distortion photos still depends on the complex joint manual processing of professional technicians with professional equipment and various software. The technical requirements of the method are high, and the efficiency is low.
Disclosure of Invention
The application provides an image restoration method and device, which can realize efficient and automatic restoration of a mixed distorted picture and effectively improve the restoration effect.
In order to achieve the purpose, the following technical scheme is adopted in the application:
a method of image inpainting, comprising:
determining the number N of image repairing stages; n is a positive integer;
performing feature extraction on an image to be repaired, taking the extracted image features as image features input in a first stage, sequentially performing convolutional neural network processing in each stage to obtain output features of the corresponding stage, inputting the output features into a next stage until the output features of the stage are obtained in the last stage, and outputting the output features of the last stage to be processed to be used as an image repairing result;
wherein the processing of the convolutional neural network at each stage comprises:
selecting a plurality of strategies in a preset selectable strategy set in the current stage according to the image characteristics input in the current stage and the convolutional neural network parameters trained by combining N stages in advance, and calculating the attention weight corresponding to each selected strategy;
processing the image characteristics input in the current stage by using each selected strategy, multiplying the processing result by the attention weight corresponding to each strategy, cascading all the product results, and adding the processed result to the image characteristics input in the current stage after 1-by-1 convolutional layer processing to obtain self-adaptive optimization characteristics;
generating a residual error recovery image of the current stage according to the self-adaptive optimization characteristics, and generating a staged recovery image of the current stage by using the residual error recovery image;
performing 1-1 convolution processing on the staged recovered image, and then obtaining a low-information feature suppression mask through Sigmoid excitation; multiplying the low information feature suppression mask by a result obtained by performing 1 × C convolution processing on the self-adaptive optimization feature, and adding the product result to the staged recovery image to obtain an output feature of the current stage as an input image feature of the next stage.
Preferably, the determining the number of stages N of image restoration is performed according to the processing capability of the restoration device and/or a user instruction.
Preferably, the extracting the features of the image to be restored includes: sequentially carrying out four residual processing with the same structure on the image to be repaired, and taking the output characteristic of the last residual processing as the extracted image characteristic;
wherein the residual processing comprises: and sequentially performing convolution processing of 3 x 3 times on the input features, adding the processing result and the input features, applying Relu excitation on the addition result, and performing the next residual processing as the output features.
Preferably, the set of selectable policies is determined based on processing power of the repair device and/or user instructions.
Preferably, the calculating the attention weight corresponding to each selected policy includes:
calculating the average characteristic value of the channels in each channel according to the input characteristics of each channel in the image characteristics input at the current stage;
and calculating attention weight values corresponding to the selected strategies by using the channel average characteristic values of all the channels.
Preferably, the generating the residual error recovery image of the current stage according to the adaptive optimization feature comprises:
and performing convolution processing of 1 × 3 × 12 on the self-adaptive optimization features to generate a residual error recovery image of the current stage.
Preferably, the generating a staged recovered image of the current stage by using the residual recovered image includes: and adding the residual error recovery image and the image to be repaired to obtain the staged recovery image of the current stage.
An apparatus for image restoration, comprising: the device comprises a stage number determining module, a feature extracting module, a plurality of cascaded same stage processing modules and an output module; wherein the stage processing module comprises an adaptive optimization unit and a visually supervised stage attention unit;
the stage number determining module is used for determining the stage number N of image restoration;
the characteristic extraction module is used for extracting the characteristics of the image to be repaired and inputting the extracted image characteristics into the first stage processing module;
the adaptive optimization unit in the stage processing module is used for selecting a plurality of strategies in a preset selectable strategy set in the current stage according to the image characteristics input in the current stage and the convolutional neural network parameters trained by combining N stages in advance and calculating the attention weight corresponding to each selected strategy; processing the image characteristics input in the current stage by using each selected strategy, multiplying the processing result by the attention weight corresponding to each strategy, cascading all the product results, processing by a 1-by-1 convolution layer, and adding the processed result to the image characteristics input in the current stage to obtain a visual supervision stage attention unit of which the self-adaptive optimization characteristics are input into the processing module in the current stage;
the visual supervision stage attention unit in the stage processing module is used for generating a residual error recovery image of the current stage according to the self-adaptive optimization characteristics and generating a staged recovery image of the current stage by using the residual error recovery image; performing 1-1 convolution processing on the staged recovered image, and then obtaining a low-information feature suppression mask through Sigmoid excitation; the low information feature suppression mask is multiplied by a result obtained after the adaptive optimization feature is subjected to 1 × C convolution processing, and the product result is added to the staged recovery image to obtain an output feature which is input into a stage processing module of the next stage;
in the stage processing module of the last stage, the visual supervision stage attention unit inputs the output characteristics of the current stage into the output module after acquiring the output characteristics of the current stage, and stops processing;
and the output module is used for outputting the received output characteristics of the last stage as an image recovery result and outputting the result to a user.
According to the technical scheme, the image to be restored is subjected to feature extraction, and then the extracted image features are input into the plurality of cascaded convolutional neural networks to restore the mixed distorted image. In the processing of each convolutional neural network, selecting strategies according to the input image characteristics and the joint training results of a plurality of cascaded convolutional neural networks, matching the attention weights of the selected strategies, and performing adaptive optimization; generating a residual error recovery image of the current stage according to the output characteristics obtained by self-adaptive optimization, and generating a staged recovery image of the current stage by using the residual error recovery image; performing 1-1 convolution processing on the staged recovered image, and then obtaining a low-information characteristic inhibition mask through Sigmoid excitation; and multiplying the result obtained by performing 1 × C convolution processing on the self-adaptive optimization feature by using the low-information feature suppression mask, and adding the product result and the staged recovery image to obtain the output feature of the current stage. Through the processing of the low-information characteristic inhibition mask, the output characteristics in the current stage are ensured to shield the low-information characteristics, and the output characteristics are ensured to include the high-information characteristics so as to enter the processing of the next convolutional neural network, so that the stage stacking number required for achieving the optimal recovery effect is effectively reduced, and the lightweight goal of the whole system is realized.
Drawings
FIG. 1 is a diagram illustrating a conventional method for repairing a mixed distortion image;
FIG. 2 is a block diagram of an image restoration apparatus according to the present application;
FIG. 3a is a schematic diagram of a feature extraction module;
FIG. 3b is a schematic diagram of the internal structure of each residual block;
FIG. 4 is a schematic diagram of an alternative strategy set consisting of 11 convolutional layers of different specifications in parallel;
FIG. 5 is a block diagram of the structure and flow of a phase attention unit of visual surveillance;
fig. 6 is a processing flow of an image restoration method in a certain scene.
Detailed Description
For the purpose of making the objects, technical means and advantages of the present application more apparent, the present application will be described in further detail with reference to the accompanying drawings.
The basic idea of the application is that: combining a plurality of convolutional neural networks for image recovery, and when the parameters of the convolutional neural networks are trained, combining the plurality of convolutional neural networks for training instead of independently training each convolutional neural network; meanwhile, in each neural network processing, it is attempted to reduce the amount of data processed and to ensure the image restoration quality.
Fig. 2 is a block diagram of the image restoration apparatus according to the present application. Based on this basic result, the most basic image restoration method in the present application includes:
determining the number of stages N of image restoration, which can be completed by the stage number determining module of fig. 2, wherein N can be a positive integer;
and extracting the features of the image to be repaired, taking the extracted image features as the image features input in the first stage, sequentially processing the convolutional neural network in each stage to obtain the output features of the corresponding stage, inputting the output features of the corresponding stage into the next stage until the output features of the stage are obtained in the last stage, and outputting the output features of the last stage to be processed to be used as the image repairing result.
The processing of performing the convolutional neural network at each stage may be completed in the stage processing module of fig. 2, and the specific processing may include:
the adaptive optimization unit of each stage processing module selects a plurality of strategies in a preset selectable strategy set in the current stage and calculates attention weights corresponding to the selected strategies according to the image characteristics input in the current stage and the convolutional neural network parameters trained by combining N stages in advance; processing the image characteristics input in the current stage by using each selected strategy, multiplying the processing result by the attention weight corresponding to each strategy, cascading all the product results, and adding the processed result to the image characteristics input in the current stage after 1-by-1 convolutional layer processing to obtain self-adaptive optimization characteristics;
the visual supervision stage attention unit can generate a residual error recovery image of the current stage according to the self-adaptive optimization characteristics, and the residual error recovery image is used for regenerating a staged recovery image of the current stage; performing 1-1 convolution processing on the staged recovered image, and then obtaining a low-information characteristic inhibition mask through Sigmoid excitation; and multiplying the low information feature suppression mask by a result obtained by performing 1 × C convolution processing on the self-adaptive optimization feature, and adding the product result to the staged recovery image to obtain the output feature of the current stage as the input image feature of the next stage. For the processing of the last stage, after the output characteristics of the current stage are acquired, the stage attention unit of the visual supervision of the stage outputs the output characteristics as the image repairing result after the output processing.
The foregoing is the most basic image restoration method and apparatus of the present application. The structure and processing of each part are described in detail in the embodiments below.
First, stage number determining module
The determination of the number of stages N may be made according to the processing capability of the picture restoration device and/or a user instruction. For example, when the user can directly select the number of stages by an instruction, or the hardware detection can be performed by the system, and the processing capability of the device is determined according to the detection result, so that the number of stages N matching the processing capability is selected.
The above is processing of the stage number determination module in the image recovery process, and the specific stage number is directly set for controlling how many stages are subsequently subjected to the joint training without changing the module in the multi-stage convolutional neural network joint training process.
Second, feature extraction processing in the feature extraction module
The feature extraction can adopt the existing processing mode. Unlike the complex multi-layer feature extraction network adopted by the general image processing scheme, in the embodiment, in consideration of the requirement of lightweight system processing, preferably, 4 layers of identical simple residual blocks can be adopted to be connected in series to form a lightweight feature extraction module, as shown in fig. 3a, for acquiring the feature map of the mixed distorted input image. Each residual block has the same internal structure, and as shown in fig. 3b, the input features of the residual units are sequentially convolved twice by 3, the processing result is added to the input features, and the addition result is processed by the Relu excitation function and then input as the output feature to the next residual unit for the next residual processing.
The above is the processing of the feature extraction module in the image recovery process, and the same is also applied to the processing in the multi-stage convolutional neural network joint training process.
Through the processing, the complexity of feature extraction is reduced, and the aim of light weight of the system can be better achieved.
Three, adaptive optimization unit in stage processing module
The unit is used for carrying out self-adaptive analysis on input characteristics and specially selecting and combining recovery strategies according to the training result of the convolutional neural network. The processing of this unit mainly includes: the strategies used at this stage are selected in the selectable set of strategies, and attention weights for each strategy are calculated, which will be described separately below.
1) Policy selection
Each stage provides an optional strategy set, the system selects the strategy used at this time according to the input characteristics in the optional strategy set, and the selected strategy can be one or a plurality of strategies.
The optional set of policies is actually a series of optional classical convolutional or pooling layers, as shown in FIG. 4. Tests show that the design of the selectable strategy set and the weight distribution of the adaptive attention module can realize targeted optimization for different types of distortion input, and can classify strategy trends for different types of distortion.
Fig. 4 shows an alternative strategy set consisting of 11 convolutional layers of different specifications in parallel, respectively: 4 convolutional layers with size of 1 x 1,3 x 3,5 x 5, 7 x 7, 3 void convolutional layers with size of 3 x 3,5 x 5, 7 x 7, an expansion rate of 2, an average pooling layer and a maximum pooling layer with size of 3 x 3, and an average pooling layer and a maximum pooling layer with size of 7 x 7.
The policies in the optional policy set of the present application are not limited to the configuration shown in fig. 4, and some policies may be added or deleted flexibly according to the performance of the target operating device. For example, in order to operate on a device with limited performance, only a part of strategies of 1 × 1,3 × 3,5 × 5 may be used for networking training, and a good recovery effect is still maintained after testing. Based on this, the set of alternative policies may be determined from the processing power of the remediation device and/or user instructions.
In more detail, through the pre-training of the convolutional neural network, for different input image features, corresponding selection strategies and parameter values used in calculating attention weight values of the strategies are obtained through training. Further, in the present application, since training is performed in association with a plurality of stages (i.e., processing of a plurality of convolutional neural networks), it is possible to obtain parameter values for selecting a strategy and calculating a weight corresponding to an input image feature in correspondence with different stages. Based on the above, when selecting the strategies, firstly, the current stage is determined, then, according to the input image characteristics and the convolutional neural network parameters obtained by training in the stage, a plurality of strategies are selected, the input image characteristics are processed by utilizing the selected strategies, and the attention weights of the strategies are correspondingly calculated.
2) Calculating attention weights for strategies
As described above, according to the joint training of the convolutional neural network, for each stage, the attention weight calculation parameters of each strategy at the stage are obtained through training. Here, the attention weight value of each strategy is calculated by using the attention weight calculation parameter of each strategy according to the training result. The specific attention weight value may be calculated by using an existing calculation method and according to the input features of the current stage.
In order to further reduce the processing complexity and the calculation amount and achieve the goal of system light weight, preferably, when performing the attention weight calculation of the strategy, the calculation is not performed directly based on the input image features, but a global average pooling value of the input image features in each channel is first calculated, and then the calculation of the strategy attention weight is performed based on the average pooling value of each channel, so that the calculation amount is greatly reduced.
The following description will take an example of processing of a certain channel c. Specifically, a channel average feature value (i.e., a global average pooling value of channels) is first calculated from the input image features of each channel:
wherein H, W, C represents the height, width and depth (i.e. channel number) of the input feature map input at this stage, xijcRepresenting the input feature value.
According to the design, the attention weight mapping value corresponding to a certain optional strategy a at this stage is calculated as:
Ms(x)＝W2*Relu(W1*Avgc)
W1and W2The learnable parameters of the partial network, that is, the parameters determined through the pre-training process, correspond to different strategies, the corresponding parameter values may be different, Relu represents the excitation function Relu.
Finally, according to the input x, the specific execution weight of a certain policy a at the stage is as follows:
Vsa＝exp(Msa(x))/∑exp(Ms All(x))
Msaexpressing the attention weight mapping value corresponding to the strategy a, and obtaining the attention weight value V of the strategy a after normalization processingsa. In summary, the attention weight is calculated by calculating the channel average feature value instead of the whole feature map through the input image features, so that the calculation overhead is remarkably reduced, and the lightweight features of the network are ensured.
After the calculation of the attention weight values corresponding to the strategies is completed, all the attention weight values can be combined into a list, the selectable strategies are selected in a centralized mode, then strategy processing is carried out on the input image through each selected strategy, the processing results are multiplied by the attention weight values of the corresponding strategies in the list to obtain weighting processing results of the corresponding strategies, then the weighting processing results of the strategies are added, namely are cascaded together, finally the weighting processing results are output through a 1 x 1 convolution layer, the output and the input of the stage are added to serve as the output characteristic of the self-adaptive optimization unit, the output characteristic is called the self-adaptive optimization characteristic, and the output characteristic is input into the visual supervision stage attention unit of the stage to be supervised.
The above is the processing of the adaptive optimization unit in the image recovery process, and the processing in the multi-stage convolutional neural network joint training process is the same as the above.
Visual supervised phase attention unit in phase processing module
FIG. 5 is a block diagram of the structure and flow of a phase attention unit of visual surveillance. As shown in fig. 5, the output characteristic diagram of the adaptive optimization unit is received, and a residual error recovery image R of the current stage is generated through a convolution layer of 1 × 3iThe image is added to the original input image to be restored (i.e. the input of the feature extraction module) to generate a staged restored image S of the current stagei. Then, the stepwise restored image SiThrough a convolution layer processing of 1 x 1 and then a Sigmoid excitation, a low information feature suppression mask with the size of H x W x C is generated, the mask is used for calibrating an output feature map of a previous unit (namely an adaptive optimization unit), the low information feature of the current stage is suppressed, and only useful and reliable information features are allowed to be transmitted to the next stage for repair. Based on the above, the adaptive optimization features output by the adaptive optimization unit are processed by the convolution layer with 1 × C, and the processing result and the low confidence are obtainedAnd multiplying by an information characteristic inhibition mask, shielding low information characteristics in the self-adaptive optimization characteristics, reserving high information characteristics, adding the high information characteristics and the self-adaptive optimization characteristics to obtain output characteristics of a visual supervision stage attention unit, and using the output characteristics as a stage recovery characteristic diagram. And then, determining a subsequent processing mode according to the current stage.
And if the current stage is not the last stage, inputting the staged recovery feature map of the stage into an adaptive optimization unit in the stage processing module of the next stage. Through the processing, the output characteristics after the self-adaptive optimization can be shielded for low information characteristics, the characteristic quantity entering the next stage of processing is reduced, and the information containing high characteristic quantity entering the next stage of processing is ensured, so that the image recovery performance and effect can be ensured while the calculated quantity is greatly reduced, the number of stage stacks required by the optimal recovery effect is effectively reduced, and the lightweight target of the whole system is better realized.
And if the current stage is the last stage for processing, outputting the stage recovery characteristic diagram of the stage as a final repaired image after the output processing. The specific output process may be to pass the stepwise recovery feature map of the last stage (assumed to be stage N) through a 3 × 3 convolution layer process, and to combine the processing result with the stepwise recovery image S of the previous stage (stage N-1)iAnd adding the images and outputting the images to a user as final recovered images.
The above is the process of the attention unit in the stage of visual supervision in the image recovery process. In addition, in the multi-stage convolutional neural network joint training process, the visual supervised stage attention unit needs to calculate the loss function value by comparing the staged recovery image of the current stage with the recovery image of the real result in each stage of processing, so as to measure the performance of the recovery operation performed in the current stage, and to train and correct various parameters of the convolutional neural network in the current stage.
The calculation of the loss function can be performed in the existing manner. One specific example is given below:
for the staged recovery image SiIntroducing a True-result restored picture G (usually an undistorted original image corresponding to the image to be repaired) as a Ground True, and the specific loss function design is explained as follows:
Loss＝LossCharb+α*Lossedge
here, a mixture of the ratio of the sabonier loss and the edge loss is used as the loss function of the present module, and α is a control parameter for controlling the ratio of the importance of both, and is set to 0.07 in the present example.
Wherein:
Sirepresenting a staged visual restoration image, G being the expected final restoration result for that image, i.e. a group True image, L representing the laplacian, and β representing the adjustment bias, set to 0.0000001 in this example.
Five, output module
And 3 × 3 convolution processing is performed on the received output feature (namely, the staged recovery feature map) of the last stage, and the processing result is added to the staged recovery image of the previous stage to be used as an image recovery result and output to the user.
The foregoing is a specific implementation of the image restoration method and apparatus in the present application. Some specific examples are given below.
Fig. 6 is a processing flow of an image restoration method in a certain scene, which specifically includes the following processing steps:
step S101: the user takes a picture with the smartphone, the picture contains distortion of unknown type and degree, and the user selects the picture as the picture to be repaired.
Step S102: alternatively, the user may manually select solution models with different pre-trained stages, such as a Stage-4 solution model and a Stage-8 solution model, which are mainly based on the performance of the hardware on which the user runs the solution of the present patent. The scheme of Stage 4 is recommended for the devices with limited performance. Alternatively, if the user does not want to manually set, the solution model type can be automatically set through a preset hardware detection recommendation function.
Step S103: after the solution model is selected, the picture to be repaired is used as input and input into a feature extraction module of the network to generate a feature map.
Step S104: after the characteristic diagram is generated, a configurable self-adaptive optimization module-Stage 1 is input, and meanwhile, a photo to be repaired is used as input to be input into a configured Stage attention module for visual supervision of each Stage.
Step S105: and the adaptive optimization module-Stage 1 enters a processing flow, calculates the weight value of each strategy according to the characteristic matching of the input image, matches the output characteristic diagram of the strategy cost reaching Stage according to the weight value, and inputs the output characteristic diagram into the visual supervision Stage attention module equipped in the Stage.
Step S106: and the visual supervision stage attention module generates a residual error repairing image by processing the input characteristic diagram, and the residual error repairing image is added with the previously input original image to generate a visual restoration result of the stage.
Step S107: and the visual supervision Stage attention module simultaneously processes and generates a low-information feature mask in the current Stage, is used for correcting and optimizing an output feature map in the current Stage, and inputs the feature map into a next-Stage adaptive optimization module, namely Stage 2 for processing. Stage 2 repeats the processing logic of Stage 1, and since the input feature maps are different at this time, Stage 2 generates a new round of output results, inputs the results into the Stage attention module of the visualization supervision after the Stage, and generates a new round of visualization processing results and feature optimization masks. And repeating the process according to the setting of a user or a designer until the last input adapts to an optimization module-Stage N.
Step S108: and at the moment, the output characteristic diagram of Stage N is input to an output module to generate a final repair result picture.
Step S109: and returning the final repair result picture to the user for the user to use.
In addition, the image restoration method of the application is also simulated. For the restoration of mixed distortion images, the Div2k test set was used to simulate the scheme of the present application and other existing solutions (DnCNN, VDSR-s and RL-Restore, respectively), and table 1 gives the quantized simulation results. As can be seen from table 1, when a mixed distortion image is repaired, the scheme of the present application has a better repairing effect, and is significantly superior to other existing solutions in terms of peak signal-to-noise ratio and structural similarity under various levels of distortion.
Quantitative test results on the Div2K test set
TABLE 1
Meanwhile, in order to show that the method has an excellent restoration effect on the restoration of a single distorted image, the image restoration method is simulated by adopting a model of raindrop interference, fuzzy distortion, noise interference and compression distortion, and a corresponding simulation result is given in table 2.
Quantification results on a single-distortion image test set
TABLE 2
It should be additionally noted that, compared with the prior art, the system model of the present application has the advantages of light weight, high efficiency, and being capable of being configured on a mobile device with limited performance, and detailed quantitative indicators thereof are shown in table 3.
| Model (model) | DnCNN | VDSR | VDSR-s | RL-Restore | This patent |
| Number of parameters (. times.10)5) | 6.69 | 6.67 | 2.09 | 1.96 | 1.04 |
| Calculated amount (× 10)9) | 2.66 | 2.65 | 0.828 | 0.474 | 0.286 |
TABLE 3
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (8)
1. A method of image inpainting, comprising:
determining the number N of image repairing stages; n is a positive integer;
performing feature extraction on an image to be repaired, taking the extracted image features as image features input in a first stage, sequentially performing convolutional neural network processing in each stage to obtain output features of the corresponding stage, inputting the output features into a next stage until the output features of the stage are obtained in the last stage, and outputting the output features of the last stage to be processed to be used as an image repairing result;
wherein the processing of the convolutional neural network at each stage comprises:
selecting a plurality of strategies in a preset selectable strategy set in the current stage according to the image characteristics input in the current stage and the convolutional neural network parameters trained by combining N stages in advance, and calculating the attention weight corresponding to each selected strategy;
processing the image characteristics input in the current stage by using each selected strategy, multiplying the processing result by the attention weight corresponding to each strategy, cascading all the product results, and adding the processed result to the image characteristics input in the current stage after 1-by-1 convolutional layer processing to obtain self-adaptive optimization characteristics;
generating a residual error recovery image of the current stage according to the self-adaptive optimization characteristics, and generating a staged recovery image of the current stage by using the residual error recovery image;
performing 1-1 convolution processing on the staged recovered image, and then obtaining a low-information feature suppression mask through Sigmoid excitation; multiplying the low information feature suppression mask by a result obtained by performing 1 × C convolution processing on the self-adaptive optimization feature, and adding the product result to the staged recovery image to obtain an output feature of the current stage as an input image feature of the next stage.
2. The method according to claim 1, wherein the determining of the number of stages N of image restoration is performed according to processing capability of a restoration device and/or a user instruction.
3. The method according to claim 1, wherein the feature extraction of the image to be repaired comprises: sequentially carrying out four residual processing with the same structure on the image to be repaired, and taking the output characteristic of the last residual processing as the extracted image characteristic;
wherein the residual processing comprises: and sequentially performing convolution processing of 3 x 3 times on the input features, adding the processing result and the input features, applying Relu excitation on the addition result, and performing the next residual processing as the output features.
4. The method according to claim 1, characterized in that the set of selectable policies is determined from processing capabilities and/or user instructions of the repair device.
5. The method according to claim 1, wherein the calculating the attention weight corresponding to each selected policy comprises:
calculating the average characteristic value of the channels in each channel according to the input characteristics of each channel in the image characteristics input at the current stage;
and calculating attention weight values corresponding to the selected strategies by using the channel average characteristic values of all the channels.
6. The method of claim 1, wherein generating the residual restored image of the current stage according to the adaptive optimization features comprises:
and performing convolution processing of 1 × 3 × 12 on the self-adaptive optimization features to generate a residual error recovery image of the current stage.
7. The method according to claim 1 or 6, wherein the generating a staged recovered image of a current stage using the residual recovered image comprises: and adding the residual error recovery image and the image to be repaired to obtain the staged recovery image of the current stage.
8. An apparatus for image restoration, comprising: the device comprises a stage number determining module, a feature extracting module, a plurality of cascaded same stage processing modules and an output module; wherein the stage processing module comprises an adaptive optimization unit and a visually supervised stage attention unit;
the stage number determining module is used for determining the stage number N of image restoration;
the characteristic extraction module is used for extracting the characteristics of the image to be repaired and inputting the extracted image characteristics into the first stage processing module;
the adaptive optimization unit in the stage processing module is used for selecting a plurality of strategies in a preset selectable strategy set in the current stage according to the image characteristics input in the current stage and the convolutional neural network parameters trained by combining N stages in advance and calculating the attention weight corresponding to each selected strategy; processing the image characteristics input in the current stage by using each selected strategy, multiplying the processing result by the attention weight corresponding to each strategy, cascading all the product results, processing by a 1-by-1 convolution layer, and adding the processed result to the image characteristics input in the current stage to obtain a visual supervision stage attention unit of which the self-adaptive optimization characteristics are input into the processing module in the current stage;
the visual supervision stage attention unit in the stage processing module is used for generating a residual error recovery image of the current stage according to the self-adaptive optimization characteristics and generating a staged recovery image of the current stage by using the residual error recovery image; performing 1-1 convolution processing on the staged recovered image, and then obtaining a low-information feature suppression mask through Sigmoid excitation; the low information feature suppression mask is multiplied by a result obtained after the adaptive optimization feature is subjected to 1 × C convolution processing, and the product result is added to the staged recovery image to obtain an output feature which is input into a stage processing module of the next stage;
in the stage processing module of the last stage, the visual supervision stage attention unit inputs the output characteristics of the current stage into the output module after acquiring the output characteristics of the current stage, and stops processing;
and the output module is used for outputting the received output characteristics of the last stage as an image recovery result and outputting the result to a user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111072986.XA CN113689356B (en) | 2021-09-14 | 2021-09-14 | Image restoration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111072986.XA CN113689356B (en) | 2021-09-14 | 2021-09-14 | Image restoration method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113689356A true CN113689356A (en) | 2021-11-23 |
| CN113689356B CN113689356B (en) | 2023-11-24 |
Family
ID=78586354
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111072986.XA Active CN113689356B (en) | 2021-09-14 | 2021-09-14 | Image restoration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113689356B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023207778A1 (en) * | 2022-04-24 | 2023-11-02 | 腾讯科技（深圳）有限公司 | Data recovery method and device, computer, and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109360171A (en) * | 2018-10-26 | 2019-02-19 | 北京理工大学 | A kind of real-time deblurring method of video image neural network based |
| CN111275638A (en) * | 2020-01-16 | 2020-06-12 | 湖南大学 | Face restoration method for generating confrontation network based on multi-channel attention selection |
| CN111915531A (en) * | 2020-08-06 | 2020-11-10 | 温州大学 | Multi-level feature fusion and attention-guided neural network image defogging method |
| CN112365414A (en) * | 2020-11-04 | 2021-02-12 | 天津大学 | Image defogging method based on double-path residual convolution neural network |
| JP6830707B1 (en) * | 2020-01-23 | 2021-02-17 | 同▲済▼大学 | Person re-identification method that combines random batch mask and multi-scale expression learning |
| WO2021104056A1 (en) * | 2019-11-27 | 2021-06-03 | 中国科学院深圳先进技术研究院 | Automatic tumor segmentation system and method, and electronic device |
| US20210241470A1 (en) * | 2019-04-30 | 2021-08-05 | Beijing Sensetime Technology Development Co., Ltd. | Image processing method and apparatus, electronic device, and storage medium |
-
2021
- 2021-09-14 CN CN202111072986.XA patent/CN113689356B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109360171A (en) * | 2018-10-26 | 2019-02-19 | 北京理工大学 | A kind of real-time deblurring method of video image neural network based |
| US20210241470A1 (en) * | 2019-04-30 | 2021-08-05 | Beijing Sensetime Technology Development Co., Ltd. | Image processing method and apparatus, electronic device, and storage medium |
| WO2021104056A1 (en) * | 2019-11-27 | 2021-06-03 | 中国科学院深圳先进技术研究院 | Automatic tumor segmentation system and method, and electronic device |
| CN111275638A (en) * | 2020-01-16 | 2020-06-12 | 湖南大学 | Face restoration method for generating confrontation network based on multi-channel attention selection |
| JP6830707B1 (en) * | 2020-01-23 | 2021-02-17 | 同▲済▼大学 | Person re-identification method that combines random batch mask and multi-scale expression learning |
| CN111915531A (en) * | 2020-08-06 | 2020-11-10 | 温州大学 | Multi-level feature fusion and attention-guided neural network image defogging method |
| CN112365414A (en) * | 2020-11-04 | 2021-02-12 | 天津大学 | Image defogging method based on double-path residual convolution neural network |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023207778A1 (en) * | 2022-04-24 | 2023-11-02 | 腾讯科技（深圳）有限公司 | Data recovery method and device, computer, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113689356B (en) | 2023-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110163808B (en) | Single-frame high-dynamic imaging method based on convolutional neural network | |
| CN110728633B (en) | Multi-exposure high-dynamic-range inverse tone mapping model construction method and device | |
| Liang et al. | Cameranet: A two-stage framework for effective camera isp learning | |
| CN109064423B (en) | Intelligent image repairing method for generating antagonistic loss based on asymmetric circulation | |
| CN110223251B (en) | Convolution neural network underwater image restoration method suitable for artificial and natural light sources | |
| Jaroensri et al. | Generating training data for denoising real rgb images via camera pipeline simulation | |
| WO2022000397A1 (en) | Low-illumination image enhancement method and apparatus, and computer device | |
| CN111915513B (en) | Image denoising method based on improved adaptive neural network | |
| CN109784358B (en) | No-reference image quality evaluation method integrating artificial features and depth features | |
| CN113450290B (en) | Low-illumination image enhancement method and system based on image inpainting technology | |
| CN107895350B (en) | HDR image generation method based on self-adaptive double gamma transformation | |
| CN113628146A (en) | Image denoising method based on deep convolutional network | |
| CN113689356B (en) | Image restoration method and device | |
| US20130287299A1 (en) | Image processing apparatus | |
| CN106971375B (en) | Image amplification processing method and device | |
| US11443414B2 (en) | Image signal processing | |
| CN117456230A (en) | Data classification method, system and electronic equipment | |
| CN111860287A (en) | Target detection method and device and storage medium | |
| CN116051419A (en) | Image denoising method based on deep learning Relu and channel correlation attention | |
| CN116433518A (en) | Fire image smoke removing method based on improved Cycle-Dehaze neural network | |
| Pavithra et al. | Automatic image quality tuning framework for optimization of ISP parameters based on multi-stage optimization approach | |
| CN112950509B (en) | Image processing method and device and electronic equipment | |
| CN115170811A (en) | Denoising method based on convolutional neural network | |
| Li et al. | A fast image dehazing algorithm for highway tunnel based on artificial multi-exposure image fusion | |
| TWI590192B (en) | Adaptive high dynamic range image fusion algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |