CN116757966A - Image enhancement method and system based on multi-level curvature supervision - Google Patents
Image enhancement method and system based on multi-level curvature supervision Download PDFInfo
- Publication number
- CN116757966A CN116757966A CN202311036758.6A CN202311036758A CN116757966A CN 116757966 A CN116757966 A CN 116757966A CN 202311036758 A CN202311036758 A CN 202311036758A CN 116757966 A CN116757966 A CN 116757966A
- Authority
- CN
- China
- Prior art keywords
- layer
- curvature
- denoising
- convolution
- network model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 49
- 239000010410 layer Substances 0.000 claims abstract description 141
- 238000012545 processing Methods 0.000 claims abstract description 45
- 238000009826 distribution Methods 0.000 claims abstract description 40
- 239000002355 dual-layer Substances 0.000 claims abstract description 18
- 230000008569 process Effects 0.000 claims abstract description 16
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 7
- 230000004927 fusion Effects 0.000 claims description 23
- 238000000605 extraction Methods 0.000 claims description 22
- 230000006870 function Effects 0.000 claims description 22
- 238000013528 artificial neural network Methods 0.000 claims description 16
- 238000013507 mapping Methods 0.000 claims description 13
- 230000004913 activation Effects 0.000 claims description 10
- 238000011156 evaluation Methods 0.000 claims description 10
- 238000010606 normalization Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 6
- 238000007499 fusion processing Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004458 analytical method Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 2
- 238000003860 storage Methods 0.000 claims description 2
- 238000005457 optimization Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 2
- 238000005315 distribution function Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 238000005452 bending Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
- 238000003892 spreading Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/64—Analysis of geometric attributes of convexity or concavity
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Image Processing (AREA)
Abstract
The invention provides an image enhancement method and system based on multi-level curvature supervision, belongs to the technical field of image data processing, and aims at the problems in the image denoising process in the prior art, denoising processing is carried out on image data by constructing a double-layer denoising network model, wherein a first layer in the double-layer denoising network model provides noise level estimation for a second layer so as to remove prior noise distribution. In addition, the second layer in the dual-layer denoising network model adopts an encoder-decoder architecture, and curvature is introduced into the encoder-decoder architecture and the curvature supervision module so as to obtain an efficient and flexible convolutional neural network. Aiming at different blind noise reconstruction models, the invention can obviously improve the image texture and the denoising strength in the directions of image denoising, deblurring, single image superdivision and the like.
Description
Technical Field
The invention relates to the technical field of image data processing, in particular to an image enhancement method and system based on multi-level curvature supervision.
Background
With the rapid development of 5G technology, the requirements of multiple fields such as medicine, weather, monitoring and the like on image quality are higher and higher, and the acquired images are unclean due to the fact that the actual imaging is subjected to external environment, object movement and the like, especially in the process of spreading through compression processing and the like, the pixel characteristics of the images are further lost, so that the method has important practical significance for researching an image denoising model.
In early research of an image denoising technology, the problem of additive noise is solved by constructing a maximum probability estimation model, but the method cannot establish an effective mapping relation under the condition of facing real complex noise, and has strong limitation in solving the problem of blind noise.
In recent years, with the application of deep learning in image processing, an image denoising model based on a convolutional neural network obtains a qualitative leap, but a large progress space still exists in the aspects of feature extraction and module optimization.
Disclosure of Invention
The invention aims to: an image enhancement method and system based on multi-level curvature supervision are provided to solve the above problems in the prior art.
The technical scheme is as follows: in a first aspect, an image enhancement method based on multi-level curvature supervision is provided, the method comprising the steps of:
step 1, constructing a double-layer denoising network model based on a neural network; the first layer of the double-layer denoising network model is used for executing mapping learning, and the second layer is used for executing image denoising processing;
step 2, the double-layer denoising network model receives original image data to be processed, executes convolution operation, and divides a convolution result into four branches to synchronously execute feature extraction operation;
step 3, fusing the feature extraction results of the four branches to obtain fusion features;
step 4, performing noise distribution and noise level learning mapping on the fusion characteristics based on a convolution layer to obtain noise level estimation and noise distribution estimation;
step 5, fusing the original image data with the noise distribution estimation, and calculating the corresponding curvature;
step 6, taking the curvature, noise level estimation and original image data as input data of a second layer of the double-layer denoising network model;
step 7, analyzing and processing the received data by adopting a four-layer neural network at the second layer of the double-layer denoising network model; the four layers of neural networks are embedded with CSAM models in each layer of neural networks through downsampling operation;
and 8, outputting an analysis processing result of the second layer of the double-layer denoising network model to obtain denoised image data.
In some implementations of the first aspect, before the dual-layer denoising network model performs image denoising, further includes: and evaluating the double-layer denoising network model by constructing an evaluation loss function, and optimizing parameters of the double-layer denoising network model according to an evaluation result to obtain the double-layer denoising network model with optimized final performance.
Wherein, the expression corresponding to the estimated loss function is:
in the method, in the process of the invention,representing noise level loss;representing noise distribution loss;representing an estimated loss function;representing the weight ratio of the adjusted noise distribution loss and the noise level loss.
The first layer of the dual-layer denoising network model sequentially comprises: conv convolution layer, four parallel block layers, convolution kernel and Conv layer composed of activation function. In the process of executing data processing, the Conv convolution layer executes convolution operation on the received original image data to be processed, divides a convolution result into four branches, synchronously inputs the four parallel layers of block layers to execute multi-scale feature extraction, and finally inputs the fused processing result into the Conv layer to execute learning mapping of noise distribution and noise level to obtain noise level estimation and noise distribution estimation.
In some implementations of the first aspect, the process of obtaining the curvature includes the steps of:
step 5.1, fusing the original image data with the noise distribution estimation to obtain fused image data;
step 5.2, selecting a local area with a preset size for each pixel in the fused image data to calculate the local geometric information of the current pixel;
step 5.3, calculating gradient information of pixel values in a selected pixel neighborhood window by using a Hessian matrix;
and 5.4, converting the calculated curvature value into a attention weight section of the pixel through a mapping function.
A process for performing data processing by a CSAM model, comprising the steps of:
step 7.1, performing 1×1 convolution on a curvature map obtained according to curvature;
step 7.2, performing sigmoid activation on the convolution result in the step 7.1;
step 7.3, fusing the activation result in the step 7.2 with the fusion characteristic;
step 7.4, generating curvature supervision attention map through convolution of the fusion result in the step 7.3 with the convolution kernel size of 1 multiplied by 1;
step 7.5, performing feature extraction on the original image data, and calculating attention weights of pixels of the original image data based on the curvature map;
and 7.6, judging whether the pixels in the current original image data belong to noise or effective information according to the curvature information of the pixels, and distributing corresponding weights.
In some implementations of the first aspect, the second layer of the two-layer denoising network model uses an encoder-decoder une convolutional neural network as a network architecture, including a ConvNeXt-Block structure. In the process of executing data processing by the ConvNeXt-Block structure, a data processing method combining a reverse bottleneck structure and deep convolution is adopted.
The ConvNeXt-Block structure specifically comprises: layerNorm normalization module, 7×7 large depth convolutional layer, 1×1 convolutional layer, simpleGate module. The process of executing the data processing comprises the following steps:
after passing through a layerNorm normalization module, a 7×7 large-depth convolution layer is utilized to enlarge a receptive field for a received image;
the characteristic dimension is increased by using a 1 multiplied by 1 convolution layer aiming at the data after the receptive field is enlarged, and the dimension is reduced by using the 1 multiplied by 1 convolution layer again after the data passes through a simpleGate module;
carrying out fusion processing on the result processed by the layerNorm normalization module and the image with reduced dimensionality, and sequentially inputting the fusion processing result into the layerNorm normalization module, the 1 multiplied by 1 convolution layer, the simpleGate module and the 1 multiplied by 1 convolution layer; and finally, fusing the fusion processing result with the processing result obtained at present to obtain the processing result of the ConvNeXt-Block structure, and outputting the processing result.
In a second aspect, an image enhancement system based on multi-level curvature supervision is provided, for implementing an image enhancement method, the system comprising the following modules:
the model construction module is used for constructing a double-layer denoising network model based on the neural network;
the feature extraction module is used for receiving the original image data and extracting features through parallel branch processing;
fusing the branch processing results to obtain a feature fusion module of fusion features;
a noise estimation module for performing noise level estimation and noise distribution estimation according to the obtained fusion characteristics;
a curvature calculation module for calculating a corresponding curvature according to the obtained noise distribution estimation and the original image data;
an image denoising module for performing image denoising processing by adopting an encoder-decoder structure according to curvature, noise level estimation and original image data;
and the image output module is used for outputting the denoised image data.
In a third aspect, an image enhancement device based on multi-level curvature supervision is presented, the device comprising: a processor and a memory storing computer program instructions, wherein the processor reads and executes the computer program instructions to implement an image enhancement method based on multi-level curvature supervision.
In a fourth aspect, a computer readable storage medium having computer program instructions stored thereon, which when executed by a processor, implement an image enhancement method based on multi-level curvature supervision is presented.
The beneficial effects are that: the invention provides an image enhancement method and system based on multi-level curvature supervision, which execute denoising processing on image data by constructing a double-layer denoising network model, wherein a first layer in the double-layer denoising network model provides noise level estimation for a second layer so as to perform priori noise distribution removal. In addition, the second layer in the dual-layer denoising network model adopts an encoder-decoder architecture, and curvature is introduced into the encoder-decoder architecture and the curvature supervision module so as to obtain an efficient and flexible convolutional neural network. Aiming at different blind noise reconstruction models, the invention can obviously improve the image texture and the denoising strength in the directions of image denoising, deblurring, single image superdivision and the like.
Drawings
FIG. 1 is a flow chart of data processing according to the present invention.
FIG. 2 is a Block diagram of the improved ConvNeXt-Block architecture of the present invention.
Fig. 3 is a block diagram of a CSAM curvature monitor attention module.
Detailed Description
In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the invention may be practiced without one or more of these details. In other instances, well-known features have not been described in detail in order to avoid obscuring the invention.
The existing stage has a certain theoretical and practical basis for the known noise model, and a better prediction model exists; on the other hand, when the unknown noise is faced, the denoising effect is greatly degraded, and based on the fact that the method disclosed by the invention is used for solving the problem by using a two-time end-to-end algorithm model. Firstly, estimating the noise distribution condition, taking the noise distribution condition as the input of a second-layer network, and further converting model training of unknown noise into Gaussian noise removal of known distribution condition.
Example 1
In one embodiment, aiming at the defects existing in the image denoising process in the prior art, an image enhancement method based on multi-level curvature supervision is provided, and denoising processing is carried out on image data by constructing a curvature supervision model, so that the processed image data meets the requirement of balancing denoising performance and texture detail.
Specifically, the image enhancement method based on multi-level curvature supervision is shown in fig. 1, and comprises the following steps:
step 1, constructing a double-layer denoising network model based on a neural network, performing mapping learning on a first layer, and performing image denoising processing on a second layer;
specifically, a first layer of the double-layer denoising network model acquires noise distribution estimation and noise level estimation according to a mapping relation; and a second layer of the double-layer denoising network model receives the data output by the first layer and adopts an encoder-decoder architecture to perform denoising processing.
Step 2, the double-layer denoising network model receives the original image data to be processed, executes convolution operation, and divides the convolution result into four branches to synchronously execute feature extraction operation;
specifically, the first layer of the dual-layer denoising network model sequentially comprises: the convolution kernel is a Conv convolution layer with a size of 3*3, a parallel four-layer block layer, and a Conv layer formed by the convolution kernel and an activation function. The four parallel block layers adopt convolution kernels with different sizes, and the sizes of the convolution kernels are 1*1, 3*3, 5*5 and 7*7 from top to bottom in sequence and are used for multi-scale feature extraction. In order to keep the feature sizes consistent, different channel numbers are set for convolution kernels with different sizes, the channel numbers are 64, 32, 16 and 8 from top to bottom in sequence, and after feature extraction is respectively executed, feature fusion of branches is carried out.
Step 3, fusing the feature extraction results of the four branches to obtain fusion features;
step 4, performing noise distribution and noise level learning mapping on the fusion characteristics based on a convolution layer to obtain noise level estimation and noise distribution estimation;
step 5, fusing the original image data with the noise distribution estimation, and calculating the corresponding curvature;
specifically, the curvature is calculated by measuring the curve or the curved surface bending degree to evaluate the local geometric characteristics of the pixels on the feature map, so that in order to improve the processing capability of the double-layer denoising network model on the noise image edge and the texture, the embodiment fuses the original image data and the noise distribution estimation.
First, for each pixel, selecting a local area with a fixed size to calculate local geometric information of the pixel; then, in the selected pixel neighborhood window, gradient information of pixel values is calculated by using a Hessian matrix, and curvature values are obtained; and finally, converting the calculated curvature value into an attention weight section of the pixel through a mapping function, limiting the curvature value in a preset section, and avoiding that the calculated curvature value section exceeds a preset range.
Step 6, taking the curvature, noise level estimation and original image data as input data of a second layer of the double-layer denoising network model;
specifically, according to the noise distribution estimation obtained by the first layer of the double-layer denoising network model, the second layer of the double-layer denoising network model only needs to denoise the original image data according to the known noise distribution, and the output of the first layer is used as the input of the second layer, namely curvature, noise level and noise image are input into the second layer.
Step 7, analyzing and processing the received data by adopting a four-layer neural network at the second layer of the double-layer denoising network model; the four layers of neural networks are embedded with CSAM models in each layer of neural networks through downsampling operation;
specifically, a 4-layer neural network is selected as a second layer of the double-layer denoising network model, and a CSAM (curvature supervised attention module) denoising model is added into each layer of network through a downsampling module, so that detail textures are reinforced while denoising is realized.
Specifically, the CSAM module mainly fuses the curvature map and the attention mechanism to enhance the weight of the effective features of the image, firstly, carries out 1×1 convolution on the curvature map, then carries out sigmoid activation, finally fuses the curvature map and the feature map obtained in the previous step, and generates curvature supervision attention map through convolution with the convolution kernel size of 1×1.
The CSAM attention mechanism calculates attention weight of image pixels based on a curvature graph after feature extraction, judges whether the pixels belong to noise or effective information according to curvature information of the pixels, and distributes corresponding weight, so that noise in an image can be processed pertinently for a subsequent denoising model, and the quality of a denoising image is improved.
And 8, outputting an analysis processing result of the second layer of the double-layer denoising network model to obtain denoised image data.
In a further embodiment, the first layer in the two-layer denoising network model provides noise level estimation to the second layer for prior noise distribution removal, and the corresponding expression is:
；
in the method, in the process of the invention,representing a noise image;representing a clean image;representing noise level;representing a standard deviation radius ofIs a gaussian noise of (c).
Alternatively, the classical maximum a posteriori framework (MAP) may be determined by maximizing the posterior distribution functionTo estimateAnd since the maximum posterior distribution function is equal to the minimum log likelihood function, an estimate is obtainedThe calculation formula of (2), namely:
；
in the method, in the process of the invention,representation independent of noisy imagesIs a de-noised image of (2)Is a priori of (2); in addition, in the case of the optical fiber,wherein, the liquid crystal display device comprises a liquid crystal display device,the regularization term is represented as a function of the regularization term,is a preset positive integer, thus, estimatesIs given by a priori expression:
；
and providing noise level estimation to the second layer through the first layer in the double-layer denoising network model, and obtaining an priori expression to execute prior noise distribution removal.
Example two
In a further embodiment based on the first embodiment, in order to improve the denoising performance of the dual-layer denoising network model, an evaluation loss function is constructed to evaluate the dual-layer denoising network model, and parameter optimization of the dual-layer denoising network model is performed according to an evaluation result, so that the dual-layer denoising network model with optimized final performance is obtained.
Wherein, the expression corresponding to the estimated loss function is:
；
in the method, in the process of the invention,representing noise level loss;representing noise distribution loss;representing an estimated loss function;representing the weight ratio of the adjusted noise distribution loss and the noise level loss.
In the preferred embodiment, noise distribution lossAdopting a cross entropy loss function; noise level lossAn absolute value error loss function is employed.
In a further embodiment, the synthetic noise image set and the natural noise image set are employed to construct training data and test data for performance optimization of a two-layer denoising network model.
In a further embodiment, the performance of the optimized dual-layer denoising network model is evaluated by using a PSNR evaluation index result.
Example III
In one embodiment, the second layer of the dual-layer denoising network model takes an encoder-decoder Unet convolutional neural network as a network architecture, and comprises a ConvNeXt-Block structure; the structure of ConvNeXt-Block is shown in FIG. 2, the denoising sub-network in the double-layer denoising network takes an encoder-decoder Unet convolutional neural network as a network architecture, and a ConvNeXt module is integrated into UNet so as to improve the capability of feature extraction. The advantages of a reverse bottleneck structure and deep convolution are adopted, the network width is expanded, the condition of losing tiny features is avoided, and the diversity of feature information is enriched.
As shown in fig. 2, unlike the conventional bottleneck structure, the present invention uses the inverse bottleneck structure to expand the receptive field by 7×7 large-depth convolution, and then uses 1×1 convolution to make the feature dimension first and then make the feature dimension second, so as to enrich the feature information to ensure the expressive power of the model. Furthermore, a layerNorm module is used for normalizing the feature graphs of each dimension, so that the stability of feature data distribution is ensured; the convolution kernel size is 7 multiplied by 7, the large-depth convolution and the convolution kernel sizes of 2 are 1 multiplied by 1 are adopted to improve the feature extraction capability, and the input parameter quantity is greatly reduced under the condition of extremely little precision loss; in a preferred embodiment, the activation function selects a GELU, which increases the network computing speed while saving memory resources.
Example IV
In one embodiment, an image enhancement system is presented for implementing an image enhancement method based on multi-level curvature supervision, the system comprising the following modules: the system comprises a model construction module, a feature extraction module, a feature fusion module, a noise estimation module, a curvature calculation module, an image denoising module and an image output module.
The model building module is used for building a double-layer denoising network model based on a neural network; the feature extraction module is used for receiving the original image data and extracting features in a parallel branch processing mode; the feature fusion module is used for fusing the branch processing results to obtain fusion features; the noise estimation module performs noise level estimation and noise distribution estimation according to the obtained fusion characteristics; the curvature calculation module is used for fusing the obtained noise distribution estimation and the original image data to calculate the corresponding curvature; the image denoising module performs image denoising processing by adopting an encoder-decoder structure according to curvature, noise level estimation and original image data; the image output module is used for outputting the denoised image data.
In a further embodiment, a first layer of the dual layer denoising network model is used to perform map learning and a second layer is used to perform image denoising processing.
The first layer of the double-layer denoising network model sequentially comprises: the convolution kernel is a Conv convolution layer with a size of 3*3, a parallel four-layer block layer, and a Conv layer formed by the convolution kernel and an activation function. The four parallel block layers adopt convolution kernels with different sizes, and the sizes of the convolution kernels are 1*1, 3*3, 5*5 and 7*7 from top to bottom in sequence and are used for multi-scale feature extraction. In order to keep the feature sizes consistent, different channel numbers are set for convolution kernels with different sizes, the channel numbers are 64, 32, 16 and 8 from top to bottom in sequence, and after feature extraction is respectively executed, feature fusion of branches is carried out.
The second layer of the two-layer denoising network model adopts an encoder-decoder architecture to perform denoising processing.
In a further embodiment, in order to improve the denoising performance of the dual-layer denoising network model, the system further comprises a performance optimization module, wherein the performance optimization module evaluates the dual-layer denoising network model by constructing an evaluation loss function, and performs parameter optimization of the dual-layer denoising network model according to an evaluation result to obtain the dual-layer denoising network model with optimized final performance.
As described above, although the present invention has been shown and described with reference to certain preferred embodiments, it is not to be construed as limiting the invention itself. Various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. An image enhancement method based on multi-level curvature supervision is characterized by comprising the following steps:
step 1, constructing a double-layer denoising network model based on a neural network; the first layer of the double-layer denoising network model is used for executing mapping learning, and the second layer is used for executing image denoising processing;
step 2, the double-layer denoising network model receives original image data to be processed, executes convolution operation, and divides a convolution result into four branches to synchronously execute feature extraction operation;
step 3, fusing the feature extraction results of the four branches to obtain fusion features;
step 4, performing noise distribution and noise level learning mapping on the fusion characteristics based on a convolution layer to obtain noise level estimation and noise distribution estimation;
step 5, fusing the original image data with the noise distribution estimation, and calculating the corresponding curvature;
step 6, taking the curvature, noise level estimation and original image data as input data of a second layer of the double-layer denoising network model;
step 7, analyzing and processing the received data by adopting a four-layer neural network at the second layer of the double-layer denoising network model; the four layers of neural networks are embedded with CSAM models in each layer of neural networks through downsampling operation;
and 8, outputting an analysis processing result of the second layer of the double-layer denoising network model to obtain denoised image data.
2. The method of claim 1, wherein the dual layer denoising network model further comprises, before performing image denoising:
step 1.1, evaluating the double-layer denoising network model by constructing an evaluation loss function, and optimizing parameters of the double-layer denoising network model according to an evaluation result to obtain a double-layer denoising network model with optimized final performance;
the expression corresponding to the evaluation loss function is as follows:
；
in the method, in the process of the invention,representing noise level loss; />Representing noise distribution loss; />Representing an estimated loss function; />Representing the weight ratio of the adjusted noise distribution loss and the noise level loss.
3. The method of claim 1, wherein the first layer of the two-layer denoising network model comprises, in order: conv convolution layer, parallel four-layer block layer, convolution kernel and Conv layer formed by activation function;
and the Conv convolution layer executes convolution operation on the received original image data to be processed, divides the convolution result into four branches, synchronously inputs the four parallel block layers, executes multi-scale feature extraction, and finally inputs the fused processing result into the Conv layer, executes learning mapping of noise distribution and noise level, and obtains noise level estimation and noise distribution estimation.
4. The image enhancement method based on multi-level curvature supervision according to claim 1, wherein the process of acquiring the curvature comprises the steps of:
step 5.1, fusing the original image data with the noise distribution estimation to obtain fused image data;
step 5.2, selecting a local area with a preset size for each pixel in the fused image data to calculate the local geometric information of the current pixel;
step 5.3, calculating gradient information of pixel values in a selected pixel neighborhood window by using a Hessian matrix;
and 5.4, converting the calculated curvature value into a attention weight section of the pixel through a mapping function.
5. The image enhancement method based on multi-level curvature supervision according to claim 1, wherein the CSAM model performs a data processing procedure, comprising the steps of:
step 7.1, performing 1×1 convolution on a curvature map obtained according to curvature;
step 7.2, performing sigmoid activation on the convolution result in the step 7.1;
step 7.3, fusing the activation result in the step 7.2 with the fusion characteristic;
step 7.4, generating curvature supervision attention map through convolution of the fusion result in the step 7.3 with the convolution kernel size of 1 multiplied by 1;
step 7.5, performing feature extraction on the original image data, and calculating attention weights of pixels of the original image data based on the curvature map;
and 7.6, judging whether the pixels in the current original image data belong to noise or effective information according to the curvature information of the pixels, and distributing corresponding weights.
6. The image enhancement method based on multi-level curvature supervision according to claim 1, wherein the second layer of the dual-layer denoising network model uses an encoder-decoder une convolutional neural network as a network architecture, and comprises a ConvNeXt-Block structure;
in the process of executing data processing by the ConvNeXt-Block structure, a data processing method combining a reverse bottleneck structure and deep convolution is adopted.
7. The image enhancement method based on multi-level curvature supervision according to claim 6, wherein the convnex-Block structure specifically comprises: a layerNorm normalization module, a 7×7 large depth convolutional layer, a 1×1 convolutional layer, and a simpleGate module;
the process of executing the data processing comprises the following steps:
after passing through a layerNorm normalization module, a 7×7 large-depth convolution layer is utilized to enlarge a receptive field for a received image;
the characteristic dimension is increased by using a 1 multiplied by 1 convolution layer aiming at the data after the receptive field is enlarged, and the dimension is reduced by using the 1 multiplied by 1 convolution layer again after the data passes through a simpleGate module;
carrying out fusion processing on the result processed by the layerNorm normalization module and the image with reduced dimensionality, and sequentially inputting the fusion processing result into the layerNorm normalization module, the 1 multiplied by 1 convolution layer, the simpleGate module and the 1 multiplied by 1 convolution layer; and finally, fusing the fusion processing result with the processing result obtained at present to obtain the processing result of the ConvNeXt-Block structure, and outputting the processing result.
8. Image enhancement system based on multi-level curvature supervision for implementing an image enhancement method based on multi-level curvature supervision according to any one of the claims 1 to 7, comprising the following modules:
the model construction module is used for constructing a double-layer denoising network model based on a neural network;
the feature extraction module is arranged to receive the original image data and extract features in a parallel branch processing mode;
the feature fusion module is arranged for fusing the branch processing results to obtain fusion features;
a noise estimation module configured to perform noise level estimation and noise distribution estimation based on the obtained fusion features;
the curvature calculation module is used for carrying out fusion according to the obtained noise distribution estimation and the original image data and calculating the corresponding curvature;
an image denoising module configured to perform image denoising processing using an encoder-decoder structure according to curvature, noise level estimation, original image data;
and an image output module configured to output the denoised image data.
9. An image enhancement device based on multi-level curvature supervision, the device comprising:
a processor and a memory storing computer program instructions;
the processor reads and executes the computer program instructions to implement the multi-level curvature supervision based image enhancement method as claimed in any one of the claims 1 to 7.
10. A computer readable storage medium, having stored thereon computer program instructions which, when executed by a processor, implement the multi-level curvature supervision based image enhancement method according to any one of the claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311036758.6A CN116757966A (en) | 2023-08-17 | 2023-08-17 | Image enhancement method and system based on multi-level curvature supervision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311036758.6A CN116757966A (en) | 2023-08-17 | 2023-08-17 | Image enhancement method and system based on multi-level curvature supervision |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116757966A true CN116757966A (en) | 2023-09-15 |
Family
ID=87953631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311036758.6A Pending CN116757966A (en) | 2023-08-17 | 2023-08-17 | Image enhancement method and system based on multi-level curvature supervision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116757966A (en) |
Citations (19)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111007566A (en) * | 2019-12-27 | 2020-04-14 | 西南石油大学 | Curvature-driven diffusion full-convolution network seismic data bad channel reconstruction and denoising method |
| WO2020140986A1 (en) * | 2019-01-04 | 2020-07-09 | Oppo广东移动通信有限公司 | Image denoising method and apparatus, storage medium and terminal |
| CN112200750A (en) * | 2020-10-21 | 2021-01-08 | 华中科技大学 | Ultrasonic image denoising model establishing method and ultrasonic image denoising method |
| CN112233038A (en) * | 2020-10-23 | 2021-01-15 | 广东启迪图卫科技股份有限公司 | True image denoising method based on multi-scale fusion and edge enhancement |
| CN112529886A (en) * | 2020-12-18 | 2021-03-19 | 西北工业大学 | Attention DenseUNet-based MRI glioma segmentation method |
| WO2021113550A1 (en) * | 2019-12-03 | 2021-06-10 | Click Therapeutics, Inc. | Apparatus, system, and method for determining demographic information to facilitate mobile application user engagement |
| CN113450290A (en) * | 2021-09-01 | 2021-09-28 | 中科方寸知微(南京)科技有限公司 | Low-illumination image enhancement method and system based on image inpainting technology |
| US20210358086A1 (en) * | 2020-05-13 | 2021-11-18 | Ambu A/S | Method for adaptive denoising and sharpening and visualization systems implementing the method |
| US20220066456A1 (en) * | 2016-02-29 | 2022-03-03 | AI Incorporated | Obstacle recognition method for autonomous robots |
| CN114219719A (en) * | 2021-10-27 | 2022-03-22 | 浙江工业大学 | CNN medical CT image denoising method based on dual attention and multi-scale features |
| CN114511480A (en) * | 2022-01-25 | 2022-05-17 | 江苏科技大学 | Underwater image enhancement method based on fractional order convolution neural network |
| WO2022126588A1 (en) * | 2020-12-18 | 2022-06-23 | 深圳先进技术研究院 | Pet-mri image denoising method and apparatus based on dual-encoding fusion network model |
| CN114861709A (en) * | 2022-04-13 | 2022-08-05 | 济南大学 | Bearing noise mode identification and mute optimization method |
| CN115345801A (en) * | 2022-10-18 | 2022-11-15 | 中科方寸知微(南京)科技有限公司 | Image compression and filter removal method and system based on image denoising idea |
| CN115564782A (en) * | 2022-10-13 | 2023-01-03 | 重庆大学 | 3D blood vessel and trachea segmentation method and system |
| CN116342406A (en) * | 2023-02-16 | 2023-06-27 | 南京大学 | Ultra-high resolution image denoising method based on deep learning |
| CN116523800A (en) * | 2023-07-03 | 2023-08-01 | 南京邮电大学 | Image noise reduction model and method based on residual dense network and attention mechanism |
| CN116563146A (en) * | 2023-04-26 | 2023-08-08 | 中科南京人工智能创新研究院 | Image enhancement method and system based on leachable curvature map |
| CN116580195A (en) * | 2023-04-26 | 2023-08-11 | 齐鲁工业大学(山东省科学院) | Remote sensing image semantic segmentation method and system based on ConvNeXt convolution |
-
2023
- 2023-08-17 CN CN202311036758.6A patent/CN116757966A/en active Pending
Patent Citations (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220066456A1 (en) * | 2016-02-29 | 2022-03-03 | AI Incorporated | Obstacle recognition method for autonomous robots |
| WO2020140986A1 (en) * | 2019-01-04 | 2020-07-09 | Oppo广东移动通信有限公司 | Image denoising method and apparatus, storage medium and terminal |
| WO2021113550A1 (en) * | 2019-12-03 | 2021-06-10 | Click Therapeutics, Inc. | Apparatus, system, and method for determining demographic information to facilitate mobile application user engagement |
| CN111007566A (en) * | 2019-12-27 | 2020-04-14 | 西南石油大学 | Curvature-driven diffusion full-convolution network seismic data bad channel reconstruction and denoising method |
| US20210358086A1 (en) * | 2020-05-13 | 2021-11-18 | Ambu A/S | Method for adaptive denoising and sharpening and visualization systems implementing the method |
| WO2022083026A1 (en) * | 2020-10-21 | 2022-04-28 | 华中科技大学 | Ultrasound image denoising model establishing method and ultrasound image denoising method |
| CN112200750A (en) * | 2020-10-21 | 2021-01-08 | 华中科技大学 | Ultrasonic image denoising model establishing method and ultrasonic image denoising method |
| CN112233038A (en) * | 2020-10-23 | 2021-01-15 | 广东启迪图卫科技股份有限公司 | True image denoising method based on multi-scale fusion and edge enhancement |
| WO2022126588A1 (en) * | 2020-12-18 | 2022-06-23 | 深圳先进技术研究院 | Pet-mri image denoising method and apparatus based on dual-encoding fusion network model |
| CN112529886A (en) * | 2020-12-18 | 2021-03-19 | 西北工业大学 | Attention DenseUNet-based MRI glioma segmentation method |
| CN113450290A (en) * | 2021-09-01 | 2021-09-28 | 中科方寸知微(南京)科技有限公司 | Low-illumination image enhancement method and system based on image inpainting technology |
| CN114219719A (en) * | 2021-10-27 | 2022-03-22 | 浙江工业大学 | CNN medical CT image denoising method based on dual attention and multi-scale features |
| CN114511480A (en) * | 2022-01-25 | 2022-05-17 | 江苏科技大学 | Underwater image enhancement method based on fractional order convolution neural network |
| CN114861709A (en) * | 2022-04-13 | 2022-08-05 | 济南大学 | Bearing noise mode identification and mute optimization method |
| CN115564782A (en) * | 2022-10-13 | 2023-01-03 | 重庆大学 | 3D blood vessel and trachea segmentation method and system |
| CN115345801A (en) * | 2022-10-18 | 2022-11-15 | 中科方寸知微(南京)科技有限公司 | Image compression and filter removal method and system based on image denoising idea |
| CN116342406A (en) * | 2023-02-16 | 2023-06-27 | 南京大学 | Ultra-high resolution image denoising method based on deep learning |
| CN116563146A (en) * | 2023-04-26 | 2023-08-08 | 中科南京人工智能创新研究院 | Image enhancement method and system based on leachable curvature map |
| CN116580195A (en) * | 2023-04-26 | 2023-08-11 | 齐鲁工业大学(山东省科学院) | Remote sensing image semantic segmentation method and system based on ConvNeXt convolution |
| CN116523800A (en) * | 2023-07-03 | 2023-08-01 | 南京邮电大学 | Image noise reduction model and method based on residual dense network and attention mechanism |
Non-Patent Citations (6)
| Title |
|---|
| ADIL ELYAR等: ""Cascaded U-Net with Training Wheel Attention Module for Change Detection in Satellite Images"", 《REMOTE SENSING》, vol. 14, no. 24, pages 6361 - 6361 * |
| WHETHERFAILBUTTRY: ""图像多尺度特征融合、特征金字塔总结"", 《HTTPS://BLOG.CSDN.NET/QQ_40602790/ARTICLE/DETAILS/124079984》, pages 1 - 4 * |
| YUTONG LI等: ""CurvPnP: Plug-and-play Blind Image Restoration with Deep Curvature Denoiser"", 《ARXIV》, pages 4 - 8 * |
| 张岩等: ""基于深度学习的鲁棒地震数据去噪"", 《石油地球物理勘探》, vol. 57, no. 1, pages 12 - 25 * |
| 李梦星等: ""基于玻尔兹曼函数和磁畴能量最小化的电工钢片磁滞特性模拟"", 《中国电机工程学报》, pages 1 - 13 * |
| 赖国良等: ""基于语义似然与高精度地图匹配的智能车辆同时定位与检测"", 《上海交通大学学报》, pages 1 - 20 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113673307B (en) | Lightweight video action recognition method | |
| CN112132959B (en) | Digital rock core image processing method and device, computer equipment and storage medium | |
| CN109671029A (en) | Image denoising algorithm based on gamma norm minimum | |
| CN111681180B (en) | Priori-driven deep learning image defogging method | |
| CN115841447A (en) | Detection method for surface defects of magnetic shoe | |
| CN113870124B (en) | Weak supervision-based double-network mutual excitation learning shadow removing method | |
| CN114581330A (en) | Terahertz image denoising method based on multi-scale mixed attention | |
| CN115063318A (en) | Adaptive frequency-resolved low-illumination image enhancement method and related equipment | |
| CN116563146A (en) | Image enhancement method and system based on leachable curvature map | |
| CN117197627B (en) | Multi-mode image fusion method based on high-order degradation model | |
| Yi et al. | Towards Compact Single Image Dehazing via Task-related Contrastive Network | |
| CN113240581A (en) | Real world image super-resolution method for unknown fuzzy kernel | |
| CN117391981A (en) | Infrared and visible light image fusion method based on low-light illumination and self-adaptive constraint | |
| CN117058079A (en) | Thyroid imaging image automatic diagnosis method based on improved ResNet model | |
| CN116703750A (en) | Image defogging method and system based on edge attention and multi-order differential loss | |
| Liu et al. | Joint dehazing and denoising for single nighttime image via multi-scale decomposition | |
| CN116385281A (en) | Remote sensing image denoising method based on real noise model and generated countermeasure network | |
| CN116757966A (en) | Image enhancement method and system based on multi-level curvature supervision | |
| CN116309221A (en) | Method for constructing multispectral image fusion model | |
| CN112712520A (en) | Choroid layer segmentation method based on ARU-Net | |
| CN116523767B (en) | Image defogging method and system combined with fog concentration classification | |
| CN117095208B (en) | Lightweight scene classification method for photoelectric pod reconnaissance image | |
| CN113487496B (en) | Image denoising method, system and device based on pixel type inference | |
| Shaliniswetha et al. | RESIDUAL LEARNING BASED IMAGE DENOISING AND COMPRESSION USING DNCNN. | |
| CN117474797B (en) | Image denoising method and device for multi-scale complementary learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |