CN113766250A - Compressed image quality improving method based on sampling reconstruction and feature enhancement - Google Patents
Compressed image quality improving method based on sampling reconstruction and feature enhancement Download PDFInfo
- Publication number
- CN113766250A CN113766250A CN202011057427.7A CN202011057427A CN113766250A CN 113766250 A CN113766250 A CN 113766250A CN 202011057427 A CN202011057427 A CN 202011057427A CN 113766250 A CN113766250 A CN 113766250A
- Authority
- CN
- China
- Prior art keywords
- feature
- quality
- compressed image
- image
- residual
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 76
- 238000005070 sampling Methods 0.000 title claims abstract description 36
- 230000006872 improvement Effects 0.000 claims abstract description 62
- 230000008707 rearrangement Effects 0.000 claims abstract description 32
- 238000012549 training Methods 0.000 claims abstract description 30
- 238000012545 processing Methods 0.000 claims abstract description 23
- 238000012360 testing method Methods 0.000 claims abstract description 16
- 238000010191 image analysis Methods 0.000 claims abstract description 11
- 238000007906 compression Methods 0.000 claims description 57
- 230000006835 compression Effects 0.000 claims description 54
- 230000004927 fusion Effects 0.000 claims description 21
- 238000003708 edge detection Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 14
- 230000009467 reduction Effects 0.000 claims description 13
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000011176 pooling Methods 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 4
- 230000002708 enhancing effect Effects 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 230000004913 activation Effects 0.000 claims description 2
- 230000008859 change Effects 0.000 claims description 2
- 238000010606 normalization Methods 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims 1
- 238000004458 analytical method Methods 0.000 abstract description 5
- 230000005540 biological transmission Effects 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 10
- 230000000903 blocking effect Effects 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 238000001914 filtration Methods 0.000 description 3
- 238000013441 quality evaluation Methods 0.000 description 3
- 230000001629 suppression Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 101100161752 Mus musculus Acot11 gene Proteins 0.000 description 1
- 230000035508 accumulation Effects 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 238000011084 recovery Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/85—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression
- H04N19/88—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using pre-processing or post-processing specially adapted for video compression involving rearrangement of data among different coding units, e.g. shuffling, interleaving, scrambling or permutation of pixel data or permutation of transform coefficient data among different blocks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/102—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or selection affected or controlled by the adaptive coding
- H04N19/132—Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/146—Data rate or code amount at the encoder output
- H04N19/149—Data rate or code amount at the encoder output by estimating the code amount by means of a model, e.g. mathematical model or statistical model
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/154—Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- Algebra (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Image Processing (AREA)
Abstract
The invention provides a method for improving the quality of a compressed image based on sampling reconstruction and feature enhancement. The method mainly comprises the following steps: designing a depth model for improving the quality of a compressed image, wherein the depth model comprises modules of sampling based on overlapped pixel rearrangement, residual prediction based on feature enhancement, reconstruction based on overlapped pixel rearrangement and the like; constructing a training image set for training a compressed image quality improvement depth model, wherein the training image set comprises a compressed image set and a high-quality image set which have a corresponding relation; training a compressed image quality enhancement depth model based on a training image set; processing the input test image by using the trained compressed image quality improvement depth model; and evaluating the quality improvement result or further analyzing and understanding the quality improvement result. The method can obviously improve the quality of the compressed image, has lower complexity and can improve the performance of image analysis and understanding. The invention has important application value in the directions of image and video storage, transmission, analysis and the like.
Description
Technical Field
The invention relates to a compressed image quality enhancement technology, in particular to a compressed image quality improvement method based on sampling recombination and feature enhancement, and belongs to the field of image processing.
Background
With the development of imaging devices or systems such as mobile phones, cameras, monitors and the like and the popularization of internet technology, images and videos are more and more widely and deeply applied in daily life, entertainment, education, medical treatment, aerospace, military and other directions, and play a significant role in the fields. With the consequent rapid increase in the data volume of images and videos, enormous pressure and challenges are brought to the corresponding storage and transmission systems. In order to relieve the storage and transmission pressure of images and videos, data redundancy is usually reduced by lossy compression to reduce the data amount, for example, JPEG and JPEG 2000 are more common still image compression methods, and MPEG, h.264 and HEVC are video compression technologies. However, lossy compression reduces the amount of data at the cost of sacrificing the quality of the image or video, i.e., the decompressed image or video is not consistent with the original information, but has some distortion, i.e., compression noise. Particularly, when the compression ratio is large, severe compression noise exists in the decompressed image or video, which brings blocking effect, banding effect, ringing effect, blurring effect, etc., and seriously affects the visual effect or the subsequent analysis and utilization thereof. How to improve the quality of the lossy compressed image or video to better reflect the original information before compression is one of the difficulties to be solved in the field of image and video coding.
For quality improvement of compressed images and video, the main objective is to suppress the compression noise introduced by the lossy compression process, such as blocking, banding, ringing, etc., while recovering some image structures that were destroyed in the compression process. In particular, the quality improvement method of the compressed image mainly includes a filtering-based method, a priori-based method, learning-based method, and the like. The filtering-based method can inhibit noise such as blocking effect in a filtering smoothing mode; the prior-based method is to utilize the characteristics of local smoothness, non-local similarity, sparsity and the like which are met by a natural image to constrain the image and realize the removal of compression noise; the learning-based method realizes the suppression of compression noise through the mapping relation between the compressed image and the original image, thereby improving the quality. In general, the learning-based method is superior in quality improvement performance, and particularly, the deep learning-based compressed image quality improvement method, which has been rapidly developed in recent years, is used. However, many compressed image quality improvement methods based on deep learning improve performance by enlarging models, which results in large parameters and large calculation amount of the models, and thus are difficult to be applied in practice. How to further improve the performance of quality enhancement of compressed images without increasing the parameters and the computational complexity requires more intensive research.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a method for improving the quality of a compressed image based on sampling reconstruction and feature enhancement.
In order to achieve the purpose, the invention adopts the following technical scheme:
a compressed image quality improving method based on sampling reconstruction and feature enhancement mainly comprises the following steps:
(1.1) sampling an image input to the compressed image quality improvement model based on the overlapped pixel rearrangement operation;
(1.2) designing a residual error prediction module based on feature enhancement based on basic components such as a convolutional layer and a nonlinear layer and combining dense connection, an attention mechanism, a residual error structure and the like, extracting features and enhancing features from the sampled image in the step (1.1), and predicting residual error components;
(1.3) reconstructing the residual predicted in step (1.2) based on an overlapping pixel rearrangement operation opposite to step (1.1);
and (1.4) overlapping the residual image reconstructed in the step (1.3) to the input image based on the global residual connection to obtain a prediction result of improving the quality of the compressed image.
(2.1) collecting a plurality of high-quality images to construct a high-quality image set;
(2.2) compressing and decompressing the high-quality image constructed in the step (2.1) based on an image compression method to obtain a compressed image set;
and (2.3) combining the high-quality image set constructed in the step (2.1) and the compressed image set obtained in the step (2.2) to construct a training image set with one-to-one correspondence.
(3.1) establishing a loss function for measuring quality improvement errors;
(3.2) inputting the images in the compressed image set constructed in the step (2.2) into the compressed image quality improvement depth model designed in the step (1) for processing to obtain a quality improvement result;
(3.3) acquiring a high-quality image corresponding to the input compressed image used in the step (3.2) from the high-quality image set constructed in the step (2.1), and comparing the high-quality image with the quality improvement result predicted in the step (3.2), namely measuring the prediction error by using the loss function established in the step (3.1);
(3.4) updating parameters in the compressed image quality enhancement depth model by using a depth model learning optimization algorithm based on the prediction error calculated in the step (3.3);
and (3.5) repeating the steps (3.2) to (3.4) until the prediction error is converged, and finishing the training of the compressed image quality enhancement depth model.
And 4, taking the compressed image for testing as input, and processing by using the compressed image quality improvement depth model trained in the step 3 to obtain a quality improvement result corresponding to the tested image.
And 5, evaluating the quality improvement result obtained in the step 4, or further analyzing and understanding the quality improvement result, such as edge detection and the like.
By adopting the technical scheme, the invention has the following advantages: (1) the invention integrates the sampling module and the reconstruction module based on overlapping pixel rearrangement at the head part and the tail part of the depth model for improving the quality of the compressed image respectively, thereby greatly reducing the calculation complexity to improve the processing speed and simultaneously enlarging the receptive field of the depth model to improve the performance. (2) The invention adopts multi-scale cavity convolution to extract more abundant characteristics, and utilizes a characteristic fusion unit based on an attention mechanism to fully utilize the characteristics so as to improve the processing effect. (3) The invention combines the high-efficiency 1 multiplied by 1 convolution to reduce the parameter complexity and the calculation complexity of the depth network model, reduce the model and improve the processing speed. (4) According to the invention, a multi-level residual structure is constructed by utilizing the connection of the residual errors of the feature domains and the space domains of different levels, so that the training of a stable depth network is facilitated, and meanwhile, the performance can be improved. (5) The invention combines the quality improvement technology of the compressed image, constructs an image analysis understanding framework suitable for the compressed image, and can improve the accuracy of image analysis understanding.
Drawings
FIG. 1 is a block diagram of a flow chart of a compressed image quality improving method based on sampling reconstruction and feature enhancement according to the present invention;
FIG. 2 is a block diagram of a compressed image quality enhancement network based on sample reconstruction and feature enhancement according to the present invention;
FIG. 3 is a schematic diagram of sampling and reconstruction based on overlapping pixel rearrangement in accordance with the present invention;
FIG. 4 is a schematic block diagram of an image analysis understanding method of the present invention for joint compression image quality enhancement;
FIG. 5 is a comparison graph of the quality improvement results of the test image "Barbara" according to the present invention and three methods (compression method: JPEG, compression quality factor: 10), wherein (a) is the original uncompressed image, (b) is the JPEG compressed image, and (c) (d) (e) (f) are the results of method 1, method 2, method 3 and the present invention, respectively;
FIG. 6 is a comparison graph of the quality improvement results of the test image "Carnivaldolls" according to the present invention and three methods (compression method: JPEG, compression quality factor: 10), wherein (a) is the original uncompressed image, (b) is the JPEG compressed image, and (c) (d) (e) (f) are the results of method 1, method 2, method 3, and the present invention, respectively;
FIG. 7 is a comparison graph of the model parameter quantity and the average Peak Signal to Noise Ratio (PSNR) of the present invention and three methods (test image set: class 5, compression method: JPEG, compression quality factor: 10);
FIG. 8 is a comparison of the joint edge detection results of the present invention, wherein (a) is the original uncompressed image and its edge detection result, (b) is the JPEG-compressed image and its edge detection result, and (c) is the quality enhancement result and its edge detection result.
Detailed Description
In order that the invention may be more clearly understood, the invention will now be described in more detail with reference to the accompanying drawings, in conjunction with the detailed description. It is to be understood that the present invention may be embodied in various forms without being limited by the accompanying drawings and the embodiments described below. The drawings and the embodiments described below are provided so that the present invention will be more fully and accurately understood by those skilled in the art.
In fig. 1, a method for improving the quality of a compressed image based on sampling reconstruction and feature enhancement may specifically include the following five steps:
(1.1) sampling an image input to the compressed image quality improvement model based on the overlapped pixel rearrangement operation;
(1.2) designing a residual error prediction module based on feature enhancement based on basic components such as a convolutional layer and a nonlinear layer and combining dense connection, an attention mechanism, a residual error structure and the like, extracting features and enhancing features from the sampled image in the step (1.1), and predicting residual error components;
(1.3) reconstructing the residual predicted in step (1.2) based on an overlapping pixel rearrangement operation opposite to step (1.1);
and (1.4) overlapping the residual image reconstructed in the step (1.3) to the input image based on the global residual connection to obtain a prediction result of improving the quality of the compressed image.
(2.1) collecting a plurality of high-quality images to construct a high-quality image set;
(2.2) compressing and decompressing the high-quality image constructed in the step (2.1) based on an image compression method to obtain a compressed image set;
and (2.3) combining the high-quality image set constructed in the step (2.1) and the compressed image set obtained in the step (2.2) to construct a training image set with one-to-one correspondence.
(3.1) establishing a loss function for measuring quality improvement errors;
(3.2) inputting the images in the compressed image set constructed in the step (2.2) into the compressed image quality improvement depth model designed in the step (1) for processing to obtain a quality improvement result;
(3.3) acquiring a high-quality image corresponding to the input compressed image used in the step (3.2) from the high-quality image set constructed in the step (2.1), and comparing the high-quality image with the quality improvement result predicted in the step (3.2), namely measuring the prediction error by using the loss function established in the step (3.1);
(3.4) updating parameters in the compressed image quality enhancement depth model by using a depth model learning optimization algorithm based on the prediction error calculated in the step (3.3);
and (3.5) repeating the steps (3.2) to (3.4) until the prediction error is converged, and finishing the training of the compressed image quality enhancement depth model.
And 4, taking the compressed image for testing as input, and processing by using the compressed image quality improvement depth model trained in the step 3 to obtain a quality improvement result corresponding to the tested image.
And 5, evaluating the quality improvement result obtained in the step 4, or further analyzing and understanding the quality improvement result, such as edge detection and the like.
Specifically, in step 1, the compressed image quality enhancement depth model, i.e. the compressed image quality enhancement network based on sampling reconstruction and feature enhancement as shown in fig. 2, takes the residual error network as the main structure, and mainly comprises a sampling module based on overlapping pixel rearrangement, a residual error prediction module based on feature enhancement, and a reconstruction module based on overlapping pixel rearrangement, which are expressed as a whole
In which is shown
And IcRespectively representing the result of the quality enhancement processing of the compressed image and the input compressed image, Fds(. is a sampling module based on overlapping pixel rearrangement, Frp(. is a residual prediction module based on feature enhancement, Frc(. is a reconstruction module based on overlapping pixel rearrangement.
Further, in the step (1.1), the sampling operation based on the overlapped pixel rearrangement is as shown in fig. 3, wherein the left side is the input compressed image
Right side of theAs a result of sampling
In particular to
Wherein,
c＝1,…,d2and c ═ d (x)0-1)+y0；(x0,y0) Rearranging the coordinates of the start point of the sample based on the overlapped pixels, and x0＝1,…,d，y 01, …, d. In the present invention, d is 4 as an example, but other values may be used. The above formula is realized incIn (x)0,y0) Sampling at intervals in horizontal and vertical directions as starting points, and measuring the sampling size
As a result of sampling of
D (x) of0-1)+y0A channel. In particular, the invention is first to IcIs mirrored to ensure d2The sizes of the sampling images are all
The overlapped pixels in different channels are used as the input of the residual prediction network, so that the network receptive field can be enlarged, and the neighborhood information is more fully utilized.
Further, in the step (1.2), the residual prediction network based on feature enhancement is shown in fig. 2, which is mainly composed of KFThe dense connection residual error module based on the characteristic enhancement and the characteristic fusion module based on the attention re-calibration constituteIs shown as
Wherein,
shallow features extracted based on convolutional layer (convolutional kernel size: 3X 3, number of convolutional kernels: 64), Ffe(. h) is a corresponding operation; f. offrb_kPredicting the output of the kth feature enhancement based densely connected residual module in the network for the residual, and having ffrb_k＝Ffrb_k(ffrb_k-1)，Ffrb_k(. h) is a corresponding operation; fgafb() is a global feature fusion operation that fuses the output features of the feature enhanced dense connection residual module, which is implemented by a feature fusion module based on attention re-calibration; frr(. is) based on convolutional layers (convolutional kernel size: 3X 3, number of convolutional kernels: d)2) The residual reconstruction operation of (1). In the present invention, K is usedFAs an example, 6, but other values may be taken.
One of the main components of the residual prediction network based on feature enhancement is a dense connection residual module based on feature enhancement, as shown in fig. 2, which uses the dense connection residual module as a backbone network and performs feature enhancement by combining multi-scale feature extraction. The kth feature enhancement based dense connection residual module can be represented as
Wherein, Flafb_k(.) is a local feature fusion operation in the kth feature enhancement based dense connected residual module, which is implemented by the feature fusion module based on attention re-targeting; f. ofk_mfb_pAnd representing the output characteristics of the p-th multi-scale characteristic extraction module in the k-th dense connection residual module based on characteristic enhancement. In the present invention, P isMFor example, 8, but other values may be used. Feature enhanced dense connection residual moduleThe multi-scale feature extraction module used in (1) is shown in fig. 2. For the p-th multi-scale feature extraction module in the k-th feature enhanced dense connection residual module, the input of the k-th multi-scale feature extraction module comprises the input f of the k-th feature enhanced dense connection residual modulefrb_k-1And the output { f of the first p-1 multi-scale feature extraction modulesk_mfb_1,…,fk_mfb_p-1Expressed as a whole
Wherein, Fcct(. for characteristic connection operations in the direction of the passage, Ffdr1(. to) convolution operation (convolution kernel size: 1X 1, number of convolution kernels: 64), F, for compressing the connected features of the channelmfe(. is a multi-scale feature extraction based on hole convolution, Ffdr2(. cndot.) is a convolution operation that compresses the multi-scale features (convolution kernel size: 1 × 1, number of convolution kernels: 64). In order to obtain multi-scale information and expand the network receptive field, the multi-scale feature extraction is realized based on the combination of the hole convolutions with different step lengths, which is expressed as
Wherein f ismfeDenotes from ffdr1Multi-scale features extracted from (1), ffdr1Is Ffdr1(ii) an output of the operation; fdfe1(·)、Fdfe2(·)、Fdfe3(. and F)dfe4(. cndot.) is a hole convolution operation with a convolution kernel size of 3 x 3, a number of convolution kernels of 32, and step sizes of 1, 2, 3, and 4, respectively.
One of the main components of the residual prediction network based on feature enhancement is a feature fusion module based on attention re-calibration, as shown in fig. 2, which performs adaptive re-calibration and dimension reduction on input features based on an attention mechanism to realize feature fusion. The feature fusion module based on attention re-calibration mainly comprises attention-based feature re-calibration and feature dimension reduction. For convenience of presentation, let us say thatOne set of characteristics is f1kK＝Fcct(f1,…,fk,…,fK) Feature fusion F based on attention re-targetingafb(. represents) as
Wherein f isafbAs a result of the fusion, Fafrb(. denotes attention-based feature recalibration, Ffdrb(. cndot.) is characteristic dimension reduction. Attention-based feature recalibration establishes a dependency relationship between channels in input features, which includes processes of absolute value quantization, global mean and maximum pooling, feature compression and stretching, scale normalization, recalibration, and the like, and is expressed as
Wherein, Fafrb(. and f)afrbRespectively the function and corresponding output realized by the attention-based feature recalibration module, Fabs(. to be operations for calculating absolute values, Fgap(. is a global mean pooling, Fgmp(. is a global maximum pooling, Ffss(. cndot.) is the characteristic compression and stretching, and σ (·) is the Sigmoid function. The characteristic compression and stretching are realized based on full convolution layers positioned at two sides of a ReLU activation function, and s times of compression and stretching are respectively carried out on the characteristic, wherein s is taken as 16 as an example in the invention. After the coefficient is normalized by sigma (-), the coefficient is multiplied by the input characteristic to realize recalibration. The feature dimension reduction module performs dimension reduction and fusion on the features after the re-calibration so as to reduce complexity and improve performance, and is expressed as
Wherein, Ffdrb(. and f)fdrbRespectively representing a feature dimension reduction module and its output, comprising convolution layers for dimension reduction (convolution kernel size: 1) and a function1, number of convolution kernels: 64) and convolutional layers to better fuse information (convolutional kernel size: 3 × 3, number of convolution kernels: 64) respectively with Ffdr3(. o) well and Fff(. -) represents.
Further, in the step (1.3), the reconstruction operation based on the overlapped pixel rearrangement is shown in fig. 3, wherein the right side is the residual error of the residual error network prediction based on the feature enhancement
And is
Left side is the reconstructed result
The reconstruction operation based on the overlapped pixel rearrangement in the step (1.3) and the sampling operation based on the overlapped pixel rearrangement in the step (1.1) are inverse processes, which is expressed as
Wherein,
is composed of
The second characteristic channel of (a) is,
to represent
Accumulating the channels according to the corresponding original positions; ω is a coefficient that normalizes the accumulated sum, which depends on the number of accumulations for each point.
Further, in the step (1.4), based on global residual learning, from the compressed image IcProcessing the obtained quality improvement result
Is composed of
Specifically, in the step 2, the image data comprises N high-quality images
Is represented as
The process of constructing the training image set of the compressed image quality enhancement depth model is expressed as
Wherein,
representing high quality images
Corresponding compressed image, Fc(. represents a compression process, S)cFor the corresponding compression method, PcSetting compression parameters. Then there is a change in the number of,
and
a high quality image set and a compressed image set that have a one-to-one correspondence. In the present invention, the JPEG standard, compression quality factor 10, is taken as an example, but it may be possible to use other settings.
Specifically, in the step 3, based on the compressed image set constructed in the step 2
And corresponding high quality image sets
And (3) training the compressed image quality improvement depth model designed in the step (1), namely a compressed image quality improvement network based on sampling reconstruction and feature enhancement.
Further, in the step (3.1), L is adopted2The norm establishes a loss function that measures quality improvement error.
Further, in the step (3.2), the compressed image set established in the step 2 is used
Image of (1)
As input, the depth model F is boosted with the compressed image quality in step 1ciqenet(v.) processing to obtain a quality improvement result, expressed as
Wherein,
is composed of
Compressed image quality enhancement depth model FciqenetResults of the treatment.
Further, in the step (3.3), based on the loss function established in the step (3.1), the image compressed in the step (3.2) is subjected to
Predicted quality improvement results
Is measured and expressed as
Wherein,
from high quality image sets
For compressing images
Corresponding high quality images. The above equation only illustrates the calculation of the prediction error by taking a single image as an example.
Further, in the step (3.4), based on the prediction error calculated in the step (3.3), an Adam optimizer is used to update the parameters of the compression quality promotion depth model. Wherein the optimizer may select other methods.
Further, in the step (3.5), the steps (3.2) to (3.4) are repeated until the prediction error calculated in the step (3.3) converges, which indicates that the training of the compressed image quality enhancement depth model is completed, and the model is saved for use in subsequent tests.
Specifically, in the step 4, the compressed image quality trained in the step 3 is used to improve the depth model FciqenetFor compressed images for testing
Performing a treatment represented by
Wherein,
is composed of
The corresponding quality improvement result.
Specifically, in the step 5, the quality improvement result predicted in the step 4 is obtained
For evaluation or for subsequent analysis and understanding.
Further, in the step 5, the quality improvement processing result is processed
The evaluation of (2) can be combined with subjective quality evaluation and objective parameter evaluation. Subjective quality evaluation is mainly based on human eye observation, and the conditions of compression noise suppression and detail keeping recovery are comprehensively considered. For objective parameter evaluation, e.g. presence of test images
Corresponding original uncompressed image ItestThe evaluation can be performed by using all reference indexes such as Peak Signal to Noise Ratio (PSNR) and Structure Similarity Index (SSIM); if there is no ItestThen quality evaluation can be performed using the index parameters without reference.
Further, in the step 5, for the subsequent analysis understanding of the compressed image quality improvement result, the flow chart thereof is shown in fig. 4. Firstly, improving the quality of a compressed image as a preprocessor to improve the quality of an input compressed image so as to reduce the influence of compression noise on subsequent analysis and understanding; and then, taking the quality improvement result of the compressed image as input, and carrying out subsequent image analysis and understanding processing such as edge extraction.
In order to verify the effectiveness of the method, the invention uses standard test image sets classic5 and LIVE1 which are commonly used in the field of compressed image quality improvement processing to carry out experiments. For the raw high quality images in class 5 and LIVE1, compression was performed using the JPEG compression encoding standard, with the compression quality factor set to 10. Selecting three compressed image quality improvement processing algorithms based on deep learning as comparison methods, specifically:
the method comprises the following steps: the methods proposed by Dong et al, references "C.Dong, Y.Deng, C.L.Chen, and X.Tang," Compression aspects reduction by a depth compatibility network, "in Proceedings of the International Conference on Computer Vision (ICCV)," 2015, pp.576-584 "
The method 2 comprises the following steps: methods proposed by Chen et al, references "H.Chen, X.He, L.Qing, S.Xiong, and T.Q.Nguyen," DPWSDNet: Dual pixel-level domain depth CNNs for soft decoding of JPEG-compressed images, "in Proceedings of the IEEE Conference Computer Vision and Pattern Recording Workstations (CVPRW),2018, pp.711-720"
The method 3 comprises the following steps: the method proposed by Zhang et al, references "Y.Zhang, K.Li, K.Li, B.Zhang, and Y.Fu," scientific non-local authentication networks for image retrieval, "in International Conference on Learning retrieval (ICLR), 2019"
The experimental contents of the comparison are as follows:
table 1 and table 2 show the average PSNR and average SSIM values obtained by JPEG and different compressed image quality improvement methods on the standard test image sets class 5 and LIVE1, respectively. In this experiment, the JPEG compression encoding standard was used for compression, and the compression quality factor was set to 10.
Fig. 5 and 6 respectively compare the visual effects of different compressed image quality improving methods on the processing results of the standard test images "barbarara" and "Carnivaldolls". In fig. 5 and 6, (a) is the original uncompressed image, (b) is the JPEG compressed image, and (c) (d) (e) (f) are the results of method 1, method 2, method 3, and the present invention, respectively. In this experiment, the JPEG compression encoding standard was used for compression, and the compression quality factor was set to 10.
FIG. 7 compares the model parameters for different compressed image quality enhancement methods and the average PSNR values obtained at class 5. In this experiment, the JPEG compression encoding standard was used for compression, and the compression quality factor was set to 10.
Fig. 8 illustrates the effect of understanding the subsequent image analysis when the method of the present invention is used as a preprocessor, taking the edge detection of a typical image analysis understanding task as an example. Wherein, (a) is the original uncompressed image and its edge detection result, (b) is the JPEG compressed image and its edge detection result, and (c) is the quality improvement result and its edge detection result of the present invention. In the experiment, JPEG compression coding standard is adopted for compression, and the compression quality factor is set to be 10; the Canny operator method is adopted for edge detection.
As can be seen from the PSNR values given in Table 1 and the SSIM values given in Table 2, the method of the present invention achieves the highest values on both the two standard test image sets and the two indexes, and has the best quality from the viewpoint of objective indexes. Compared with JPEG compressed images, the PSNR and SSIM of the invention have very obvious promotion amplitude, and the method of the invention is also comprehensively superior to the three compared methods. Comparing the original image, the JPEG compressed image and the quality improvement results of the methods shown in fig. 5 and fig. 6, it can be seen that: the JPEG compressed image has serious compression noise which is reflected as blocking effect, banding effect, ringing effect, fuzzy effect and the like, and the visual perception is seriously influenced; after the quality improvement treatment is carried out by each method, the quality is obviously improved, but the method of the invention is better in noise suppression and detail maintenance on the whole. The effectiveness and advantages of the present invention are illustrated by the comparison of subjective visual effects and objective parameters.
As can be seen from the comparison between the model parameters and the objective parameter PSNR of the different methods shown in FIG. 7, the present invention improves the effect while reducing the model parameters, which shows that the present invention better balances the parameter complexity and the quality improvement performance.
As can be seen from the edge detection result shown in fig. 8, the compression noise significantly affects the accuracy of edge detection, and especially the blocking effect and the banding effect are falsely detected as edges. However, the invention is used as preprocessing to improve the quality of the compressed image and then carry out edge detection, so that more accurate edges can be obtained. The results illustrate the role of the present invention in image analysis understanding tasks such as edge detection.
In summary, the present invention is an effective method for improving quality of compressed images, and can be used to assist other image analysis and understanding tasks.
TABLE 1
TABLE 2
Claims (9)
1. A compressed image quality improving method based on sampling reconstruction and feature enhancement is characterized by comprising the following steps:
step 1, designing a depth model for improving the quality of a compressed image, wherein the depth model comprises a sampling module based on overlapping pixel rearrangement, a residual prediction module based on feature enhancement and a reconstruction module based on overlapping pixel rearrangement, and the depth model comprises the following specific steps:
(1.1) sampling an image input to the compressed image quality improvement model based on the overlapped pixel rearrangement operation;
(1.2) designing a residual error prediction module based on feature enhancement based on basic components such as a convolutional layer and a nonlinear layer and combining dense connection, an attention mechanism, a residual error structure and the like, extracting features and enhancing features from the sampled image in the step (1.1), and predicting residual error components;
(1.3) reconstructing the residual predicted in step (1.2) based on an overlapping pixel rearrangement operation opposite to step (1.1);
(1.4) based on global residual connection, overlapping the residual image reconstructed in the step (1.3) to an input image to obtain a prediction result of improving the quality of the compressed image;
step 2, constructing a training image set for training a compressed image quality enhancement depth model, wherein the training image set comprises a high-quality image set and compressed image sets corresponding to the high-quality image set one by one, and the specific steps are as follows:
(2.1) collecting a plurality of high-quality images to construct a high-quality image set;
(2.2) compressing and decompressing the high-quality image constructed in the step (2.1) based on an image compression method to obtain a compressed image set;
(2.3) combining the high-quality image set constructed in the step (2.1) and the compressed image set obtained in the step (2.2) to construct a training image set with one-to-one correspondence;
step 3, training the compressed image quality enhancement depth model designed in the step 1 based on the training image set which is constructed in the step 2 and comprises the compressed image set and the corresponding high-quality image set, and specifically comprises the following steps:
(3.1) establishing a loss function for measuring quality improvement errors;
(3.2) inputting the images in the compressed image set constructed in the step (2.2) into the compressed image quality improvement depth model designed in the step (1) for processing to obtain a quality improvement result;
(3.3) acquiring a high-quality image corresponding to the input compressed image used in the step (3.2) from the high-quality image set constructed in the step (2.1), and comparing the high-quality image with the quality improvement result predicted in the step (3.2), namely measuring the prediction error by using the loss function established in the step (3.1);
(3.4) updating parameters in the compressed image quality enhancement depth model by using a depth model learning optimization algorithm based on the prediction error calculated in the step (3.3);
(3.5) repeating the steps (3.2) to (3.4) until the prediction error is converged, and finishing the training of the compressed image quality enhancement depth model;
step 4, taking the compressed image for testing as input, and processing by using the compressed image quality improvement depth model trained in the step 3 to obtain a quality improvement result corresponding to the tested image;
and 5, evaluating the quality improvement result obtained in the step 4, or further analyzing and understanding the quality improvement result, such as edge detection and the like.
2. The method according to claim 1, wherein the compressed image quality improvement network based on sampling reconstruction and feature enhancement in step 1 has a residual network as a main structure, and mainly comprises a sampling module based on overlapping pixel rearrangement, a residual prediction module based on feature enhancement, and a reconstruction module based on overlapping pixel rearrangement, and is represented as
In which is shown
And IcRespectively representing the result of the quality enhancement processing of the compressed image and the input compressed image, Fds(. denotes a sampling module based on overlapping pixel rearrangement, Frp(. denotes a residual prediction module based on feature enhancement, Frc(. cndot.) denotes a reconstruction module based on overlapping pixel rearrangement.
3. The method according to claim 1, wherein the sampling operation based on overlapped pixel rearrangement in step (1.1) is expressed as
Wherein,
representing the input image of the overlapping pixel resample block,
is the result after sampling;
c＝1,…,d2and c ═ d (x)0-1)+y0；(x0,y0) Rearranging the sampled start coordinates for overlapping pixels, and x0＝1,…,d，y0＝1,…,d。
4. The method according to claim 1, wherein the residual prediction network based on feature enhancement in step (1.2) is mainly composed of KFA dense connection residual error module based on feature enhancement and a feature fusion module based on attention re-calibration, which are expressed as
Wherein,
network input image prediction from residual error based on convolution layer
The shallow feature of (A) extracted from (B)fe(. h) is a corresponding operation; f. offrb_kPredicting the output of the kth feature enhancement based densely connected residual module in the network for the residual, and having ffrb_k＝Ffrb_k(ffrb_k-1)，Ffrb_k() represents the operation corresponding to the kth densely connected residual module; fgafbAs a set of pairs
Performing a fused global feature fusion operation, which is realized by a feature fusion module based on attention re-calibration; frr(. cndot.) is a convolutional layer-based residual reconstruction operation.
5. The method according to claim 1, wherein the dense connection residual error module based on feature enhancement in step (1.2) uses the dense connection residual error module as a backbone network, and combines with PMMultiple multi-scale feature extraction moduleCarrying out characteristic enhancement on the blocks; kth feature enhancement based densely connected residual module F in residual prediction networkfrb_kCan be represented as
ffrb_k＝Ffrb_k(ffrb_k-1)
＝Flafb_k(ffrb_k-1,fk_mfb_1,…,fk_mfb_p,…,fk_mfb_PM)+ffrb_k-1
Wherein f isfrb_k-1And ffrb_kFor the input and output of the kth feature-based enhanced densely-connected residual module, Flafb_k(. for the kth feature enhancement based dense connected residual module
Performing local fusion operation, which is realized by a feature fusion module based on attention re-calibration; f. ofk_mfb_pAnd representing the output characteristics of the p-th multi-scale characteristic extraction module in the k-th dense connection residual module based on characteristic enhancement.
6. The method according to claim 1, wherein the input of the multi-scale feature extraction module in step (1.2) comprises the input f of the kth feature enhancement dense connection residual module for the pth multi-scale feature extraction module in the kth feature enhancement dense connection residual modulefrb_k-1And the output { f of the first p-1 multi-scale feature extraction modulesk_mfb_1,…,fk_mfb_p-1Output is fk_mfb_pIs integrally represented as
fk_mfb_p＝Ffdr2(Fmfe(Ffdr1(Fcct(ffrb_k-1,fk_mfb_1,…,fk_mfb_p-1))))+Ffdr1(Fcct(ffrb_k-1,fk_mfb_1,…,fk_mfb_p-1))
Wherein, Fcct(. for characteristic connection operations in the direction of the passage, Ffdr1For pressing features after channel connectionReduced convolution operation, Fmfe(. is a multi-scale feature extraction based on hole convolution, Ffdr2() is a convolution operation that compresses the multi-scale features; in order to obtain multi-scale information and expand the network receptive field, the multi-scale feature extraction is realized based on the combination of the hole convolutions with different step lengths, which is expressed as
fmfe＝Fmfe(ffdr1)
＝Fcct(Fdfe1(ffdr1),Fdfe2(ffdr1),Fdfe3(ffdr1),Fdfe4(ffdr1))
Wherein f ismfeDenotes from ffdr1Multi-scale features extracted from (1), ffdr1Is Ffdr1Output of (. cndot.) operation, Fdfe1(·)、Fdfe2(·)、Fdfe3(. and F)dfe4(. cndot.) is a hole convolution operation of different step sizes.
7. The method according to claim 1, wherein the feature fusion module based on attention re-calibration in step (1.2) mainly comprises attention-based feature re-calibration and feature dimensionality reduction, which is expressed as
fafb＝Fafb(f1kK)
＝Ffdrb(Fafrb(f1kK))
Wherein, Fafb(. represents a feature fusion operation based on attention retargeting, fafbAs a result of feature fusion, f1kK＝Fcct(f1,…,fk,…,fK) Representing a set of features to be fused f1,…,fk,…,fKConnection of Fcct(. for characteristic connection operations in the direction of the passage, Fafrb(. denotes attention-based feature recalibration, Ffdrb(. h) is a characteristic dimension reduction; feature recalibration F based on attentionafrb(. 2) establishing the dependency relationship among channels in the input features, which comprises absolute value change, global mean value and maximum value poolThe processes of transformation, feature compression and stretching, dimension normalization, recalibration and the like are expressed as
fafrb＝Fafrb(f1kK)
＝σ(Ffss(Fgap(Fabs(f1kK)))+Ffss(Fgmp(Fabs(f1kK))))·f1kK+f1kK
Wherein, Fafrb(. and f)afrbRespectively the function and corresponding output realized by the attention-based feature recalibration module, Fabs(. to be operations for calculating absolute values, Fgap(. is a global mean pooling, Fgmp(. is a global maximum pooling, Ffss(.) is the characteristic compression and stretching, σ (·) is the Sigmoid function; the feature compression and the feature stretching are realized based on full convolution layers positioned at two sides of a ReLU activation function, and the feature compression and the feature stretching are performed by s times respectively; after the coefficient is normalized by sigma (-), multiplying the coefficient by the input characteristic to realize recalibration; the feature dimension reduction module performs dimension reduction and fusion on the features after the re-calibration so as to reduce complexity and improve performance, and is expressed as
ffdrb＝Ffdrb(fafrb)
＝Fff(Ffdr3(fafrb))
Wherein, Ffdrb(. and f)fdrbRespectively representing a feature dimension reduction module and its output, comprising a convolution layer for reducing the dimension and a convolution layer for better fusing the information, respectively with Ffdr3(. o) well and Fff(. -) represents.
8. A method for improving the quality of a compressed image based on sampling reconstruction and feature enhancement according to claim 1, characterized in that the reconstruction operation based on overlapped pixel rearrangement in step (1.3); reconstruction based on overlapping pixel rearrangement is the inverse of overlapping pixel rearrangement sampling, which is denoted as
Wherein,
is the predicted residual, is the input to the reconstruction operation;
is the result of the reconstruction;
is composed of
The c-th feature channel of (1);
to represent
Accumulating the channels according to the corresponding original positions; ω is a coefficient that normalizes the accumulated sum.
9. The method according to claim 1, wherein the image analysis and understanding in combination with the compressed image quality improvement processing in step 5 is performed by firstly using the compressed image quality improvement processing as a preprocessor to improve the quality of the input compressed image so as to reduce the influence of compression noise on subsequent processing, and then using the compressed image quality improvement result as an input for image analysis and understanding.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011057427.7A CN113766250B (en) | 2020-09-29 | 2020-09-29 | Compressed image quality improving method based on sampling reconstruction and feature enhancement |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011057427.7A CN113766250B (en) | 2020-09-29 | 2020-09-29 | Compressed image quality improving method based on sampling reconstruction and feature enhancement |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113766250A true CN113766250A (en) | 2021-12-07 |
| CN113766250B CN113766250B (en) | 2022-05-27 |
Family
ID=78785786
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011057427.7A Active CN113766250B (en) | 2020-09-29 | 2020-09-29 | Compressed image quality improving method based on sampling reconstruction and feature enhancement |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113766250B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170200067A1 (en) * | 2016-01-08 | 2017-07-13 | Siemens Healthcare Gmbh | Deep Image-to-Image Network Learning for Medical Image Analysis |
| CN110120019A (en) * | 2019-04-26 | 2019-08-13 | 电子科技大学 | A kind of residual error neural network and image deblocking effect method based on feature enhancing |
| CN110415170A (en) * | 2019-06-24 | 2019-11-05 | 武汉大学 | A kind of image super-resolution method based on multiple dimensioned attention convolutional neural networks |
| CN111192200A (en) * | 2020-01-02 | 2020-05-22 | 南京邮电大学 | Image super-resolution reconstruction method based on fusion attention mechanism residual error network |
| US20200202587A1 (en) * | 2017-09-29 | 2020-06-25 | General Electric Company | Systems and methods for deep learning-based image reconstruction |
| CN111583112A (en) * | 2020-04-29 | 2020-08-25 | 华南理工大学 | Method, system, device and storage medium for video super-resolution |
-
2020
- 2020-09-29 CN CN202011057427.7A patent/CN113766250B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170200067A1 (en) * | 2016-01-08 | 2017-07-13 | Siemens Healthcare Gmbh | Deep Image-to-Image Network Learning for Medical Image Analysis |
| US20200202587A1 (en) * | 2017-09-29 | 2020-06-25 | General Electric Company | Systems and methods for deep learning-based image reconstruction |
| CN110120019A (en) * | 2019-04-26 | 2019-08-13 | 电子科技大学 | A kind of residual error neural network and image deblocking effect method based on feature enhancing |
| CN110415170A (en) * | 2019-06-24 | 2019-11-05 | 武汉大学 | A kind of image super-resolution method based on multiple dimensioned attention convolutional neural networks |
| CN111192200A (en) * | 2020-01-02 | 2020-05-22 | 南京邮电大学 | Image super-resolution reconstruction method based on fusion attention mechanism residual error network |
| CN111583112A (en) * | 2020-04-29 | 2020-08-25 | 华南理工大学 | Method, system, device and storage medium for video super-resolution |
Non-Patent Citations (1)
| Title |
|---|
| HONGGANG CHENL XIAOHAI HE ET AL: "Super-resolution of real-world rock microcomputed tomography images using cycle-consistent adversarial networks", 《PHYSICAL REVIEW E 101, 023305（2020）》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113766250B (en) | 2022-05-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110933429B (en) | Video compression sensing and reconstruction method and device based on deep neural network | |
| US8223837B2 (en) | Learning-based image compression | |
| CN110971901B (en) | Processing method, device and equipment of convolutional neural network and storage medium | |
| CN110827198B (en) | Multi-camera panoramic image construction method based on compressed sensing and super-resolution reconstruction | |
| CN113822147B (en) | Deep compression method for semantic tasks of collaborative machine | |
| CN109978772B (en) | Compressed image restoration method based on deep learning and double-domain complementation | |
| CN113192147B (en) | Method, system, storage medium, computer device and application for significance compression | |
| Wang et al. | Scalable facial image compression with deep feature reconstruction | |
| CN105590296B (en) | A kind of single-frame images Super-Resolution method based on doubledictionary study | |
| CN112218094A (en) | JPEG image decompression effect removing method based on DCT coefficient prediction | |
| US20150169632A1 (en) | Method and apparatus for image processing and computer readable medium | |
| CN113379858A (en) | Image compression method and device based on deep learning | |
| CN113066022A (en) | Video bit enhancement method based on efficient space-time information fusion | |
| CN109615576A (en) | The single-frame image super-resolution reconstruction method of base study is returned based on cascade | |
| CN113902647B (en) | Image deblurring method based on double closed-loop network | |
| Wu et al. | A generative adversarial network framework for JPEG anti-forensics | |
| Sekar et al. | Deep wavelet architecture for compressive sensing recovery | |
| CN113766250B (en) | Compressed image quality improving method based on sampling reconstruction and feature enhancement | |
| CN115345801B (en) | Image compression and filter removal method and system based on image denoising idea | |
| CN116896638A (en) | Data compression coding technology for transmission operation detection scene | |
| CN115512199A (en) | Image compression model based on graph attention and asymmetric convolution network | |
| CN115272113A (en) | Image deblurring method based on multi-scale frequency separation network | |
| CN111107360B (en) | Spectrum-space dimension combined hyperspectral image lossless compression method and system | |
| Tang et al. | LMENet: A lightweight multiscale efficient convolutional neural network for image denoising | |
| Zhen et al. | A deep learning based distributed compressive video sensing reconstruction algorithm for small reconnaissance uav |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |