CN106886992A - A kind of quality evaluating method of many exposure fused images of the colour based on saturation degree - Google Patents

Abstract

The invention discloses a method for evaluating the quality of color multi-exposure fusion images based on saturation, which can solve the problem of inaccurate evaluation of color multi-exposure fusion images; using multi-exposure images and their fusion images in the MEF database as training samples, The extraction methods based on saturation and wavelet coefficients are respectively used to obtain texture information, structure information and color information; the texture similarity, structure similarity and color similarity are calculated respectively; all the obtained similarities are used as feature values, combined with the given The fixed MOS value is input into the ELM machine learning machine for training; the texture similarity, structural similarity and color similarity obtained respectively from the multi-exposure fusion image and the corresponding generated fused image are input into the trained ELM machine learning machine , to get the evaluation result.

Description

A Quality Evaluation Method of Color Multi-exposure Fusion Image Based on Saturation

technical field

The invention belongs to the field of image quality evaluation methods, in particular to a saturation-based color multi-exposure fusion image quality evaluation method.

Background technique

The current ordinary display devices can display a brightness range that is far smaller than the real scene and the brightness range that the human eye can perceive, so it is impossible to display the real scene observed by the human eye on an ordinary device. The high dynamic range (High Dynamic Range, HDR) image display technology can display as much light and dark information of the real scene on common devices as possible, and conform to the real perception effect of the human eye as much as possible. With the rapid development of the high-definition digital industry, HDR display technology has become one of the current research hotspots in the digital field. Multi-exposure image fusion technology is a simple HDR display technology, which can directly fuse multiple image sequences with different exposures into an image that can be displayed on a common display device. With the development of multi-exposure fusion technology, more and more fusion algorithms have been proposed. However, different fusion algorithms have different fusion effects. In order to evaluate the advantages and disadvantages of different fusion algorithms, an evaluation algorithm that can evaluate the effect of multi-exposure fusion is urgently needed.

At present, the quality evaluation algorithms for fusion images are mainly divided into four categories: 1) methods based on image information, this type of method mainly uses the mutual information of images before and after fusion to evaluate the quality of the image after fusion, this type of method only considers the overall information of the image 2) Image feature-based methods, which mainly use edge features in the spatial domain and edge features in the wavelet domain to evaluate the quality of the fused image, which ignore the image 3) Based on the structure similarity method, this type of method is inspired by the SSIM algorithm, which calculates the structural similarity between the image before fusion and the image after fusion, which is similar to human vision, but now Most of the evaluation algorithms based on structural similarity are based on grayscale images, ignoring the color information of the image. In reality, the multi-exposure fusion images are all color images, and the color information changes after fusion, so there are existing evaluation algorithms. It is not suitable for color images; 4) Based on the method of human eye perception, this type of method mainly uses the salient information of the image to evaluate the image, but often ignores the background information of the image, resulting in the loss of information.

Contents of the invention

In view of this, the present invention provides a color multi-exposure fusion image quality evaluation method based on saturation, which improves the evaluation accuracy of the color multi-exposure fusion image.

The specific method of implementing this program is:

Step 1. Using the multi-exposure image and its fusion image in the multi-exposure image fusion database MEF as training samples, use the extraction method based on saturation and wavelet coefficients for the multi-exposure image and fusion image to obtain texture information, structure information and color information ; According to the texture information, structure information and color information of the images before and after fusion, the texture similarity, structure similarity and color similarity are calculated respectively; the texture information similarity, structure information similarity and color information similarity are used as eigenvalues, combined to give The given evaluation score is input to the extreme learning machine ELM for training;

The method of extracting the texture information is as follows: suppose that each image in the fused image or the multi-exposure image of the information to be extracted is an image _IQ ; the image _IQ is subjected to wavelet transformation, and the obtained image _IQ is divided into a low-frequency part, an intermediate-frequency part and a low-frequency part. The wavelet coefficient set I _q of the high frequency part=[LL LH HL HH]; Wherein, LL is the wavelet coefficient of the low frequency part, LH and HL are the wavelet coefficients of the intermediate frequency part, LH corresponds to the horizontal direction phase, and HL corresponds to the vertical direction phase, HH is the wavelet coefficient of the high frequency part; based on the principle that most of the texture information of the image is concentrated in the intermediate frequency and high frequency part, the wavelet coefficient set I _q '=[LH HL HH] of the intermediate frequency and high frequency part of the image I _Q is extracted; for When the image I _Q is a fused image, the extracted wavelet coefficient set I _q ′ is the texture information; for the image I _Q is a multi-exposure image, select the wavelet coefficient set I _q ′ corresponding to each image in the multi-exposure image and Take the maximum value of each coefficient to form Vmax=[max|LH|, max|HL|, max|HH|] as the texture information of the multi-exposure image;

The extraction method of the color information is: calculating the saturation SA of the image I _Q , Among them, R, G, B are three kinds of color information of red, green and blue in the color image, and μ is the average value of the three kinds of color information; for the case where the image I _Q is a fused image, the saturation value SA is used as the color information ; For the situation that image I _Q is a multi-exposure image, select the maximum value of the saturation value SA corresponding to each image in the multi-exposure image, as the color information of the multi-exposure image;

Step 2. When evaluating the quality of the multi-exposure fused image, a fusion algorithm is used to fuse the multi-exposure images to generate a fused image to be evaluated;

Step 3, the fused image to be evaluated and the pre-fused multi-exposure image corresponding to the fused image, using the extraction method based on saturation and wavelet coefficients, respectively extract texture information, structure information and color information;

Step 4. According to the texture information, structure information and color information of the images before and after fusion obtained in step 3, calculate texture similarity, structure similarity and color similarity respectively;

Step 5. Input the three similarities obtained in step 4 as feature values into the trained ELM machine learning machine to obtain the evaluation result of the fused image to be evaluated.

Preferably, the specific methods for calculating texture similarity, structural similarity and color similarity of images before and after fusion are:

Similarity is defined as follows: Among them, I ₁ and I ₂ are the parameters of the images before and after fusion, and C is a constant; when I ₁ and I ₂ are the texture information of the images before and after fusion respectively, then the obtained texture information of the images before and after fusion is substituted into the similarity s formula , to obtain the texture similarity TS; when I ₁ and I ₂ are the saturation of the image before and after fusion, the obtained saturation is substituted into the similarity s formula to obtain the saturation similarity SAS; using the structural similarity formula Obtain the structural similarity SS, where σ _xy is the covariance of the image structure before fusion and the image structure after fusion, σ _x is the standard deviation of the image structure before fusion, and σ _y is the standard deviation of the image structure after fusion.

Preferably, the extreme learning machine ELM uses an activation function radial basis function to train the input image features, wherein the number of nodes in the hidden layer is set to 21.

Preferably, the method for extracting the structural information of the fused image is: extracting the fused image using the structural similarity SSIM algorithm.

Preferably, the similarity defines a constant C=0.001 in the formula S.

Preferably, the constant C ₂ in the structural similarity formula SS is 0.001.

Beneficial effect:

This method solves the problem of inaccurate evaluation of color multi-exposure fusion images on the basis of comprehensive consideration of single pixel and local image structure information, image texture information, human perception characteristics, image color information and image background information. ,Specifically:

1. This method calculates the saturation similarity of the image before and after fusion, and uses it as a means to measure the color information of the fusion image. It can well evaluate the change of the color information of the image before and after fusion, and makes up for the color information in the evaluation of the fusion algorithm. Vacancies for quality reviews.

2. Because the image is affected by different exposure levels during the multi-exposure process, the texture preservation is different. This method uses wavelet transform to extract the texture information of images before and after fusion, and can evaluate the degree of preservation of texture information in the fusion process of the fusion algorithm on the basis of considering the background of the image.

3. This method uses the ELM machine learning machine. This method can not only clearly describe the relationship between texture similarity, structure similarity and saturation similarity, but also quickly and accurately obtain budget results. It has great potential in engineering applications. A lot of room for improvement.

4. On the basis of considering the color information of the image, single pixel and local image structure, this method uses the SSIM algorithm to calculate the structural similarity between the image before fusion and the image after fusion, which is similar to human vision, and can evaluate the fusion very well The degree of preservation of color information during the fusion process of the algorithm.

Description of drawings

Fig. 1 is the overall flowchart of the implementation of the present invention.

Fig. 2 is a flowchart for generating texture similarity in the present invention.

detailed description

The present invention will be described in detail below with reference to the accompanying drawings and examples.

The present invention provides a color multi-exposure fusion image quality evaluation method based on saturation, and the specific process is shown in FIG. 1 .

Step 1. Using the multi-exposure images and their fusion images in MEF (Multi-exposure Image Fusion Database) as training samples, the multi-exposure images and fusion images are extracted based on saturation and wavelet coefficients to obtain texture information, structure information and Color information; according to the texture information, structure information and color information of the images before and after fusion, the texture similarity, structure similarity and color similarity are calculated respectively; the texture information similarity, structure information similarity and color information similarity are used as feature values, Combined with the given evaluation score, that is, the average subjective score MOS value, input it into the extreme learning machine ELM for training;

Among them, the training data input in the extreme learning machine ELM is the MEF database (this database comes from K.Ma, K.Zeng and Z.Wang, "Perceptual Quality Assessment for Multi-Exposure ImageFusion," in IEEE Transactions on Image Processing, vol.24, no.11, pp.3345-3356, Nov.2015), the test data is the similarity information obtained in the previous step, and after learning the ELM machine learning algorithm, the corresponding evaluation results can be obtained; ELM is Proposed by the paper (G.-B.Huang, Q.-Y.Zhu, and C.-K.Siew, "Extremelearning machine: theory and applications," Neurocomputing, vol.70, pp.489-501, 2006) A fast machine learning method. There are two common activation functions in the algorithm: Sigmoid function and radial basis function (RBF). We use the former activation function to train the input features and get the results we want. The number of nodes in the hidden layer is set to 21; the test result of the extreme learning machine ELM training is the evaluation result.

The method of extracting the texture information is as follows: suppose that each image in the fused image or the multi-exposure image of the information to be extracted is an image _IQ ; the image _IQ is subjected to wavelet transformation, and the obtained image _IQ is divided into a low-frequency part, an intermediate-frequency part and a low-frequency part. The wavelet coefficient set I _q of the high frequency part =[LL LH HL HH];, wherein, LL is the wavelet coefficient of the low frequency part, LH and HL are the wavelet coefficients of the intermediate frequency part, LH corresponds to the phase in the horizontal direction, and HL corresponds to the phase in the vertical direction , HH is the wavelet coefficient of the high frequency part; based on the principle that most of the texture information of the image is concentrated in the intermediate frequency and high frequency part, the wavelet coefficient set I _q '=[LH HL HH] of the intermediate frequency and high frequency part of the image I _Q is extracted; For the case where the image I _Q is a fused image, the extracted wavelet coefficient set I _q ′ is the texture information; for the case where the image I _Q is a multi-exposure image, select the wavelet coefficient set I _q ′ corresponding to each image in the multi-exposure image And take the maximum value of each coefficient to form Vmax=[max|LH|, max|HL|, max|HH|], as the texture information of the multi-exposure image;

In order to extract as much texture information as possible, the present invention performs three-level wavelet decomposition on images before and after fusion, and the corresponding texture similarity values obtained by each level of wavelet decomposition are TS ₁ , TS ₂ , and TS ₃ .

The extraction mode of described structure information is: ask the standard deviation of the local area of image, can characterize image structure; For the situation that image I _Q is the image after fusion, directly ask for the standard deviation of image local area; For image I _Q is multiple In the case of exposed images, since there are multiple multi-exposure images, we use the method of extracting structural information of multi-exposure images to extract a structural information from multiple multi-exposure images before fusion, and use the SSIM (Structural Similarity) algorithm to combine multiple The fused image correspondingly extracts multiple structural information; this algorithm refers to the paper (K.Ma, K.Zeng and Z.Wang, "Perceptual Quality Assessment for Multi-Exposure Image Fusion," in IEEE Transactions on Image Processing, vol.24, no.11, pp.3345-3356, Nov.2015) in the structure fusion method to extract the structure diagram of the multi-exposure image, the structure diagram can be expressed as is the structure diagram to be requested, is the image structure diagram before normalization, ||·|| represents the modulus value; the image structure diagram in Represents the weight value of the kth image, s _k represents the pixel value of the kth image, K represents the number of multi-exposure images, and its principle is to perform weighted processing on multi-exposure multiple images _sk ; weight value in is the multi-exposure kth image pixel value, p value represents Structural strength of a local area; strength where the R value represents the continuity of the structure; structural continuity in is the multi-exposure kth image pixel value. According to the above method, the structure diagram of the multi-exposure image can be extracted. After obtaining the structure diagram of the multi-exposure image, calculate the local standard deviation of the structure diagram of the multi-exposure image as the structural information of the multi-exposure image; for the obtained structure diagram of the image before fusion and the image after fusion, we directly extract the local area Standard deviation, and covariance, with a local region size of 11×11. The extraction method of the fused image structure is as follows: the SSIM algorithm is used to extract the fused image.

The extraction method of the color information is: calculating the saturation SA of the image I _Q , Among them, R, G, B are three kinds of color information of red, green and blue in the color image, and μ is the average value of the three kinds of color information; for the case where the image I _Q is a fused image, the saturation value SA is used as the color information ; For the situation that image I _Q is a multi-exposure image, select the maximum value of the saturation value SA corresponding to each image in the multi-exposure image, as the color information of the multi-exposure image;

Similarity is defined as follows: Among them, I ₁ and I ₂ are the parameters of the images before and after fusion, and C is a constant; when I ₁ and I ₂ are the texture information of the images before and after fusion respectively, then the obtained texture information of the images before and after fusion is substituted into the similarity S formula , to obtain the texture similarity TS; when I ₁ and I ₂ are the saturation of the image before and after fusion, the obtained saturation is substituted into the similarity S formula to obtain the saturation similarity SAS; using the structural similarity formula Obtain the structural similarity SS, where σ _xy is the covariance of the image structure before fusion and the image structure after fusion, σ _x is the standard deviation of the image structure before fusion, and σ _y is the standard deviation of the image structure after fusion. The constant C=0.001 in the similarity definition formula S, and the constant C ₂ =0.001 in the structural similarity formula SS.

Step 2. When evaluating the quality of the multi-exposure fused image, a fusion algorithm is used to fuse the multi-exposure images to generate a fused image to be evaluated;

The way to generate the fused image from the multi-exposure images before fusion is as follows: using different fusion algorithms, multiple multi-exposure images before fusion are correspondingly generated into different fused images, there are at least three or more images, and the number of fused images corresponds to It is related to the fusion algorithm evaluated, and one fusion algorithm corresponds to one fused image.

Step 3, the fused image to be evaluated and the pre-fused multi-exposure image corresponding to the fused image, using the extraction method based on saturation and wavelet coefficients, respectively extract texture information, structure information and color information;

Step 4. According to the texture information, structure information and color information of the images before and after fusion obtained in step 3, calculate texture similarity, structure similarity and color similarity respectively;

Fig. 2 is the process of extracting the texture similarity of the fused image to be evaluated and the corresponding pre-fused image texture of the fused image. According to the extraction method of texture information: firstly, wavelet transform is performed on the images before and after fusion, and the Haar wavelet base is selected here; then the most abundant texture information of the image _Ic before fusion is extracted as V _max = [LH _max HL _max HH _max ], and then the fusion Texture information V _{F =} [LH _f HL _f HH _f ] of the subsequent image If.

Step 5. Input the three similarities obtained in step 4 as feature values into the trained ELM machine learning machine, and finally input the feature vector into the ELM machine learning as [TS ₁ TS ₂ TS3SS SAS], and obtain the to-be-evaluated Evaluation results of the fused images.

Since then, the quality evaluation of color multi-exposure fusion images has been completed.

To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the present invention. In summary, the above are only preferred embodiments of the present invention, and are not intended to limit the present invention. protection scope of the invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims (6)

1. a kind of quality assessment method based on saturation color multi-exposure fusion image, it is characterized in that, the method concrete steps comprise: Step 1. Using the multi-exposure image and its fusion image in the multi-exposure image fusion database MEF as training samples, use the extraction method based on saturation and wavelet coefficients for the multi-exposure image and fusion image to obtain texture information, structure information and color information ; According to the texture information, structure information and color information of the images before and after fusion, the texture similarity, structure similarity and color similarity are calculated respectively; the texture information similarity, structure information similarity and color information similarity are used as eigenvalues, combined to give The given evaluation score is input to the extreme learning machine ELM for training; The method of extracting the texture information is as follows: suppose that each image in the fused image or the multi-exposure image of the information to be extracted is an image _IQ ; the image _IQ is subjected to wavelet transformation, and the obtained image _IQ is divided into a low-frequency part, an intermediate-frequency part and a low-frequency part. The wavelet coefficient set I _q of the high frequency part=[LL LH HL HH]; Wherein, LL is the wavelet coefficient of the low frequency part, LH and HL are the wavelet coefficients of the intermediate frequency part, LH corresponds to the horizontal direction phase, and HL corresponds to the vertical direction phase, HH is the wavelet coefficient of the high frequency part; based on the principle that most of the texture information of the image is concentrated in the intermediate frequency and high frequency part, the wavelet coefficient set I _q '=[LH HL HH] of the intermediate frequency and high frequency part of the image I _Q is extracted; for When the image I _Q is a fused image, the extracted wavelet coefficient set I _q ′ is the texture information; for the image I _Q is a multi-exposure image, select the wavelet coefficient set I _q ′ corresponding to each image in the multi-exposure image and Take the maximum value of each coefficient to form Vmax=[max|LH|, max|HL|, max|HH|] as the texture information of the multi-exposure image; The extraction method of the color information is: calculating the saturation SA of the image I _Q , Among them, R, G, B are three kinds of color information of red, green and blue in the color image, and μ is the average value of the three kinds of color information; for the case where the image I _Q is a fused image, the saturation value SA is used as the color information ; For the situation that image I _Q is a multi-exposure image, select the maximum value of the saturation value SA corresponding to each image in the multi-exposure image, as the color information of the multi-exposure image; Step 2. When evaluating the quality of the multi-exposure fused image, a fusion algorithm is used to fuse the multi-exposure images to generate a fused image to be evaluated; Step 3, the fused image to be evaluated and the pre-fused multi-exposure image corresponding to the fused image, using the extraction method based on saturation and wavelet coefficients, respectively extract texture information, structure information and color information; Step 4. According to the texture information, structure information and color information of the images before and after fusion obtained in step 3, calculate texture similarity, structure similarity and color similarity respectively; Step 5. Input the three similarities obtained in step 4 as feature values into the trained ELM machine learning machine to obtain the evaluation result of the fused image to be evaluated. 2. as claimed in claim 1, a kind of quality evaluation method based on color multi-exposure fusion image of saturation, it is characterized in that, the concrete method of calculating texture similarity, structural similarity and color similarity of image before and after fusion is: Similarity is defined as follows: Among them, I ₁ and I ₂ are the parameters of the images before and after fusion, and C is a constant; when I ₁ and I ₂ are the texture information of the images before and after fusion respectively, then the obtained texture information of the images before and after fusion is substituted into the similarity S formula , to obtain the texture similarity TS; when I ₁ and I ₂ are the saturation of the image before and after fusion, the obtained saturation is substituted into the similarity S formula to obtain the saturation similarity SAS; using the structural similarity formula Obtain the structural similarity SS, where σ _xy is the covariance of the image structure before fusion and the image structure after fusion, σ _x is the standard deviation of the image structure before fusion, and σ _y is the standard deviation of the image structure after fusion. 3. as claimed in claim 1, a kind of quality evaluation method based on color multi-exposure fusion image of saturation, it is characterized in that, extreme learning machine ELM adopts activation function radial basis function to train the image feature of input, and wherein, hidden layer The number of nodes is set to 21. 4. a kind of quality evaluation method based on the color multi-exposure fusion image of saturation as claimed in claim 1, the extraction method of the image structure information after fusion is: extract the image after fusion and adopt structural similarity SSIM algorithm. 5. A method for evaluating the quality of a color multi-exposure fusion image based on saturation as claimed in claim 2, the constant C=0.001 in the similarity definition formula S. 6. A method for evaluating the quality of a color multi-exposure fusion image based on saturation as claimed in claim 2, wherein the constant C ₂ in the structural similarity formula SS is 0.001.