CN102903107B - Three-dimensional picture quality objective evaluation method based on feature fusion - Google Patents

Abstract

The invention discloses an objective evaluation method of stereoscopic image quality based on feature fusion, which first calculates the cyclopean diagram of the original undistorted stereoscopic image and the cyclopean diagram of the distorted stereoscopic image to be evaluated, and calculates the cyclopean diagram of the two stereoscopic images The mean and standard deviation of the pixel values of each pixel in the image are obtained to obtain the objective evaluation value of each pixel in the cyclopean image of the distorted stereo image to be evaluated, and then the saliency map and the two The distortion map between the cyclopean images, and the objective evaluation measurement value of each pixel in the cyclopean image of the distorted stereo image to be evaluated is fused to obtain the image quality objective evaluation prediction value of the distorted stereo image to be evaluated, The advantage is that the obtained cyclopean image can well simulate the binocular stereo fusion process, and the saliency image and the distortion image are used for fusion, which can effectively improve the correlation between the objective evaluation result and the subjective perception.

Description

An Objective Evaluation Method of Stereo Image Quality Based on Feature Fusion

technical field

The invention relates to an image quality evaluation method, in particular to an objective evaluation method for stereoscopic image quality based on feature fusion.

Background technique

With the rapid development of image coding technology and stereoscopic display technology, stereoscopic image technology has received more and more attention and applications, and has become a current research hotspot. Stereoscopic image technology utilizes the binocular parallax principle of the human eye. Both eyes independently receive left and right viewpoint images from the same scene, and form binocular parallax through brain fusion, so as to enjoy a stereoscopic image with a sense of depth and realism. Due to the impact of acquisition system, storage compression and transmission equipment, stereoscopic images will inevitably introduce a series of distortions. Compared with single-channel images, stereoscopic images need to ensure the image quality of two channels at the same time. very important meaning. However, there is currently no effective objective evaluation method to evaluate the stereoscopic image quality. Therefore, it is of great significance to establish an effective objective evaluation model for stereoscopic image quality.

The current objective evaluation method of stereoscopic image quality is to directly apply the planar image quality evaluation method to evaluate the stereoscopic image quality. However, the process of fusing the left and right viewpoint images of the stereoscopic image to produce a stereoscopic effect is not a simple process of superimposing the left and right viewpoint images. , it is difficult to use simple mathematical methods to express, therefore, how to effectively simulate binocular stereo fusion in the process of stereo image quality evaluation, how to extract effective feature information to fuse the evaluation results, so that the objective evaluation results are more in line with The human visual system is a problem that needs to be studied and solved in the process of objective quality evaluation of stereoscopic images.

Contents of the invention

The technical problem to be solved by the present invention is to provide an objective evaluation method for stereoscopic image quality based on feature fusion that can effectively improve the correlation between objective evaluation results and subjective perception.

The technical scheme adopted by the present invention to solve the above-mentioned technical problems is: an objective evaluation method of stereoscopic image quality based on feature fusion, which is characterized in that its processing process is as follows: first, according to the left viewpoint image of the original undistorted stereoscopic image and The even symmetric frequency response and the odd symmetric frequency response of each pixel in the right view point image at different scales and directions, and the disparity image between the left view point image and the right view point image of the original undistorted stereo image, to obtain the original The cyclopean image of the undistorted stereo image; the even and odd symmetric frequency responses of each pixel in different scales and directions according to the left view point image and the right view point image of the distorted stereo image to be evaluated, and the original The parallax image between the left view point image and the right view point image of the undistorted stereo image obtains the cyclopean image of the distorted stereo image to be evaluated; secondly, according to the average sum of the pixel values of each pixel in the two cyclopia images Standard deviation, obtain the objective evaluation metric value of each pixel in the cyclopean image of the distorted stereo image to be evaluated; again, obtain the corresponding saliency map according to the amplitude and phase of the cyclope image of the original undistorted stereo image; According to the amplitude and phase of the cyclopean image of the distorted stereo image to be evaluated, the corresponding saliency map is obtained; then, according to the two saliency maps and the distortion map between the two cyclopean images, the cyclopean image of the distorted stereo image to be evaluated The objective evaluation measurement value of each pixel in the fusion is carried out to obtain the image quality objective evaluation prediction value of the distorted stereoscopic image to be evaluated; finally, according to the above-mentioned processing process, a plurality of distorted stereoscopic images of different distortion types and different degrees of distortion are obtained The predictive value of the image quality objective evaluation.

The concrete steps of a kind of stereoscopic image quality objective evaluation method based on feature fusion of the present invention are:

①Let S _org be the original undistorted stereo image, let S _dis be the distorted stereo image to be evaluated, record the left viewpoint image of S _org as {L _org (x,y)}, and let the right viewpoint image of S _org The image is recorded as {R _org (x,y)}, the left view image of S _dis is recorded as {L _dis (x,y)}, and the right view image of S _dis is recorded as {R _dis (x,y)} , where (x, y) represents the coordinate position of the pixel in the left viewpoint image and the right viewpoint image, 1≤x≤W, 1≤y≤H, W represents the width of the left viewpoint image and the right viewpoint image, H represents the height of the left view point image and the right view point image, L _org (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {L _org (x, y)}, R _org (x, y) y) means the pixel value of the pixel point whose coordinate position is (x, y) in {R _org (x, y)}, and L _dis (x, y) means that the coordinate position in {L _dis (x, y)} is ( The pixel value of the pixel point of x, y), R _dis (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in {R _dis (x, y)};

②According to each pixel in {L _org (x,y)}, {R _org (x,y)}, {L _dis (x,y)}, {R _dis (x,y)} at different scales The even symmetric frequency response and odd symmetric frequency response in the and direction, correspondingly get {L _org (x,y)}, {R _org (x,y)}, {L _dis (x,y)}, {R _dis (x ,y)}, and then according to the amplitude of each pixel in {L _org (x,y)} and {R _org (x,y)} and {L _org (x,y) )} and {R _org (x,y)} between the pixel value of each pixel in the disparity image, calculate the Cyclops of S _org , denoted as {CM _org (x,y)}, and according to {L The amplitude of each pixel in _dis (x,y)} and {R _dis (x,y)} and the disparity image between {L _org (x,y)} and {R _org (x,y)} The pixel value of each pixel in , calculate the Cyclops of S _dis , recorded as {CM _dis (x, y)}, where CM _org (x, y) represents the coordinates in {CM _org (x, y)} The pixel value of the pixel point whose position is (x, y), CM _dis (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in {CM _dis (x, y)};

③According to the mean and standard deviation of the pixel values of each pixel in {CM _org (x,y)} and {CM _dis (x,y)}, calculate each of {CM _dis (x,y)} The objective evaluation measurement value of the pixel point, the objective evaluation measurement value of the pixel point whose coordinate position is (x, y) in {CM _dis (x, y)} is recorded as Q _image (x, y);

④ According to the amplitude and phase of {CM _org (x,y)}, calculate the saliency map of {CM _org (x,y)}, denoted as {SM _org (x,y)}, and according to {CM _dis (x, y)} amplitude and phase, calculate the saliency map of {CM _dis (x,y)}, denoted as {SM _dis (x,y)}, where, SM _org (x,y) means {SM _org (x, The pixel value of the pixel whose coordinate position is (x, y) in y)}, SM _dis (x, y) means the pixel of the pixel whose coordinate position is (x, y) in {SM _dis (x, y)} value;

⑤ Calculate the distortion map between {CM _org (x,y)} and {CM _dis (x,y)}, record it as {DM(x,y)}, and set the coordinate position in {DM(x,y)} Be that the pixel value of the pixel point of (x, y) is denoted as DM (x, y), DM (x, y)=(CM _org (x, y)-CM _dis (x, y)) ² ;

⑥ According to {SM _org (x,y)} and {SM _dis (x,y)} and {DM(x,y)}, the objective evaluation of each pixel in {CM _dis (x,y)} The measured values are fused to obtain the predicted value of the objective evaluation of S _dis image quality, denoted as Q, $Q={\left[\frac{\underset{(x,\mathrm{the\; y})\∈\mathrm{\Ω}}{\mathrm{\Σ}}{Q}_{\mathrm{image}}(x,\mathrm{the\; y})\×\mathrm{SM}(x,\mathrm{the\; y})}{\underset{(x,\mathrm{the\; y})\∈\mathrm{\Ω}}{\mathrm{\Σ}}\mathrm{SM}(x,\mathrm{the\; y})}\right]}^{\mathrm{\γ}}\×{\left[\frac{\underset{(x,\mathrm{the\; y})\∈\mathrm{\Ω}}{\mathrm{\Σ}}{Q}_{\mathrm{image}}(x,\mathrm{the\; y})\×\mathrm{DM}(x,\mathrm{the\; y})}{\underset{(x,\mathrm{the\; y})\∈\mathrm{\Ω}}{\mathrm{\Σ}}\mathrm{DM}(x,\mathrm{the\; y})}\right]}^{\mathrm{\β}},$ Among them, Ω represents the range of the pixel domain, SM(x, y) = max(SM _org (x, y), SM _dis (x, y)), max() is the maximum value function, γ and β are weight coefficients;

⑦Using n original undistorted stereoscopic images, establish a set of distorted stereoscopic images under different distortion types and different degrees of distortion. The average subjective score difference of each distorted stereoscopic image in the image set is recorded as DMOS, DMOS=100-MOS, where MOS represents the mean subjective score, DMOS∈[0,100], n≥1;

8. According to the operation of calculating the image quality objective evaluation prediction value of S _dis according to step ① to step ⑥, respectively calculate the image quality objective evaluation prediction value of each distorted stereoscopic image in the distorted stereoscopic image set.

The concrete process of described step 2. is:

②-1. Perform filtering on {L _org (x, y)} to obtain the even symmetric frequency response and odd symmetric frequency response of each pixel in {L _org (x, y)} in different scales and directions, The even symmetric frequency response of the pixel at the coordinate position (x, y) in {L _org (x, y)} in different scales and directions is recorded as e _{α, θ} (x, y), and {L _org (x , y)}, the odd symmetric frequency response of the pixel at the coordinate position (x, y) in different scales and directions is denoted as o _{α, θ} (x, y), where α represents the scale of the filter used for filtering Factor, 1≤α≤4, θ indicates the direction factor of the filter used for filtering, 1≤θ≤4;

②-2. According to the even symmetric frequency response and odd symmetric frequency response of each pixel in {L _org (x,y)} in different scales and directions, calculate each pixel in {L _org (x,y)} The amplitude of the pixel point, the amplitude of the pixel point whose coordinate position is (x, y) in {L _org (x, y)} is recorded as ${\mathrm{GE}}_{\mathrm{org}}^L(x,\mathrm{the\; y})=\underset{\mathrm{\θ}=1}{\overset{4}{\mathrm{\Σ}}}\underset{\mathrm{\α}=1}{\overset{4}{\mathrm{\Σ}}}\sqrt{e_{\mathrm{\α},\mathrm{\θ}}{(x,\mathrm{the\; y})}^2+o_{\mathrm{\α},\mathrm{\θ}}{(x,\mathrm{the\; y})}^2};$

②-3. Obtain the amplitude of each pixel in {L _org (x,y)} according to step ②-1 to step ②-2, and obtain {R _org (x,y)} in the same way, The amplitude of each pixel in {L _dis (x, y)} and {R _dis (x, y)}, the pixel whose coordinate position is (x, y) in {R _org (x, y)} The amplitude of The amplitude of the pixel at the coordinate position (x, y) in {L _dis (x, y)} is recorded as Record the amplitude of the pixel at the coordinate position (x, y) in {R _dis (x, y)} as

②-4. Calculate the parallax image between {L _org (x, y)} and {R _org (x, y)} by block matching method, denoted as in, express The pixel value of the pixel point whose middle coordinate position is (x, y);

② _{- 5} . According to the amplitude and The pixel value of each pixel in , calculate the Cyclops image of S _org , which is recorded as {CM _org (x, y)}, and the pixel whose coordinate position is (x, y) in {CM _org (x, y)} The pixel value of a point is denoted as CM _org (x,y), ${\mathrm{CM}}_{\mathrm{org}}(x,\mathrm{the\; y})=\frac{{\mathrm{GE}}_{\mathrm{org}}^L(x,\mathrm{the\; y})\×L_{\mathrm{org}}(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{org}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})\×R_{\mathrm{org}}(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})}{{\mathrm{GE}}_{\mathrm{org}}^L(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{org}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})},$ in, Indicates that the coordinate position in {R _org (x,y)} is The amplitude of the pixel point, Indicates that the coordinate position in {R _org (x,y)} is The pixel value of the pixel point;

_② -6 _. According to the amplitude and The pixel value of each pixel in , calculate the Cyclops image of S _dis , which is recorded as {CM _dis (x, y)}, and the pixel whose coordinate position is (x, y) in {CM _dis (x, y)} The pixel value of the point is recorded as CM _dis (x,y), ${\mathrm{CM}}_{\mathrm{dis}}(x,\mathrm{the\; y})=\frac{{\mathrm{GE}}_{\mathrm{dis}}^L(x,\mathrm{the\; y})\×L_{\mathrm{dis}}(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{dis}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})\×R_{\mathrm{dis}}(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})}{{\mathrm{GE}}_{\mathrm{dis}}^L(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{dis}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})},$ in, Indicates that the coordinate position in {R _dis (x,y)} is The amplitude of the pixel point, Indicates that the coordinate position in {R _dis (x,y)} is The pixel value of the pixel.

The filter used for filtering {L _org (x, y)} in the step ②-1 is a log-Garbor filter.

The concrete process of described step 3. is:

③-1. Calculate the mean and standard deviation of the pixel values of each pixel in {CM _org (x, y)} and {CM _dis (x, y)}, and put {CM _org (x, y)} in The mean and standard deviation of the pixel values at the coordinate position (x ₁ ,y ₁ ) are recorded as μ _org (x ₁ ,y ₁ ) and σ _org (x ₁ ,y ₁ ) respectively, and {CM _dis (x ,y)}, the mean and standard deviation of the pixel values of the pixel at the coordinate position (x ₁ ,y ₁ ) are recorded as μ _dis (x ₁ ,y ₁ ) and σ _dis (x ₁ ,y ₁ ), respectively,

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; org</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\sqrt{\frac{\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; CM</span>}}_{\mathrm{<span\; class="notranslate">\; org</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; org</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; m</span>}}}<span\; class="notranslate">\; ,</span>$

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; dis</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; CM</span>}}_{\mathrm{<span\; class="notranslate">\; dis</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; ,</span>$

${\mathrm{\&sigma;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1)=\sqrt{\frac{\underset{(x_1,{\mathrm{the\; y}}_1)\&Element;N(x_1,{\mathrm{the\; y}}_1)}{\mathrm{\&Sigma;}}{({\mathrm{CM}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1)-{\mathrm{\&mu;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1))}^2}{m}},$ Among them, 1≤x ₁ ≤W, 1≤y ₁ ≤H, N(x ₁ , y ₁ ) represents an 8×8 neighborhood window centered on the pixel at the coordinate position (x ₁ , y ₁ ), M Indicates the number of pixels within N(x ₁ ,y ₁ ), CM _org (x ₁ ,y ₁ ) indicates the pixel at the coordinate position (x ₁ ,y ₁ ) in {CM _org (x,y)} The pixel value of , CM _dis (x ₁ , y ₁ ) represents the pixel value of the pixel whose coordinate position is (x ₁ , y ₁ ) in {CM _dis (x, y)};

③-2. According to the mean and standard deviation of the pixel values of each pixel in {CM _org (x, y)} and {CM _dis (x, y)}, calculate {CM _dis (x, y)} The objective evaluation metric value of each pixel in {CM _dis (x,y)}, the objective evaluation metric value of the pixel whose coordinate position is (x ₁ ,y ₁ ) in {CM dis (x,y)} is recorded as Q _image (x ₁ ,y ₁ ), $Q_{\mathrm{image}}(x_1,{\mathrm{the\; y}}_1)=\frac{4\&times;({\mathrm{\&mu;}}_{\mathrm{org}}(x_1,{\mathrm{the\; y}}_1)\&times;{\mathrm{\&mu;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1))\&times;({\mathrm{\&sigma;}}_{\mathrm{org}}(x_1,{\mathrm{the\; y}}_1)\&times;{\mathrm{\&sigma;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1))+C}{({\mathrm{\&mu;}}_{\mathrm{org}}{(x_1,{\mathrm{the\; y}}_1)}^2+{\mathrm{\&mu;}}_{\mathrm{dis}}{(x_1,{\mathrm{the\; y}}_1)}^2)\&times;({\mathrm{\&sigma;}}_{\mathrm{org}}{(x_1,{\mathrm{the\; y}}_1)}^2+{\mathrm{\&sigma;}}_{\mathrm{dis}}{(x_1,{\mathrm{the\; y}}_1)}^2)+C},$ Among them, C is the control parameter.

The concrete process of described step 4. is:

④-1. Perform discrete Fourier transform on {CM _org (x,y)} to obtain the amplitude and phase of {CM _org (x,y)}, which are recorded as {M _org (u,v)} and {A _org (u,v)}, where u represents the amplitude or phase width of the transform domain, v represents the amplitude or phase height of the transform domain, 1≤u≤W, 1≤v≤H, M _org (u,v) Indicates the amplitude value of the pixel point whose coordinate position is (u,v) in {M _org (u,v)}, and A _org (u,v) indicates that the coordinate position in {A _org (u,v)} is (u, The phase value of the pixel point of v);

④-2. Calculate the amplitude of the high-frequency component of {M _org (u,v)}, record it as {R _org (u,v)}, and set the coordinate position in {R _org (u,v)} as (u, v) The amplitude value of the high-frequency component of the pixel point is recorded as R _org (u,v), R _org (u,v)=log(M _org (u,v))-h _m (u,v)*log (M _org (u,v)), where log() is a logarithmic function based on e, e=2.718281828, "*" is the convolution operation symbol, h _m (u,v) means m×m mean filtering;

④-3. Perform inverse discrete Fourier transform according to {R _org (u, v)} and {A _org (u, v)}, and use the obtained inverse transformed image as the saliency map of {CM _org (x, y)}, Recorded as {SM _org (x, y)}, where, SM _org (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {SM _org (x, y)};

④-4. Follow steps ④-1 to ④-3 to obtain the saliency map of {CM _org (x, y)}, and obtain the saliency map of {CM _dis (x, y)} in the same way, denoted as {SM _dis (x, y)}, wherein, SM _dis (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {SM _dis (x, y)}.

Compared with the prior art, the present invention has the advantages of:

1) The method of the present invention calculates the cyclopean diagram of the original undistorted stereoscopic image and the monocular diagram of the distorted stereoscopic image to be evaluated respectively, and directly evaluates the monocular diagram of the distorted stereoscopic image, so that the binocular The stereo fusion process is simulated, avoiding the process of linearly weighting the objective evaluation metrics of the left-viewpoint image and the right-viewpoint image.

2) The method of the present invention calculates the saliency map of the cyclopean diagram of the original undistorted stereoscopic image and the cyclopean diagram of the distorted stereoscopic image to be evaluated and the distortion diagram between the two cyclopean diagrams, and the distorted stereoscopic image to be evaluated The fusion of the objective evaluation measurement value of each pixel in the Cyclops image can make the evaluation result more in line with the human visual system, thus effectively improving the correlation between the objective evaluation result and subjective perception.

Description of drawings

Fig. 1 is the overall realization block diagram of the inventive method;

Fig. 2 a is the left viewpoint image of Akko (size is 640 * 480) stereoscopic image;

Fig. 2b is the right viewpoint image of Akko (size is 640 * 480) stereoscopic image;

Fig. 3 a is the left viewpoint image of Altmoabit (size is 1024 * 768) stereoscopic image;

Fig. 3b is the right viewpoint image of Altmoabit (size is 1024 * 768) stereoscopic image;

Fig. 4 a is the left viewpoint image of the stereoscopic image of Balloons (size is 1024 * 768);

Fig. 4b is the right viewpoint image of the stereoscopic image of Balloons (size is 1024 * 768);

Fig. 5 a is the left viewpoint image of the stereoscopic image of Doorflower (size is 1024 * 768);

Fig. 5b is the right viewpoint image of the stereoscopic image of Doorflower (size is 1024 * 768);

Fig. 6 a is the left viewpoint image of Kendo (size is 1024 * 768) stereoscopic image;

Fig. 6b is the right viewpoint image of the stereoscopic image of Kendo (size is 1024 * 768);

Fig. 7a is the left view point image of LeaveLaptop (size is 1024 * 768) stereoscopic image;

Fig. 7b is the right viewpoint image of LeaveLaptop (size is 1024 * 768) stereoscopic image;

Fig. 8a is the left viewpoint image of the stereoscopic image of Lovebierd1 (size is 1024 * 768);

Fig. 8b is the right viewpoint image of the stereoscopic image of Lovebierd1 (size is 1024 * 768);

Fig. 9a is the left viewpoint image of the stereoscopic image of Newspaper (the size is 1024 * 768);

Fig. 9b is the right viewpoint image of the stereoscopic image of Newspaper (the size is 1024×768);

Fig. 10a is the left viewpoint image of Puppy (size is 720 * 480) stereoscopic image;

Fig. 10b is the right viewpoint image of Puppy (size is 720 * 480) stereoscopic image;

Fig. 11a is the left viewpoint image of Soccer2 (size is 720 * 480) stereoscopic image;

Fig. 11b is the right viewpoint image of Soccer2 (size is 720 * 480) stereoscopic image;

Fig. 12a is the left viewpoint image of the stereoscopic image of Horse (size is 720 * 480);

Fig. 12b is the right viewpoint image of the stereoscopic image of Horse (the size is 720×480);

Fig. 13a is the left viewpoint image of Xmas (size is 640 * 480) stereoscopic image;

Fig. 13b is the right viewpoint image of Xmas (size is 640 * 480) stereoscopic image;

FIG. 14 is a scatter diagram of the difference between the predicted image quality objective evaluation value and the average subjective evaluation value of each distorted stereo image in the distorted stereo image set.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments.

A kind of stereoscopic image quality objective evaluation method based on feature fusion proposed by the present invention, its overall implementation block diagram is shown in Figure 1, and its processing process is: first, according to the left viewpoint image and the right viewpoint image of the original undistorted stereoscopic image The even symmetric frequency response and odd symmetric frequency response of each pixel in different scales and directions, and the disparity image between the left view point image and the right view point image of the original undistorted stereo image, to obtain the original undistorted stereoscopic image Cyclopsogram of the stereo image; according to the even and odd symmetry frequency responses of each pixel in the left view image and the right view image of the distorted stereo image to be evaluated at different scales and directions, and the original undistorted The disparity image between the left view point image and the right view point image of the stereo image, obtain the monocular image of the distorted stereo image to be evaluated; secondly, according to the mean and the standard deviation of the pixel values of each pixel in the two cyclopia images, Obtain the objective evaluation metric value of each pixel in the cyclopean image of the distorted stereo image to be evaluated; again, obtain the corresponding saliency map according to the amplitude and phase of the cyclopean image of the original undistorted stereo image; Then, according to the two saliency maps and the distortion map between the two cyclopean maps, each element in the cyclopean map of the distorted stereo image to be evaluated The objective evaluation measurement value of each pixel is fused to obtain the image quality objective evaluation prediction value of the distorted stereoscopic image to be evaluated; finally, the image quality of the distorted stereoscopic image of multiple different distortion types and different degrees of distortion is obtained according to the above-mentioned process. Objectively evaluate the predicted value.

The inventive method specifically comprises the following steps:

①Let S _org be the original undistorted stereo image, let S _dis be the distorted stereo image to be evaluated, record the left viewpoint image of S _org as {L _org (x,y)}, and let the right viewpoint image of S _org The image is recorded as {R _org (x,y)}, the left view image of S _dis is recorded as {L _dis (x,y)}, and the right view image of S _dis is recorded as {R _dis (x,y)} , where (x, y) represents the coordinate position of the pixel in the left viewpoint image and the right viewpoint image, 1≤x≤W, 1≤y≤H, W represents the width of the left viewpoint image and the right viewpoint image, H represents the height of the left view point image and the right view point image, L _org (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {L _org (x, y)}, R _org (x, y) y) means the pixel value of the pixel point whose coordinate position is (x, y) in {R _org (x, y)}, and L _dis (x, y) means that the coordinate position in {L _dis (x, y)} is ( x, y), and R _dis (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {R _dis (x, y)}.

②According to each pixel in {L _org (x,y)}, {R _org (x,y)}, {L _dis (x,y)}, {R _dis (x,y)} at different scales The even symmetric frequency response and odd symmetric frequency response in the and direction, correspondingly get {L _org (x,y)}, {R _org (x,y)}, {L _dis (x,y)}, {R _dis (x ,y)}, and then according to the amplitude of each pixel in {L _org (x,y)} and {R _org (x,y)} and {L _org (x,y) )} and {R _org (x,y)} the pixel value of each pixel in the disparity image, calculate the cyclopean map of S _org , denoted as {CM _org (x,y)}, And according to the amplitude of each pixel in {L _dis (x,y)} and {R _dis (x,y)} and the relationship between {L _org (x,y)} and {R _org (x,y)} The pixel value of each pixel in the parallax image between, calculate the Cyclops of S _dis , denoted as {CM _dis (x, y)}, where, CM _org (x, y) means {CM _org (x, y )}, the pixel value of the pixel whose coordinate position is (x, y), CM _dis (x, y) means the pixel value of the pixel whose coordinate position is (x, y) in {CM _dis (x, y)} .

In this embodiment, the specific process of step ② is:

②-1. Perform filtering on {L _org (x, y)} to obtain the even symmetric frequency response and odd symmetric frequency response of each pixel in {L _org (x, y)} in different scales and directions, The even symmetric frequency response of the pixel at the coordinate position (x, y) in {L _org (x, y)} in different scales and directions is recorded as e _{α, θ} (x, y), and {L _org (x , y)}, the odd symmetric frequency response of the pixel at the coordinate position (x, y) in different scales and directions is denoted as o _{α, θ} (x, y), where α represents the scale of the filter used for filtering Factor, 1≤α≤4, θ indicates the direction factor of the filter used for filtering, 1≤θ≤4.

Here, the filter used for filtering {L _org (x, y)} is a log-Garbor filter.

②-2. According to the even symmetric frequency response and odd symmetric frequency response of each pixel in {L _org (x,y)} in different scales and directions, calculate each pixel in {L _org (x,y)} The amplitude of the pixel point, the amplitude of the pixel point whose coordinate position is (x, y) in {L _org (x, y)} is recorded as ${\mathrm{GE}}_{\mathrm{org}}^L(x,\mathrm{the\; y})=\underset{\mathrm{\&theta;}=1}{\overset{4}{\mathrm{\&Sigma;}}}\underset{\mathrm{\&alpha;}=1}{\overset{4}{\mathrm{\&Sigma;}}}\sqrt{e_{\mathrm{\&alpha;},\mathrm{\&theta;}}{(x,\mathrm{the\; y})}^2+o_{\mathrm{\&alpha;},\mathrm{\&theta;}}{(x,\mathrm{the\; y})}^2}.$

②-3. Obtain the amplitude of each pixel in {L _org (x,y)} according to step ②-1 to step ②-2, and obtain {R _org (x,y)} in the same way, The amplitude of each pixel in {L _dis (x, y)} and {R _dis (x, y)}, the pixel whose coordinate position is (x, y) in {R _org (x, y)} The amplitude of The amplitude of the pixel at the coordinate position (x, y) in {L _dis (x, y)} is recorded as Record the amplitude of the pixel at the coordinate position (x, y) in {R _dis (x, y)} as For example, the operation process of obtaining the amplitude of each pixel in {L _dis (x, y)} is: 1) Filter {L _dis (x, y)} to obtain {L _dis (x, y)} The even symmetric frequency response and odd symmetric frequency response of each pixel in different scales and directions, and the pixel with coordinate position (x, y) in {L _dis (x,y)} in different scales and directions The even symmetric frequency response is recorded as e _α,θ '(x,y), and the odd symmetric frequency response of the pixel at the coordinate position (x,y) in {L _dis (x,y)} in different scales and directions is recorded as is o _α,θ '(x,y), where α represents the scale factor of the filter used in filtering, 1≤α≤4, θ represents the direction factor of the filter used in filtering, 1≤θ≤4; 2) According to the even symmetric frequency response and odd symmetric frequency response of each pixel in {L _dis (x, y)} in different scales and directions, calculate each pixel in {L _dis (x, y)} The amplitude of the pixel point whose coordinate position is (x, y) in {L _dis (x, y)} is recorded as ${\mathrm{GE}}_{\mathrm{dis}}^L(x,\mathrm{the\; y})=\underset{\mathrm{\&theta;}=1}{\overset{4}{\mathrm{\&Sigma;}}}\underset{\mathrm{\&alpha;}=1}{\overset{4}{\mathrm{\&Sigma;}}}\sqrt{{e_{\mathrm{\&alpha;},\mathrm{\&theta;}}}^{\&prime;}{(x,\mathrm{the\; y})}^2+{o_{\mathrm{\&alpha;},\mathrm{\&theta;}}}^{\&prime;}{(x,\mathrm{the\; y})}^2}.$

②-4. Calculate the parallax image between {L _org (x, y)} and {R _org (x, y)} by block matching method, denoted as in, express The pixel value of the pixel whose middle coordinate position is (x, y).

②- ₅ . According to the _amplitude and The pixel value of each pixel in , calculate the Cyclops image of S _org , which is recorded as {CM _org (x, y)}, and the pixel whose coordinate position is (x, y) in {CM _org (x, y)} The pixel value of a point is denoted as CM _org (x,y), ${\mathrm{CM}}_{\mathrm{org}}(x,\mathrm{the\; y})=\frac{{\mathrm{GE}}_{\mathrm{org}}^L(x,\mathrm{the\; y})\&times;L_{\mathrm{org}}(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{org}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})\&times;R_{\mathrm{org}}(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})}{{\mathrm{GE}}_{\mathrm{org}}^L(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{org}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})},$ in, Indicates that the coordinate position in {R _org (x,y)} is The amplitude of the pixel point, Indicates that the coordinate position in {R _org (x,y)} is The pixel value of the pixel.

_② -6 _. According to the amplitude and The pixel value of each pixel in , calculate the Cyclops image of S _dis , which is recorded as {CM _dis (x, y)}, and the pixel whose coordinate position is (x, y) in {CM _dis (x, y)} The pixel value of the point is recorded as CM _dis (x,y), ${\mathrm{CM}}_{\mathrm{dis}}(x,\mathrm{the\; y})=\frac{{\mathrm{GE}}_{\mathrm{dis}}^L(x,\mathrm{the\; y})\&times;L_{\mathrm{dis}}(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{dis}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})\&times;R_{\mathrm{dis}}(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})}{{\mathrm{GE}}_{\mathrm{dis}}^L(x,\mathrm{the\; y})+{\mathrm{GE}}_{\mathrm{dis}}^R(x-d_{\mathrm{org}}^L(x,\mathrm{the\; y}),\mathrm{the\; y})},$ in, Indicates that the coordinate position in {R _dis (x,y)} is The amplitude of the pixel point, Indicates that the coordinate position in {R _dis (x,y)} is The pixel value of the pixel.

③According to the mean and standard deviation of the pixel values of each pixel in {CM _org (x,y)} and {CM _dis (x,y)}, calculate each of {CM _dis (x,y)} The objective evaluation measurement value of the pixel point, the objective evaluation measurement value of the pixel point whose coordinate position is (x, y) in {CM _dis (x, y)} is recorded as Q _image (x, y), and {CM _dis ( The objective evaluation metric values of all pixels in x, y)} are expressed as {Q _image (x, y)} in a set.

In the present embodiment, the specific process of step 3. is:

③-1. Calculate the mean and standard deviation of the pixel values of each pixel in {CM _org (x, y)} and {CM _dis (x, y)}, and put {CM _org (x, y)} in The mean and standard deviation of the pixel values at the coordinate position (x ₁ ,y ₁ ) are recorded as μ _org (x ₁ ,y ₁ ) and σ _org (x ₁ ,y ₁ ) respectively, and {CM _dis (x ,y)}, the mean and standard deviation of the pixel values of the pixel at the coordinate position (x ₁ ,y ₁ ) are recorded as μ _dis (x ₁ ,y ₁ ) and σ _dis (x ₁ ,y ₁ ), respectively,

${\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; org</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\sqrt{\frac{\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; CM</span>}}_{\mathrm{<span\; class="notranslate">\; org</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; org</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; m</span>}}}<span\; class="notranslate">\; ,</span>$

${\mathrm{<span\; class="notranslate">\; \&mu;</span>}}_{\mathrm{<span\; class="notranslate">\; dis</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; CM</span>}}_{\mathrm{<span\; class="notranslate">\; dis</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; ,</span>$

${\mathrm{\&sigma;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1)=\sqrt{\frac{\underset{(x_1,{\mathrm{the\; y}}_1)\&Element;N(x_1,{\mathrm{the\; y}}_1)}{\mathrm{\&Sigma;}}{({\mathrm{CM}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1)-{\mathrm{\&mu;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1))}^2}{m}},$ Among them, 1≤x ₁ ≤W, 1≤y ₁ ≤H, N(x ₁ , y ₁ ) represents an 8×8 neighborhood window centered on the pixel at the coordinate position (x ₁ , y ₁ ), M Indicates the number of pixels within N(x ₁ ,y ₁ ), CM _org (x ₁ ,y ₁ ) indicates the pixel at the coordinate position (x ₁ ,y ₁ ) in {CM _org (x,y)} The pixel value of CM _dis (x ₁ ,y ₁ ) represents the pixel value of the pixel whose coordinate position is (x ₁ ,y ₁ ) in {CM _dis (x,y)}.

③-2. According to the mean and standard deviation of the pixel values of each pixel in {CM _org (x, y)} and {CM _dis (x, y)}, calculate {CM _dis (x, y)} The objective evaluation metric value of each pixel in {CM _dis (x,y)}, the objective evaluation metric value of the pixel whose coordinate position is (x ₁ ,y ₁ ) in {CM dis (x,y)} is recorded as Q _image (x ₁ ,y ₁ ), $Q_{\mathrm{image}}(x_1,{\mathrm{the\; y}}_1)=\frac{4\&times;({\mathrm{\&mu;}}_{\mathrm{org}}(x_1,{\mathrm{the\; y}}_1)\&times;{\mathrm{\&mu;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1))\&times;({\mathrm{\&sigma;}}_{\mathrm{org}}(x_1,{\mathrm{the\; y}}_1)\&times;{\mathrm{\&sigma;}}_{\mathrm{dis}}(x_1,{\mathrm{the\; y}}_1))+C}{({\mathrm{\&mu;}}_{\mathrm{org}}{(x_1,{\mathrm{the\; y}}_1)}^2+{\mathrm{\&mu;}}_{\mathrm{dis}}{(x_1,{\mathrm{the\; y}}_1)}^2)\&times;({\mathrm{\&sigma;}}_{\mathrm{org}}{(x_1,{\mathrm{the\; y}}_1)}^2+{\mathrm{\&sigma;}}_{\mathrm{dis}}{(x_1,{\mathrm{the\; y}}_1)}^2)+C},$ Wherein, C is a control parameter, and in this embodiment, C=0.01.

④ Calculate the saliency map of {CM _org (x, y)} according to the spectral redundancy characteristics of {CM _org (x, y)}, that is, according to the amplitude and phase of {CM _org (x, y)} , denoted as {SM _org (x,y)}, and according to the spectral redundancy characteristics of {CM _dis (x,y)}, that is, according to the amplitude and phase of {CM _dis (x,y)}, calculate {CM _dis The saliency map of (x,y)} is recorded as {SM _dis (x,y)}, where SM _org (x,y) means that the coordinate position in {SM _org (x,y)} is (x,y) The pixel value of the pixel point of SM _dis (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in {SM _dis (x, y)}.

In the present embodiment, the specific process of step ④ is:

④-1. Perform discrete Fourier transform on {CM _org (x,y)} to obtain the amplitude and phase of {CM _org (x,y)}, which are recorded as {M _org (u,v)} and {A _org (u,v)}, where u represents the amplitude or phase width of the transform domain, v represents the amplitude or phase height of the transform domain, 1≤u≤W, 1≤v≤H, M _org (u,v) Indicates the amplitude value of the pixel point whose coordinate position is (u,v) in {M _org (u,v)}, and A _org (u,v) indicates that the coordinate position in {A _org (u,v)} is (u, v) The phase value of the pixel point.

④-2. Calculate the amplitude of the high-frequency component of {M _org (u,v)}, record it as {R _org (u,v)}, and set the coordinate position in {R _org (u,v)} as (u, v) The amplitude value of the high-frequency component of the pixel point is recorded as R _org (u,v), R _org (u,v)=log(M _org (u,v))-h _m (u,v)*log (M _org (u,v)), where log() is a logarithmic function based on e, e=2.718281828, "*" is the convolution operation symbol, h _m (u,v) means m×m For mean filtering, in this embodiment, m=3.

④-3. Perform inverse discrete Fourier transform according to {R _org (u, v)} and {A _org (u, v)}, and use the obtained inverse transformed image as the saliency map of {CM _org (x, y)}, It is recorded as {SM _org (x, y)}, where SM _org (x, y) represents the pixel value of the pixel at the coordinate position (x, y) in {SM _org (x, y)}.

④-4. Follow steps ④-1 to ④-3 to obtain the saliency map of {CM _org (x, y)}, and obtain the saliency map of {CM _dis (x, y)} in the same way, denoted as {SM _dis (x, y)}, wherein, SM _dis (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {SM _dis (x, y)}.

⑤ Calculate the distortion map between {CM _org (x,y)} and {CM _dis (x,y)}, denoted as {DM(x,y)}, and {DM(x,y) }, the pixel value of the pixel whose coordinate position is (x, y) is recorded as DM (x, y), DM (x, y) = (CM _org (x, y)-CM _dis (x, y)) ² .

⑥ According to {SM _org (x,y)} and {SM _dis (x,y)} and {DM(x,y)}, the objective evaluation of each pixel in {CM _dis (x,y)} The measured values are fused to obtain the predicted value of the objective evaluation of S _dis image quality, denoted as Q, $Q={\left[\frac{\underset{(x,\mathrm{the\; y})\&Element;\mathrm{\&Omega;}}{\mathrm{\&Sigma;}}{Q}_{\mathrm{image}}(x,\mathrm{the\; y})\&times;\mathrm{SM}(x,\mathrm{the\; y})}{\underset{(x,\mathrm{the\; y})\&Element;\mathrm{\&Omega;}}{\mathrm{\&Sigma;}}\mathrm{SM}(x,\mathrm{the\; y})}\right]}^{\mathrm{\&gamma;}}\&times;{\left[\frac{\underset{(x,\mathrm{the\; y})\&Element;\mathrm{\&Omega;}}{\mathrm{\&Sigma;}}{Q}_{\mathrm{image}}(x,\mathrm{the\; y})\&times;\mathrm{DM}(x,\mathrm{the\; y})}{\underset{(x,\mathrm{the\; y})\&Element;\mathrm{\&Omega;}}{\mathrm{\&Sigma;}}\mathrm{DM}(x,\mathrm{the\; y})}\right]}^{\mathrm{\&beta;}},$ Among them, Ω represents the range of the pixel domain, SM(x, y) = max(SM _org (x, y), SM _dis (x, y)), max() is the maximum value function, γ and β are weight coefficients, In this embodiment, γ=1.601 and β=0.501 are taken.

⑦Using n original undistorted stereoscopic images, establish a set of distorted stereoscopic images under different distortion types and different degrees of distortion. The average subjective score difference of each distorted stereo image in the image set is recorded as DMOS, DMOS=100-MOS, where MOS represents the mean subjective score, DMOS∈[0,100], n≥1.

In this embodiment, the stereoscopic image composed of Fig. 2a and Fig. 2b, the stereoscopic image composed of Fig. 3a and Fig. 3b, the stereoscopic image composed of Fig. 4a and Fig. 4b, the stereoscopic image composed of Fig. 5a and Fig. Stereoscopic image composed of Fig. 6b, stereoscopic image composed of Fig. 7a and Fig. 7b, stereoscopic image composed of Fig. 8a and Fig. 8b, stereoscopic image composed of Fig. 9a and Fig. 9b, stereoscopic image composed of Fig. A total of 12 (n=12) undistorted stereoscopic images of the stereoscopic image constituted by Fig. 11b, the stereoscopic image constituted by Fig. 12a and Fig. 12b, and the stereoscopic image constituted by Fig. 13a and Fig. 13b have been established in different distortion types and different degrees of distortion. The following distorted stereo image collection, the distorted stereo image collection includes 252 distorted stereo images of 4 types of distortion, including 60 distorted stereo images compressed by JPEG, 60 distorted stereo images compressed by JPEG2000, Gaussian There are 60 distorted stereo images with Gaussian Blur, and 72 distorted stereo images with H.264 encoding.

8. According to the operation of calculating the image quality objective evaluation prediction value of S _dis according to step ① to step ⑥, respectively calculate the image quality objective evaluation prediction value of each distorted stereoscopic image in the distorted stereoscopic image set.

Using 12 undistorted stereoscopic images shown in Figures 2a to 13b, 252 distorted stereoscopic images under different degrees of JPEG compression, JPEG2000 compression, Gaussian blur and H.264 encoding distortion are used to analyze the results obtained in this embodiment. Correlation between image quality objective rating predictors and mean subjective rating difference for distorted stereoscopic images. Here, four commonly used objective parameters for evaluating image quality evaluation methods are used as evaluation indicators, namely Pearson correlation coefficient (Pearson linear correlation coefficient, PLCC) and Spearman correlation coefficient (Spearman rank order correlation coefficient, SROCC) under nonlinear regression conditions, Kendall correlation coefficient (Kendall rank-order correlation coefficient, KROCC), mean square error (root mean squared error, RMSE), PLCC and RMSE reflect the accuracy of the distorted stereoscopic image evaluation objective model, and SROCC and KROCC reflect its monotonicity. The five-parameter Logistic function nonlinear fitting is done on the image quality objective evaluation prediction value of the distorted stereoscopic image calculated by the method of the present invention, the higher the PLCC, SROCC and KROCC values, the lower the RMSE value shows that the objective evaluation method and the average subjective rating The better the difference correlation. The Pearson correlation coefficient, the Spearman correlation coefficient, the Kendall correlation coefficient and the mean square error between the image quality objective evaluation prediction value and the subjective rating of the distorted stereoscopic image obtained by the method of the present invention and the method of the present invention are compared respectively, and the comparison results As shown in Table 1, Table 2, Table 3 and Table 4, as can be seen from Table 1, Table 2, Table 3 and Table 4, the final image quality objective evaluation prediction of the distorted stereoscopic image obtained by the method of the present invention The correlation between the value and the average subjective score difference is very high, indicating that the objective evaluation result is relatively consistent with the subjective perception of the human eye, which is enough to illustrate the effectiveness of the method of the present invention.

Figure 14 shows the scatter diagram of the difference between the image quality objective evaluation prediction value and the average subjective evaluation value of each distorted stereo image in the distorted stereo image set. The more concentrated the scatter points, the consistency between the objective evaluation results and the subjective perception the better. It can be seen from FIG. 14 that the scatter diagram obtained by the method of the present invention is relatively concentrated, and has a high degree of agreement with the subjective evaluation data.

Table 1 Comparison of the Pearson correlation coefficient between the objective evaluation predictive value and the subjective rating of the image quality of the distorted stereoscopic image obtained by the method of the present invention and without using the method of the present invention

Table 2 Comparison of the Spearman correlation coefficient between the image quality objective evaluation prediction value and the subjective score of the distorted stereoscopic image obtained by the method of the present invention and without using the method of the present invention

Table 3 Comparison of the Kendall correlation coefficient between the objective evaluation predictive value and the subjective rating of the image quality of the distorted stereoscopic image obtained by the method of the present invention and not utilizing the method of the present invention

Table 4 Comparison of the mean square error between the objective evaluation predictive value and the subjective rating of the image quality of the distorted stereoscopic image obtained by the method of the present invention and without using the method of the present invention

Claims (6)

1. A stereoscopic image quality objective evaluation method based on feature fusion is characterized in that its processing process is: first, according to each pixel in the left viewpoint image and the right viewpoint image of the original undistorted stereoscopic image in different The even symmetric frequency response and odd symmetric frequency response of scale and direction, and the disparity image between the left view point image and the right view point image of the original undistorted stereo image, obtain the cyclopean image of the original undistorted stereo image; according to The even symmetric frequency response and odd symmetric frequency response of each pixel in the left and right view images of the distorted stereo image at different scales and directions, and the left and right images of the original undistorted stereo image The disparity image between the viewpoint images is used to obtain the cyclopean image of the distorted stereoscopic image to be evaluated; secondly, the distorted stereoscopic image to be evaluated is obtained according to the mean and standard deviation of the pixel values of each pixel in the two cyclopean images The objective evaluation metric value of each pixel in the Cyclops image; again, according to the amplitude and phase of the original undistorted stereo image Cyclops image, the corresponding saliency map is obtained; according to the cyclops image of the distorted stereo image to be evaluated The corresponding saliency map is obtained; then, according to the distortion map between the two saliency maps and the two cyclopia maps, the objective evaluation metric value of each pixel in the cyclopia map of the distorted stereo image to be evaluated Perform fusion to obtain the image quality objective evaluation prediction value of the distorted stereoscopic image to be evaluated; finally, obtain the image quality objective evaluation prediction value of multiple distorted stereoscopic images of different distortion types and degrees according to the above processing process. 2. a kind of stereo image quality objective evaluation method based on feature fusion according to claim 1, is characterized in that it specifically comprises the following steps: ①Let S _org be the original undistorted stereo image, let S _dis be the distorted stereo image to be evaluated, record the left viewpoint image of S _org as {L _org (x,y)}, and let the right viewpoint image of S _org The image is recorded as {R _org (x,y)}, the left view image of S _dis is recorded as {L _dis (x,y)}, and the right view image of S _dis is recorded as {R _dis (x,y)} , where (x, y) represents the coordinate position of the pixel in the left viewpoint image and the right viewpoint image, 1≤x≤W, 1≤y≤H, W represents the width of the left viewpoint image and the right viewpoint image, H represents the height of the left view point image and the right view point image, L _org (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {L _org (x, y)}, R _org (x, y) y) means the pixel value of the pixel point whose coordinate position is (x, y) in {R _org (x, y)}, and L _dis (x, y) means that the coordinate position in {L _dis (x, y)} is ( The pixel value of the pixel point of x, y), R _dis (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in {R _dis (x, y)}; ②According to each pixel in {L _org (x,y)}, {R _org (x,y)}, {L _dis (x,y)}, {R _dis (x,y)} at different scales The even symmetric frequency response and odd symmetric frequency response in the and direction, correspondingly get {L _org (x,y)}, {R _org (x,y)}, {L _dis (x,y)}, {R _dis (x ,y)}, and then according to the amplitude of each pixel in {L _org (x,y)} and {R _org (x,y)} and {L _org (x,y) )} and {R _org (x,y)} between the pixel value of each pixel in the disparity image, calculate the Cyclops of S _org , denoted as {CM _org (x,y)}, and according to {L The amplitude of each pixel in _dis (x,y)} and {R _dis (x,y)} and the disparity image between {L _org (x,y)} and {R _org (x,y)} The pixel value of each pixel in , calculate the Cyclops of S _dis , recorded as {CM _dis (x, y)}, where CM _org (x, y) represents the coordinates in {CM _org (x, y)} The pixel value of the pixel point whose position is (x, y), CM _dis (x, y) represents the pixel value of the pixel point whose coordinate position is (x, y) in {CM _dis (x, y)}; ③According to the mean and standard deviation of the pixel values of each pixel in {CM _org (x,y)} and {CM _dis (x,y)}, calculate each of {CM _dis (x,y)} The objective evaluation measurement value of the pixel point, the objective evaluation measurement value of the pixel point whose coordinate position is (x, y) in {CM _dis (x, y)} is recorded as Q _image (x, y); ④ According to the amplitude and phase of {CM _org (x,y)}, calculate the saliency map of {CM _org (x,y)}, denoted as {SM _org (x,y)}, and according to {CM _dis (x, y)} amplitude and phase, calculate the saliency map of {CM _dis (x,y)}, denoted as {SM _dis (x,y)}, where, SM _org (x,y) means {SM _org (x, The pixel value of the pixel whose coordinate position is (x, y) in y)}, SM _dis (x, y) means the pixel of the pixel whose coordinate position is (x, y) in {SM _dis (x, y)} value; ⑤ Calculate the distortion map between {CM _org (x,y)} and {CM _dis (x,y)}, record it as {DM(x,y)}, and set the coordinate position in {DM(x,y)} Be that the pixel value of the pixel point of (x, y) is denoted as DM (x, y), DM (x, y)=(CM _org (x, y)-CM _dis (x, y)) ² ; ⑥ According to {SM _org (x,y)} and {SM _dis (x,y)} and {DM(x,y)}, the objective evaluation of each pixel in {CM _dis (x,y)} The measured values are fused to obtain the predicted value of the objective evaluation of S _dis image quality, denoted as Q, Among them, Ω represents the range of the pixel domain, SM(x, y) = max(SM _org (x, y), SM _dis (x, y)), max() is the maximum value function, γ and β are weight coefficients; ⑦Using n original undistorted stereoscopic images, establish a set of distorted stereoscopic images under different distortion types and different degrees of distortion. The average subjective score difference of each distorted stereoscopic image in the image set is recorded as DMOS, DMOS=100-MOS, where MOS represents the mean subjective score, DMOS∈[0,100], n≥1; 8. According to the operation of calculating the image quality objective evaluation prediction value of S _dis according to step ① to step ⑥, respectively calculate the image quality objective evaluation prediction value of each distorted stereoscopic image in the distorted stereoscopic image set. 3. a kind of stereoscopic image quality objective evaluation method based on feature fusion according to claim 2, it is characterized in that described step 2. The specific process is: ②-1. Perform filtering on {L _org (x, y)} to obtain the even symmetric frequency response and odd symmetric frequency response of each pixel in {L _org (x, y)} in different scales and directions, The even symmetric frequency response of the pixel at the coordinate position (x, y) in {L _org (x, y)} in different scales and directions is recorded as e _{α, θ} (x, y), and {L _org (x , y)}, the odd symmetric frequency response of the pixel at the coordinate position (x, y) in different scales and directions is denoted as o _{α, θ} (x, y), where α represents the scale of the filter used for filtering Factor, 1≤α≤4, θ indicates the direction factor of the filter used for filtering, 1≤θ≤4; ②-2. According to the even symmetric frequency response and odd symmetric frequency response of each pixel in {L _org (x,y)} in different scales and directions, calculate each pixel in {L _org (x,y)} The amplitude of the pixel point, the amplitude of the pixel point whose coordinate position is (x, y) in {L _org (x, y)} is recorded as ②-3. Obtain the amplitude of each pixel in {L _org (x,y)} according to step ②-1 to step ②-2, and obtain {R _org (x,y)} in the same way, The amplitude of each pixel in {L _dis (x, y)} and {R _dis (x, y)}, the pixel whose coordinate position is (x, y) in {R _org (x, y)} The amplitude of The amplitude of the pixel at the coordinate position (x, y) in {L _dis (x, y)} is recorded as Record the amplitude of the pixel at the coordinate position (x, y) in {R _dis (x, y)} as ②-4. Calculate the parallax image between {L _org (x, y)} and {R _org (x, y)} by block matching method, denoted as in, express The pixel value of the pixel point whose middle coordinate position is (x, y); ② _{- 5} . According to the amplitude and The pixel value of each pixel in , calculate the Cyclops image of S _org , which is recorded as {CM _org (x, y)}, and the pixel whose coordinate position is (x, y) in {CM _org (x, y)} The pixel value of a point is denoted as CM _org (x,y), in, Indicates that the coordinate position in {R _org (x,y)} is The amplitude of the pixel point, Indicates that the coordinate position in {R _org (x,y)} is The pixel value of the pixel point; _② -6 _. According to the amplitude and The pixel value of each pixel in , calculate the Cyclops image of S _dis , which is recorded as {CM _dis (x, y)}, and the pixel whose coordinate position is (x, y) in {CM _dis (x, y)} The pixel value of the point is recorded as CM _dis (x,y), in, Indicates that the coordinate position in {R _dis (x,y)} is The amplitude of the pixel point, Indicates that the coordinate position in {R _dis (x,y)} is The pixel value of the pixel. 4. a kind of stereoscopic image quality objective evaluation method based on feature fusion according to claim 3, is characterized in that {L _org (x, y)} is carried out to the filter that filter processing adopts in described step 2.-1 For the log-Garbor filter. 5. according to a kind of stereoscopic image quality objective evaluation method based on feature fusion according to any one of claims 2 to 4, it is characterized in that the specific process of described step 3. is: ③-1. Calculate the mean and standard deviation of the pixel values of each pixel in {CM _org (x, y)} and {CM _dis (x, y)}, and put {CM _org (x, y)} in The mean and standard deviation of the pixel values at the coordinate position (x ₁ ,y ₁ ) are recorded as μ _org (x ₁ ,y ₁ ) and σ _org (x ₁ ,y ₁ ) respectively, and {CM _dis (x ,y)}, the mean and standard deviation of the pixel values of the pixel at the coordinate position (x ₁ ,y ₁ ) are recorded as μ _dis (x ₁ ,y ₁ ) and σ _dis (x ₁ ,y ₁ ), respectively, Among them, 1≤x ₁ ≤W, 1≤y ₁ ≤H, N(x ₁ , y ₁ ) represents an 8×8 neighborhood window centered on the pixel at the coordinate position (x ₁ , y ₁ ), M Indicates the number of pixels within N(x ₁ ,y ₁ ), CM _org (x ₁ ,y ₁ ) indicates the pixel at the coordinate position (x ₁ ,y ₁ ) in {CM _org (x,y)} The pixel value of , CM _dis (x ₁ , y ₁ ) represents the pixel value of the pixel whose coordinate position is (x ₁ , y ₁ ) in {CM _dis (x, y)}; ③-2. According to the mean and standard deviation of the pixel values of each pixel in {CM _org (x, y)} and {CM _dis (x, y)}, calculate {CM _dis (x, y)} The objective evaluation metric value of each pixel in {CM _dis (x,y)}, the objective evaluation metric value of the pixel point whose coordinate position is (x ₁ ,y ₁ ) in {CM dis (x,y)} is recorded as Q _image (x ₁ ,y ₁ ), Among them, C is the control parameter. 6. a kind of stereoscopic image quality objective evaluation method based on feature fusion according to claim 5, is characterized in that the concrete process of described step 4. is: ④-1. Perform discrete Fourier transform on {CM _org (x,y)} to obtain the amplitude and phase of {CM _org (x,y)}, which are recorded as {M _org (u,v)} and {A _org (u,v)}, where u represents the amplitude or phase width of the transform domain, v represents the amplitude or phase height of the transform domain, 1≤u≤W, 1≤v≤H, M _org (u,v) Indicates the amplitude value of the pixel point whose coordinate position is (u,v) in {M _org (u,v)}, and A _org (u,v) indicates that the coordinate position in {A _org (u,v)} is (u, The phase value of the pixel point of v); ④-2. Calculate the amplitude of the high-frequency component of {M _org (u,v)}, record it as {R _org (u,v)}, and set the coordinate position in {R _org (u,v)} as (u, v) The amplitude value of the high-frequency component of the pixel point is recorded as R _org (u,v), R _org (u,v)=log(M _org (u,v))-h _m (u,v)*log (M _org (u,v)), where log() is a logarithmic function based on e, e=2.718281828, "*" is the convolution operation symbol, h _m (u,v) means m×m mean filtering; ④-3. Perform inverse discrete Fourier transform according to {R _org (u, v)} and {A _org (u, v)}, and use the obtained inverse transformed image as the saliency map of {CM _org (x, y)}, Recorded as {SM _org (x, y)}, where, SM _org (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {SM _org (x, y)}; ④-4. Follow steps ④-1 to ④-3 to obtain the saliency map of {CM _org (x, y)}, and obtain the saliency map of {CM _dis (x, y)} in the same way, denoted as {SM _dis (x, y)}, wherein, SM _dis (x, y) represents the pixel value of the pixel whose coordinate position is (x, y) in {SM _dis (x, y)}.