CN103841410B - Based on half reference video QoE objective evaluation method of image feature information - Google Patents

Abstract

The invention discloses a semi-reference video QoE objective evaluation method based on image feature information. The method includes: the operator side extracts the saliency information map and the texture information map of each frame image in the original video, and compresses the saliency information map and the texture information. Figure, to obtain the semi-reference data of the original video; the client receives the semi-reference data of the original video and the damaged video from the operator, extracts the saliency information map and the texture information map of each frame of the image in the damaged video, and obtains The semi-reference data of the damaged video, according to the semi-reference data of the original video and the damaged video, calculate the damage of the damaged video, and use the pre-trained neural network algorithm to evaluate the subjective quality MOS.

Description

An objective evaluation method of semi-reference video QoE based on image feature information

technical field

The invention relates to the field of communication technology, in particular to a semi-reference video QoE objective evaluation method based on image feature information.

Background technique

With the popularity of wireless networks and high-speed broadband access, real-time video services are experiencing rapid development. QoE (Quality of Experience) indicators can reflect the service quality of real-time video services. The objective method of QoE quality evaluation for real-time video services (also known as the QoE objective evaluation method) is to evaluate subjective scores based on specific objective service quality indicators. QoE objective Evaluation methods can be divided into three categories according to the use of raw video data, namely full reference (requires all raw data), semi-reference (requires part of the original data), and no-reference (no raw data is required).

Most of the existing QoE objective evaluation methods are full-reference or no-reference methods. The full-reference method can get the most accurate evaluation results, but it is difficult to apply; the no-reference method is easy to deploy, but usually only applies to specific damage scenarios; the semi-reference method Although a good balance can be achieved between the two, there is a lack of mature solutions.

The problems in the prior art are: both the full-reference method and the no-reference method have limitations in practicability, and the prior art has no horizontal comparison results for testing the accuracy of evaluation.

Contents of the invention

Technical Problems to be Solved by the Invention The prior art has limitations in practicability, and there is no horizontal comparison result for testing the accuracy of evaluation.

For this purpose, the present invention proposes a semi-reference video QoE objective evaluation method based on image feature information, the method comprising:

The operator side extracts the saliency information map and the texture information map of each frame image in the original video, compresses and processes the saliency information map and the texture information map, and obtains semi-reference data of the original video;

The user terminal receives the semi-reference data of the original video and the damaged video from the operator, extracts the saliency information map and texture information map of each frame of the damaged video, and obtains the semi-reference data of the damaged video. Semi-reference data of video and damaged video, calculate the damage of the damaged video, and use the pre-trained neural network algorithm to evaluate the subjective quality MOS, wherein the damaged video is the operator’s data transmitted through the lossy channel end of the original video.

Wherein, the saliency information map includes a time-domain saliency information map and a spatial-domain saliency information map.

Wherein, the saliency information map includes intensity components, color components, direction components and skin tone components with different weights.

Wherein, the texture information map includes a time-domain texture information map and a spatial-domain texture information map.

Wherein, the extraction of the texture information map of each frame image includes: edge extraction, morphological expansion processing and superposition;

Wherein, the edge extraction includes: extracting the edge information image of the current frame image;

The morphological expansion includes: performing morphological expansion processing on the edge information image of the current frame image to obtain the processed edge information image;

The superimposing includes: superimposing the processed edge information image and the current frame image to obtain the texture information map of the current frame image.

Wherein, the texture information map of each frame of image includes: the texture information map of each frame of image in the original video of the operator end, and the texture information map of each frame of image in the damaged video of the user end.

Wherein, the compression process includes:

Using wavelet transform to decompose the saliency information map and texture information map in both the space domain and the time domain to obtain different high-frequency sub-bands;

Make a histogram of all high frequency subbands;

The histograms of all high-frequency subbands were fitted using the generalized Gaussian distribution GGD and the fitting error was calculated.

Wherein, the semi-reference data includes: saliency information spatial domain damage, saliency information time domain damage, texture information spatial domain damage and texture information time domain damage.

Wherein, the calculating the damage condition of the damaged video according to the semi-reference data of the original video and the damaged video includes: calculating the damage condition of the damaged video through relative entropy according to the semi-reference data of the original video and the damaged video.

Compared with the prior art, the beneficial effect of the method provided by the present invention is: the real-time video service QoE objective quality evaluation method provided by the present invention is not sensitive to the type of video damage, that is, the damaged video caused by different reasons can be obtained relatively Accurate evaluation results; the present invention is not sensitive to the underlying transmission network, that is, it can be used for objective quality evaluation of real-time video services in various practical scenarios (including local area networks, wide area networks, and wireless environments, etc.); the present invention is easy to deploy and implement, All module functions can be implemented at the software level. If there are specific needs, hardware implementation can also be considered to speed up the processing speed.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 shows the flow chart of the semi-reference video QoE objective assessment method based on image feature information;

FIG. 2 shows the results of the evaluation of the LVQ database in Example 2.

detailed description

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are the Some, but not all, embodiments are invented. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Example 1:

This embodiment discloses a semi-reference video QoE objective evaluation method based on image feature information, the method comprising:

The operator side extracts the saliency information map and the texture information map of each frame image in the original video, compresses and processes the saliency information map and the texture information map, and obtains semi-reference data of the original video;

The user terminal receives the semi-reference data of the original video and the damaged video from the operator, extracts the saliency information map and texture information map of each frame of the damaged video, and obtains the semi-reference data of the damaged video. Semi-reference data of video and damaged video, calculate the damage of the damaged video, and use the pre-trained neural network algorithm to evaluate the subjective quality MOS, wherein the damaged video is the operator’s data transmitted through the lossy channel end of the original video.

Wherein, the saliency information map includes a time-domain saliency information map and a spatial-domain saliency information map.

Wherein, the saliency information map includes intensity components, color components, direction components and skin tone components with different weights, wherein the weight of the skin tone component is set to 2, and the weight of other components is 1, which can also be adjusted according to actual conditions.

Wherein, the texture information map includes a time-domain texture information map and a spatial-domain texture information map.

Wherein, the extraction of the texture information map of each frame image includes: edge extraction, morphological expansion processing and superposition;

Wherein, the edge extraction includes: extracting the edge information image of the current frame image;

The morphological expansion includes: performing morphological expansion processing on the edge information image of the current frame image to obtain the processed edge information image;

The superimposing includes: superimposing the processed edge information image and the current frame image to obtain the texture information map of the current frame image.

Wherein, the texture information map of each frame of image includes: the texture information map of each frame of image in the original video of the operator end, and the texture information map of each frame of image in the damaged video of the user end.

On the operator side, the compression process includes:

Using wavelet transform to decompose the saliency information map and texture information map in both the space domain and the time domain to obtain different high-frequency sub-bands;

Make a histogram of all high frequency subbands;

The histograms of all high-frequency subbands were fitted using the generalized Gaussian distribution GGD and the fitting error was calculated.

Wherein, the semi-reference data includes: saliency information spatial domain damage, saliency information time domain damage, texture information spatial domain damage and texture information time domain damage.

Wherein, the calculating the damage condition of the damaged video according to the semi-reference data of the original video and the damaged video includes: calculating the damage condition of the damaged video through relative entropy according to the semi-reference data of the original video and the damaged video.

Example 2:

This embodiment discloses a semi-reference video QoE objective evaluation method based on image feature information. As shown in FIG. There are two parts, the operator end and the user end, in which the operator end includes 5 steps, and the user end includes 6 steps. Its overall flow chart is shown in Figure 1, and each step is described below.

Operator side:

101) Extract saliency information for each frame in the original video separately. Saliency describes relatively more attention-grabbing regions in an image. First, the saliency components are constructed from four aspects of intensity, color, direction and skin tone, and then the 4 components are combined into a saliency map according to different weights. Among them, before calculating the salience component of skin tone, it is necessary to apply face recognition technology to detect whether there is a real portrait. The obtained saliency map assigns a saliency value to each pixel in the original image, and the higher the value, the more attractive the area where the pixel is located is.

The specific calculation process is described below. First, a Gaussian pyramid with a scale of 9 is established for each frame of the input image, and the central scale ce{2, 3, 4} is set, and the peripheral scale s=c+δ, where δ∈{3, 4}. Define the difference operation between two images of different scales is: perform pixel-by-pixel subtraction on the coarse-scale image to the fine-scale interpolation. Let r, g and b be the three primary color components of the original image respectively, then the intensity image I is I=(r+g+b)/3. In addition, define the wide tuning color channel R=r-(g+b)/2, G=g-(r+b)/2, B=b-(r+g)/2, and Y=(r+g )/2-|rg|/2-b. The above I, R, G, B, and Y are all for multi-scale. Next, define the intensity feature image as:

Define the color feature image as:

Next, it is necessary to use the Gabor filter on the intensity image I of each scale in the respective directions 6E (0°, 45°, 90°, 135°) to obtain the Gabor pyramid 0 (δ, θ), and define the direction feature image as:

In order to obtain each salient component, it is necessary to combine the above feature images. The specific merge operation is defined as , for two input images of different scales, scale them to scale 4 and add them pixel by pixel. Thus, the saliency components of intensity, color and direction are calculated by formulas (5) to (7) respectively, where is the normalization operator:

In addition, if a human face is detected, it is necessary to additionally calculate the saliency component of the skin pixel Each pixel value is assigned a weight by how close it is to skin tone in the original image. Finally, after merging each saliency component according to the predefined weight W _i , a single-frame saliency image is obtained:

By calculating the saliency of each frame in the video, and using the obtained saliency image to weight the original image, the weighted saliency information of the image can be obtained. The calculation of this step is given by formula (9), where p Describe the original video, S _p represents the saliency image of each frame extracted from the original video, the multiplication operation is performed for each pixel of the two images, i represents a certain frame, F represents the total number of frames, SWS represents the spatial saliency Weighted image, where SWS represent Saliency, Weighted and Spatial respectively.

SWS _P 2 (SWS _P (i)|SWS _P (i) 2 S _p (i)*Original_Video(i), i∈F) (9)

102) Extract texture information for each frame in the original video. Texture information enables the assessment of the sensitivity of various regions to damage based on the edge conditions in the image. This is based on the assumption that impairments appearing in heavily textured areas of an image are less noticeable than in flat tonal areas. In this scheme, the Gaussian Laplacian filter is first used to extract the edge information from the original image, then the edge image is morphologically expanded and each pixel value is reversed to obtain a texture map, and finally the texture map will also be used The texture information of the image is obtained by weighting the original image. Areas with complex textures can be covered by dilated edges, so that image damage in smooth tonal parts can be evaluated emphatically.

The specific calculation process is described below. First of all, the Gaussian Laplacian filter used to extract the edge first performs Gaussian filtering on the image, and then uses the Laplacian operator to obtain the edge image. The Gaussian filtering operation is given by equation (10).

L(x,y;t)=g(x,y;t)*f(x,y)(10)

Where L is the filtering result, f(x,y) is the pixel value at (x,y) of a video frame image, g(x,y;t) is a Gaussian function, described by formula (11), where The t is the filter scale.

$\mathrm{<span\; class="notranslate">\; g</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; ;</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; 1</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&pi;</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; t</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 11</span>}<span\; class="notranslate">\; )</span>$

The Laplacian is given by Equation (12).

${<span\; class="notranslate">\; \&dtri;</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 4</span>}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 12</span>}<span\; class="notranslate">\; )</span>$

The morphological dilation process is given by Equation (13), where A is the input image and B is a rectangular description with a length and width of 8.

dilation(A, B)=(a+b|a∈A, b∈B)(13)

In summary, the calculation process of extracting a single frame texture image is given by formula (14).

By calculating the texture for each frame in the video, and weighting the original image with the obtained texture image, the weighted texture information of the image can be obtained, and the areas that are particularly critical for damage assessment are highlighted, and the rest of the area All presented in dark tones. The calculation of this step is given by equation (15). in Represents the texture image of each frame extracted from the original video, TWS stands for spatial texture weighted image, and the three letters of TWS represent Texture, Weighted and Spatial respectively.

103) This step requires the saliency and texture information of each frame calculated in 101) and 102) as input. When evaluating the QoE of implementing video services, the impairments in the air and time domains are very important for subjective perception. This scheme calculates the corresponding spatial and temporal damage values for saliency information and texture information, respectively. Specifically, the evaluation in the spatial domain uses the results obtained directly from the saliency and texture information of each frame (101) and 102), while the evaluation in the temporal domain uses the saliency and texture information differences between adjacent frames value. Therefore, for each video, a total of 4 aspects need to be evaluated for impairment. The calculation of this step is given by formulas (16) and (17), SWT stands for temporal saliency weighted image, and SWT stands for Saliency, Weighted and Temporal, respectively. TWT stands for temporal texture weighted image, and TWT stands for Texture, Weighted and Temporal respectively.

SWT _P = (SWT _P (i) | SWT _P (i) = abs (SWS _P (i) - SWS _P (i-1)), i ∈ F) (16)

TWT _P =(TWT _P (i)|TWT _P (i)=abs(TWS _P (i)-TWS _P (i-l)), ieF}(17)

104) The spatial and temporal saliency and texture information obtained from 103) have a huge amount of data, which cannot be directly transmitted through the network, so compression processing is required. Among them, 104) describes the steps of wavelet transformation. For the 4 aspects of information obtained in each frame of the original video, the SteerablePyramid decomposition with the number of scales being 3 and the number of directions being 2 is respectively performed, and a total of 6 different high-frequency subbands are obtained for evaluating image damage. Then, a histogram of each high-frequency subband is made to obtain P describing the original video. The calculation of this step is given by equation (18). Among them, Wavelet represents the wavelet transform as mentioned above, and Hist represents the histogram of each high-frequency sub-band.

P=Hist(Wauelet(SWS _P ，TWS _P ，SWT _P ，TWT _P ))(18)

105) This step requires the use of a generalized Gaussian Distribution (GGD) to fit the histograms of each high-frequency subband. The GGD function only determines the shape of the curve by two parameters α and β, and can well fit the high-frequency sub-band histogram, and its definition is given by formulas (19) and (20). At the same time, it is also necessary to use the relative entropy (also known as KL divergence) formula to calculate the error ε generated when the fitting curve fits the high-frequency subband histogram. Specifically, the approximate histogram description P _m obtained by fitting the histogram with GGD is first obtained by formula (21), and then the relative entropy ε of P _m to P is calculated by formula (22). Finally, the transmission parameters corresponding to each high-frequency subband include α, β, and ε, where the value of α is an 11-bit floating-point number (8-bit mantissa, 3-bit exponent), the value of β is 8 bits, and the value of ε is 8 bits, totaling 27 bits . For each video frame, the 4 aspects of damage correspond to 6 high-frequency subbands, that is, a total of 27*4*6=648 bits need to be transmitted. If the FPS of the video is 30, the total bandwidth usage is 648 *30/8=2.43KB/s. This is perfectly acceptable for semi-reference video QoE objective quality assessment. These semi-reference data of the original video need to be transmitted using a lossless auxiliary channel.

$\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}{\mathrm{<span\; class="notranslate">\; 2</span>}\mathrm{<span\; class="notranslate">\; \&alpha;\&Gamma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; \&beta;</span>}<span\; class="notranslate">\; )</span>}{{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; (</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; x</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; /</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; )</span>}}^{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 19</span>}<span\; class="notranslate">\; )</span>$

$\mathrm{<span\; class="notranslate">\; \&Gamma;</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; z</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; \&Integral;</span>}_{\mathrm{<span\; class="notranslate">\; 0</span>}}^{<span\; class="notranslate">\; \&infin;</span>}\frac{{\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{{\mathrm{<span\; class="notranslate">\; e</span>}}^{\mathrm{<span\; class="notranslate">\; t</span>}}}\mathrm{<span\; class="notranslate">\; dt</span>}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 20</span>}<span\; class="notranslate">\; )</span>$

P _m =GGD_Fitting(P)(21)

$\mathrm{<span\; class="notranslate">\; \&epsiv;</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; KL</span>}}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; ln</span>}<span\; class="notranslate">\; (</span>\frac{{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}{\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; )</span>{\mathrm{<span\; class="notranslate">\; P</span>}}_{\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; -</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; twenty\; two</span>}<span\; class="notranslate">\; )</span>$

user terminal:

106) Same as 101) except that the input is damaged video and its saliency image. The calculation of this step is given by equation (23). where Q describes the damaged video, Represents the saliency information of each frame extracted from the damaged video.

107) The same as 102) except that the input is a damaged video and its texture image. The calculation of this step is given by equation (24). in Represents the texture information of each frame extracted from the corrupted video.

108) Same as 103) except the data entered in 106) and 107). The calculation of this step is given by equations (25) and (26).

109) Same as 104) except for the data entered in 108). The calculation of this step is given by equation (27).

110) This step requires the data from 109) and 105) as input. First of all, it is necessary to reconstruct the histogram of the high-frequency subband corresponding to each frame according to the α and β values in the received semi-reference data; then, calculate the specific damage value by formula (28), where P represents the corresponding Each histogram, Q represents each histogram corresponding to the damaged video, P _m represents each approximate histogram obtained by GGD fitting, ε=DKL(P _m ||P). Finally, for each video, four damage values will be calculated, that is, Distortion in formula (28), corresponding to spatial saliency weighted damage, spatial texture weighted damage, temporal saliency weighted damage, and temporal texture weighted damage.

111) Finally, the video damage is calculated according to 110), and the subjective perceived quality (MOS) is evaluated using a pre-trained neural network algorithm. The calculation of this step is given by equation (29).

The beneficial effects of the embodiments of the present invention are:

Considering that when human eyes watch a video, they only pay attention to some areas in the video image rather than the whole, so the video damage in key areas will especially affect the subjective experience. Aiming at this point, this scheme combines the human visual model to obtain the saliency information of the image from four aspects: color, intensity, direction and skin tone. Among them, when determining whether skin tone needs to be considered, face recognition technology is first applied to judge whether there is a portrait in the image, and only when there is a portrait, the salience component of skin tone is additionally calculated, thus making the calculation of saliency information accurate. Guaranteed, thereby improving the accuracy of damage assessment.

The edge cases in the image itself can be used to judge the sensitivity of the image to damage. Specifically, when the edges of a certain area of the image are dense, that is, the texture is complex, it is difficult to be detected by the human eye if there is damage; on the contrary, if the damage appears in an area with a smooth tone, it is easy to be captured by the human eye. Accordingly, this solution extracts the edge information of the image, and then uses the morphological expansion method to cover the pixels near the edge, so that the damage in the pixel smooth area can be evaluated emphatically, thereby improving the accuracy of damage evaluation.

In order to realize the semi-reference of the evaluation method, wavelet transform combined with probability distribution fitting is used to realize data compression, and the specific damage value is calculated by relative entropy. This method can occupy only a small amount of additional transmission bandwidth while ensuring the accuracy of the evaluation.

Example 3:

On the publicly available video subjective quality database LIVEVideoQualityDatabase (LVQD), the accuracy of the evaluation of the present invention is tested, and the test results are shown in FIG. 2 . LVQD contains 10 groups of videos, each group contains 1 original (lossless) video, and 15 damaged videos constructed from the perspectives of H.264 compression damage, MPEG2 compression damage, IP transmission damage and wireless transmission damage, so the entire database A total of 150 damaged videos are included. For each damaged video, a subjective score was made according to the ITU-RBT.500-11 standard. Subjective scoring uses a single-stimulus process, with raters giving scores on a continuous scale from 0 to 100. A total of 38 raters were invited for the experiment, and 9 of them were judged to be invalid according to the criteria and were eliminated. After the remaining 29 scores were processed according to the standard, the MOS and score variance corresponding to 150 damaged videos were obtained.

Using this method to evaluate the QoE objective quality of LVQD damaged video, the results are shown in Figure 2. The results of comparison with other mainstream full-reference QoE objective quality assessment methods are shown in Table 1 and Table 2, where Table 1 lists the Pearson correlation coefficient, and Table 2 lists the Spearman correlation coefficient. Although the semi-reference method proposed in this patent has disadvantages when compared with the full-reference method that can use all original video data, the actual comparison results show that this method is still highly competitive in terms of evaluation accuracy.

Table 1. LVQD assessment results - Pearson correlation coefficient

Table 2. LVQD evaluation results - Spearman correlation coefficient

Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention. within the bounds of the requirements.

Claims (7)

1. the half reference video QoE objective evaluation method based on image feature information, is characterized in that, the method comprises:

Operator's end extracts conspicuousness hum pattern and the texture information figure of each two field picture in original video, and compression is processed described aobviousShow property hum pattern and texture information figure, obtain half reference data of original video;

Half reference data and the impaired video of the original video transmitting held by user side reception operator, extracts each in impaired videoThe conspicuousness hum pattern of two field picture and texture information figure, obtain half reference data of impaired video, according to original video and impairedHalf reference data of video, calculates the degree of impairment of impaired video, uses the good neural network algorithm of training in advance to feel subjectivityAssessed by mass M OS, wherein, described impaired video is the original video through operator's end of Erasure channel transmission;

The extraction of the texture information figure of described each two field picture comprises: edge extracting, morphological dilations are processed and add folded;

Wherein, described edge extracting comprises: the marginal information image that extracts current frame image;

Described morphological dilations comprises: the marginal information image of current frame image is carried out to morphological dilations processing, processedAfter marginal information image;

Describedly add stacked package and draw together: marginal information image after treatment and current frame image are added folded, obtain the texture of current frame imageHum pattern;

Described according to half reference data of original video and impaired video, calculate the degree of impairment of impaired video, comprising:

According to receiving to such an extent that each α, the β value in half reference data rebuild the histogram of the high-frequency sub-band that each frame is corresponding;

Calculate concrete impairment value by formula (28), wherein P represents each histogram that original video is corresponding, and Q represents impaired lookingFrequently corresponding each histogram, P_mRepresent the each approximate histogram that GGD matching obtains, $\mathrm{\&epsiv;}=D_{KL}\left(P_m\right|\left|P\right)=\underset{i}{\&Sigma;}ln\left(\frac{P_m\left(i\right)}{P\left(i\right)}\right)P_m\left(i\right);$

For each video, will calculate 4 impairment values, i.e. Distortion in formula (28), corresponding spatial domain is aobvious respectivelyWork property weighted injury, spatial domain texture weighted injury, time domain conspicuousness weighted injury and time domain texture weighted injury;

$Distortion(P,Q)=D_{KL}\left(P_m\right|\left|Q\right)-\mathrm{\&epsiv;}=\underset{i}{\&Sigma;}ln\left(\frac{P\left(i\right)}{Q\left(i\right)}\right)P_m\left(i\right)---\left(28\right)$

2. method according to claim 1, is characterized in that, described conspicuousness hum pattern comprises time domain conspicuousness hum patternWith spatial domain conspicuousness hum pattern.

3. method according to claim 1, is characterized in that, described conspicuousness hum pattern comprises that the intensity of weighted dividesAmount, color component, durection component and skin color component.

4. method according to claim 1, is characterized in that, described texture information figure comprises time domain texture information figure and skyTerritory texture information figure.

5. method according to claim 1, is characterized in that, the texture information figure of described each two field picture comprises: operationBusiness holds the texture information figure of each two field picture in the impaired video of texture information figure, user side of each two field picture in original video.

6. method according to claim 1 and 2, is characterized in that, described compression processing comprises:

Adopt wavelet transformation to decompose conspicuousness hum pattern and the texture information figure of spatial domain and time domain two aspects, obtain different high frequencyBand;

Make the histogram of all high-frequency sub-band;

Adopt histogram the digital simulation error of all high-frequency sub-band of extensive Gaussian distribution GGD matching.

7. method according to claim 1, is characterized in that, described half reference data comprises: damage in conspicuousness information spatial domainWound, the damage of conspicuousness information time domain, the damage of texture information spatial domain and the damage of texture information time domain.