AT509032B1 - METHOD AND SYSTEM FOR VIDEO QUALITY ASSESSMENT - Google Patents

Abstract

The invention relates to a method of estimating the quality of a video sequence, e.g. in video streaming applications, wherein the content of the video sequence is evaluated and one of a group of predetermined content classes is assigned to the video sequence, after which the quality estimate is made for that content class.

Description

Austrian Patent Office AT 509 032 B1 2014-02-15

description

METHOD AND SYSTEM FOR VIDEO QUALITY ESTIMATION FIELD OF THE INVENTION

The present invention relates to a method and a system for video quality estimation, in particular for video sequences with low resolution. Such low resolution video sequences are typical in video streaming applications, e.g. mobile video streaming applications.

BACKGROUND OF THE INVENTION

In WO 03/042922 A and WO 03/101121 A an estimate of the objective quality of compressed video data is described, wherein no access to the source data is possible. After decompressing the video data using e.g. an MPEG decoder is subjected to a picture type determination to determine whether the decoder video data is intra-coded; then, a discrete cosine transform (DCT) is performed to obtain DCT coefficients and an average quantization error is determined based on the variance of the DCT coefficients, and this quantization error is used to calculate a peak signal-to-noise ratio (PSNR) Signal to noise ratio) used.

According to WO 2006/043500 A, temporal / spatial characteristics are derived on the one hand from a degraded video sequence and on the other hand from a reference video signal, and for the quality estimation the amount of temporal / spatial characteristics corresponding to the relationship between the deteriorated video and the weighted subjective rating of the user.

Mobile video streaming uses low resolutions and low bit rates. Common resolutions are Quarter Common Intermediate Format (QCIF, 176x144 pixels) for mobile phones, Common Intermediate Format (CIF, 352x288 pixels) and Standard Interchange Format (SIF, 320x240 pixels) for data cards and palmtops (PDA). The compulsory codec for UMTS (Universal Mobile Telecommunications System) streaming applications is H.263, but the 3GPP Issue No. 6 already supports a baseline profile of the H.264 / AVC codec. The corresponding encoder settings for UMTS streaming services differ for different streaming content types or streaming applications (resolution, codec). In UMTS bearers with 64-384 kbit / s are used for multimedia streaming (audio and video). Mobile terminals have limited complexity and capacity, so decoding higher quality videos is quite challenging. It can be assumed that the most supported video bitrates for the QCIF resolution are 105 kbit / s and for the CIF and SIF resolution 200 kbit / s.

Several objective metrics for perceptual video quality estimation have been proposed in recent years. The proposed metrics can be divided into two main groups: the human vision model based on video metrics, see, for example, Winkler S. et al., "Video Quality Evaluation for Mobile Applications", Proc. of SPIE Conference on Visual Communications and Image Processing, Vol. 5150 (2003), pp. 593-603, and metrics based only on the objective video parameters, see e.g. Kusuma T.M. et al., "On the Development of a Reduced-Reference Perceptual Image Quality Metrics", Proc. of the 2005 Systems Communications (ICW05), (2005), pp. 178-184. The complexity of these methods is quite high, and their computation requires tremendous computing power. These metrics are for broadband video broadcasting services and do not take into account mobile video streaming scenarios. In addition, techniques that do not require the (uncompressed) original sequence for the estimation of quality are desirable so that complexity can be reduced while at the same time expanding the possibilities of quality prediction delivery. It would therefore be desirable to have an objective measure of the video quality that is simple enough for the calculation in real time on the transmitter side. The present invention is based on the recognition that the complexity of the video quality estimation can be kept low if the character of the video content is estimated on the basis of its content dependency on subjective video quality.

SUMMARY OF THE INVENTION

Accordingly, it is an object of the present invention to provide a method and system for video quality estimation that allows fast real-time estimation with low complexity.

To solve this problem, the present invention provides a method and a system having the features of the independent claims. Preferred and advantageous embodiments are defined in the subclaims.

With the present technique, a simple, efficient and fast estimation of video quality is possible for low resolution video sequences typical of (mobile) video streaming applications. Since user-perceived video quality is significantly dependent on the spatial (edges, colors, ...) and temporal (motion speed, direction, ...) characteristics of the video sequence, the present invention, in a first aspect, is based on a two-stage approach to quality estimation. First, recordings between two scene changes are analyzed and their content class is found out. Second, the quality estimate is based on content class, frame rate (frame) and bitrate. To determine the respective content classes, a content classifier and a corresponding selection of the content classes and their characteristics are provided. In addition, the concept of quality metric is based on the average opinion value obtained by a survey. The results of a comparison of the present technique with several common methods showed that the proposed approach provides a powerful means of estimating the video quality perceived by users of low-resolution video streaming services.

In another aspect, the invention provides a quality estimate simply based on the motion character determined by the amount and direction of movement between two scene changes. The concept of the quality metric is based on content-adaptive parameters that allow a content-dependent video quality estimate, namely according to this special aspect, even without a separate determination of a corresponding content class.

In the classification of the content of the video sequence, it is to increase the accuracy of advantage in the determination of the respective content class to assess color characteristics. Furthermore, it is useful for efficiency reasons to determine the appropriate content class at the sender and to signal content class data to a receiver where the quality estimate for the respective content class is performed. In this context, it is also useful for performing the quality estimation if only parameters available on the decoder are used on the receiver side.

As stated above, one particular feature of the invention is the use of motion characteristics of the video sequence to achieve a fast and stable quality estimate, and these motion characteristics can also be used in determining the particular content class. The motion characteristics may be based on motion vector statistics where, in principle, usable algorithms are known per se.

As for the parameters used to perform the quality estimation, it is of particular advantage if the bit rate and / or frame rate of the video data is used to perform the quality estimation.

As for the motion characteristics used for content-dependent quality estimation, the motion uniformity, the motion horizontality, the zero motion vector ratio, the mean motion vector size, and / or the motion vector deviation ratio may be used and the definitions of the specific movement characteristics mentioned above are given below.

In particular, for the direct quality estimation based on the content-dependent motion characteristics, it has been found that a nonlinear model is of particular use for performing the quality estimation. It is preferred that the quality estimate be based on the equation MOS = f (BRsZ, S) N) U), where MOS is the mean opinion scare, BR is the bit rate, Z is the zero motion vector ratio within a sample, S is the ratio of Motion vector deviation within a shot, N is the mean motion vector size within a shot, and U is the motion uniformity within a shot.

A particularly advantageous universal metric for all content based on the defined motion parameters and on the bitrate is possible if the quality estimate on the

Equation ^ OS-a + b ^ BR + c-Z + dS + fN + g * ln (U) + h, S, N based, where MOS is the mean opinion value, BR is the bit rate, Z is the zero motion vector ratio within a picture, S the ratio of motion vector deviation within a shot, N the mean motion vector size within a shot, U the motion uniformity within a shot, and a, b, c, d, e, f, g and h are empirically determined metric coefficients.

In the case of content class determination, it is advantageous if the quality estimate is based on the equation MOS = f (BR, FR, Content_ Class), where MOS is the median opinion, BR is the bit rate and FR is the frame rate. Preferably, the quality estimate is based on the equation MOS-AE + B-BR + - JrD-FR + - ^ 'where a, β, C, D and E are empirically determined metric coefficients.

As regards the present system for estimating the quality of a video sequence, in particular video streaming applications, it is preferred that such a system include a content classifier is set up to evaluate the content of the respective video sequence and to determine an associated content class from a group of predetermined content classes; Furthermore, a metric module is provided, which is set up to perform a quality estimation on the basis of statistics known per se for the associated content class. The metric module is preferably configured to perform the quality estimation for the associated content class based on the bit rate and the frame rate. In order to be able to determine the respective content class in an efficient manner, the content classifier preferably has a scene change detector so that the content class can be determined for each video recording or video picture.

Preferably, the content classifier comprises a motion vector calculation unit and an associated motion vector feature extraction unit for basing the determination of the content class on the motion vector characteristics. Further, the content classifier may have a color feature extraction unit for accuracy improvement.

The content classifier is preferably arranged on the transmitter side, whereas the metric module is arranged on the receiver side.

A particular advantage of the present technique is that no reference image is needed. Furthermore, the parameters of the respective models used for the quality estimation can easily be obtained empirically by tests with persons.

Other objects, features and advantages of the invention will become apparent from the following description taken in conjunction with the accompanying drawings, which relate to preferred embodiments of the invention, to which, however, the invention is not limited.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings: Fig. 1 shows schematically a conception of a preferred content classifier as part of the system according to the invention; 2 shows in its parts FIG. 2A and FIG. 2B model ECDFs (ECDF - empirical cumulative distribution function) over the motion uniformity (FIG. 2A) and the zero motion vector ratio, respectively (FIG. 2B). [0024] FIG. for four content classes # 1 to # 4; 3 schematically illustrates a video quality estimation system according to a preferred embodiment

Embodiment of the invention; [0026] FIG. 4 shows results of the principal component analyzes (PCA) for all (five) content classes in relation to two components 1 and 2; Fig. 5 is a graph showing the relationship between estimated MOS results and subjective MOS results (MOS - Mean Opinion Score); Fig. 6 is a diagram of PCA results similar to Fig. 4, but now for the embodiment where the results apply to all content classes, meaning that certain content classes are now determined before the estimation; and [0029] FIG. 7 is a diagram similar to FIG. 5 with respect to the estimated and subjective MOS.

Results for the case of FIG. 6.

For streaming services, it is essential to provide a required level of customer satisfaction due to the perceived video stream quality. It is therefore important to choose the compression parameters and network settings to maximize the quality of the end user. Thanks to a significant video compression gain, the latest video encoding standard H.264 / AVC can provide video streaming for low bit and frame rates while maintaining perceptual quality. This is ideal for video applications in 3G wireless networks. The basis for such parameters and settings is an appropriate estimate of the video quality, and the present technique aims at estimating video quality, particularly mobile video streaming, at user level (perceptual quality of care) for any codec settings in the 3G network. Like. And for any video content types. In particular, a recognition of the most significant content types is targeted to estimate the video quality of mobile video streaming at user level (perception quality of care) and to determine the most appropriate codec settings for these common types of content.

The visual perception of video content by humans is determined by the character of the sequence being observed. Therefore, in the present technique, it is preferable to distinguish between different content characters / classes because they greatly affect subjective quality. The character of a sequence can be described by the extent of edges (spatial information) in the individual frames and by the type and direction of movement (temporal information). The data rate of the video sequence to be examined is divided by the number of frames per second. Higher frame rates result in less spatial information in individual images and possibly in some compression artifacts. Thus, using the data rate as the objective parameter, either the spatial or temporal information can be considered. In the prior art, see e.g. Kusama T.M. et al., "On the Development of a Reduced Reference Perceptual Image Quality Metric", Proc. of the 2005 Systems Communications (ICW05), (2005), p. 178-184, the focus is primarily on spatial information. Such an approach comes primarily from the quality estimation of still images. However, especially at low resolutions and after the application of compression, not only does the speed of movement (which most affects the compression rate) play a significant role in user perception, but also the type of motion. Therefore, the focus is preferably placed on motion characteristics of the video sequences that determine the perceived quality.

In Fig. 1, the formation of a content classifier 1 is shown, wherein an original video sequence at 2 a scene change detector 3 is supplied to the one hand, a unit 4 for calculating the motion vector (MV - motion vector) and on the other hand, a unit 5 for Extraction of color features are connected. A motion vector feature extraction unit 6 is connected downstream of the motion vector calculation unit 4, and its output is connected to a content class estimator 7, which is also connected to the color feature extraction unit 5 and at its output 8 provides the appropriate content classes.

In this regard, a more detailed discussion of the structure and function of Content Classifier 1 is given below: 1) Scene Change Detector 3: A video stream 2 may consist of a plurality of different scenes having different content, spatial and temporal information (ie, a typical Video sequence for news consists of some recordings of the presenter and various recordings for the reporting of the described events). Since each shot of a sequence may have a different content character, splitting a video into its basic temporal units - shots - is a first step in the process of video content classification due to changing the contents of shots within a sequence. A picture is a series of video pictures taken with a camera (e.g., zooming in or out of an object, panning along a landscape, etc.). Two successive recordings are separated from each other by a record limit, which may be abrupt or gradual. While an abrupt take limit (cut) is created by simply attaching one shot to another without modifying it, a gradual take limit is the result 5/19

Austrian Patent Office AT509 032B1 2014-02-15 of a processing made to connect two exposures.

The most suitable low complexity method for this purpose is a dynamic scene threshold scene change detection as described in Dimou A. et al., "Scene Change Detection for H.264 Using Dynamic Threshold Techniques", Proc. of the 5th EURASIP Conference on Speech and Image Processing, Multimedia Communications and Service, (2005). The method can be adapted to the present purpose: the coefficients of the threshold function are modified and ten emerging frames are additionally taken into account. The scene change detector 3 works with a precision and hit rate of over 97%. Such accuracy is more than satisfactory for the present purpose of content classification.

2) Motion vector unit 4: The block from the current picture for which a matching block is searched is known as a target block. The corresponding difference in locations between the matching block and the target block is known as the motion vector (MV). If the matching block is found at the same location as the target block, then the difference is zero and the motion vector is known as the null vector.

The difference between the target block and the matching block increases (approximately linearly) with the size of the blocks; smaller blocks describe the actual movement in the picture better. On the other hand, an increase in objective accuracy does not always imply better performance. It has been observed that in choosing too small blocks, the resulting motion vectors no longer reflect the movement as perceived by the viewer. Due to the unavoidable presence of noise in video sequences and the peculiarities of the human visual system, it is true that a movement is detected but not seen by a human viewer. Such behavior is undesirable for the present technique. After several attempts with videos of different character, it was found that a block size of 8x8 pixels would be a good compromise for QVGA resolution sequences. The 320x240 pixels are divided into 30x40 blocks, giving a total of 1200 MVs per frame.

The second part of the process, namely the one with a rather high expenditure of time and resources, is the block match. Each block in the current frame is compared to a particular search area in the past frame to find a matching block. This process can only be performed on the luminance component of the frame. A matching criterion must be used to quantify the similarity between the target block and the candidate blocks. Because of the simplicity and good performance, it is preferable to use the sum of absolute differences (SAD), which is calculated as a pixelized sum of the absolute differences between the two blocks compared:

(1) where Bn and Bm are the two blocks of size NxM and i and j denote pixel coordinates. If more than one SAD minimum is detected, priority is given to the matching block whose position is most similar to that of the target block or, equivalently, to the least significant motion vector.

[0039] 3) Extraction of Sequence Motion and Color Parameters (Units 6; 5): After receiving the motion vectors, the information about the motion (motion features) in the sequence must be extracted, and this is done in unit 6. The static or dynamic character of a sequence is one of the main reasons for the differences in perceived quality. It is intended, not just a classification in terms of " static sequences " and "dynamic sequences" but also to examine this aspect more thoroughly and to determine typical levels of exercise quantity for each major content class. The total amount of movement or even the absence of motion in an image is based on the proportion of blocks with zero vectors, i. Blocks, 6/19 Austrian Patent Office AT509 032B1 2014-02-15 which do not move from one frame to another, are easy to estimate. Therefore, the average proportion of static blocks in a sequence of frames is very useful when choosing between contents with typical different overall motion "levels". should be distinguished.

The length of a respective motion vector indicates how far the block has moved from one frame to the next, and its angle provides the information in which direction that movement occurred. Therefore, the mean motion vector size in a frame or frame sequence is an indicator of how fast the overall motion is going.

On the other hand, the exact knowledge of the direction in which the movement takes place seems quite unnecessary for the present estimate. On the other hand, the detection of a main motion direction corresponding to a large proportion of motion vectors pointing in the same direction is valuable information. It can thus be assumed that the analysis of the distribution of size and angle of the motion vectors can provide substantial information about the nature of the motion in the sequence. Thus, a set of statistical calculations on the motion vectors was used to examine their significance and to find out which features could be used to identify perceptual content types.

Finally, the content classification (see unit 7) can be based on the following statistical and resolution-dependent features of motion vectors within one shot (over all frames of the analyzed sequence): [0044] FIG. Zero MV Ratio z: This ratio z is the percentage of zero motion vectors in an image. It is the proportion of the frame that does not (or only slightly) change between two consecutive frames. It usually corresponds to the background when the camera is static within a shot.

[0046] Mean MV Size n: n denotes the proportion of the mean size of the non-zero motion vectors within a frame, normalized to the screen width and expressed in percent. This parameter n determines the amount of global motion.

[0048] Motion Uniformity u The parameter u is the percentage of motion vectors pointing in the frame in the predominant direction (the most common direction of motion vectors). For this purpose, the granularity of the direction can be set to 10 °.

[0050] Horizontal Horizons h: Horizontality is defined as a percentage of motion vectors pointing in the horizontal direction. Horizontal motion vectors have the intervals (-10; 10) ° or (170; 190) °.

To enhance the accuracy of the content classifier 1, color features may be considered, see unit 5. Color histograms provide additional information about the spatial sequence character, because the depth and brightness of the colors differ for different content types. Football sequences, for example, contain a lot of varying green colors, while animated sequences have their own rich colors. This property has significant effects on the compression and transmission artifacts. Therefore, the following parameters can be used:

[0053]. Green share G

The green component G is defined as a percentage of green pixels in a frame. For this purpose, the RGB color space can be sampled down in two bits per color component, resulting in 64 colors. Five colors of the 64 colors cover all green variants.

Preferably, five content classes # 1 through # 5 are identified based on the content features defined above. Furthermore, examples of decision algorithms for automatic content classification will now be disclosed and evaluated.

For the content classification of mobile video streaming, the five most common content classes with different user perception effects can be defined as follows: 1) Content Class # 1 (eg News): Content Class # 1 contains sequences with a small one moving area of interest (face) on a static background. The movement of the region of interest (ROI) is mainly determined only by movements of the eyes, mouth and face. The ROI covers about 15% of the screen area.

2) Content Class # 2 (e.g., Football): Content Class # 2 contains wide angle sequences with even camera movement (panning). The camera follows e.g. a small, rapidly moving object (ball) on the uniform (e.g., typically green) colored background.

3) Content Class # 3 (e.g., Animated Film): Content Class # 3 dominates the motion of objects and the background is usually static. There is almost no global movement due to the artificial origin of the films (no camera). The object of the movement has no natural character.

4) Content Class # 4 (e.g., Panorama): Content Class # 4 contains global motion sequences taken with a wide-angle panning camera. The camera movement is smooth and in a single direction.

5) Content Class # 5 (remainder): Content Class # 5 contains much global and local motion or scene change. Scenes shorter than three seconds also fall into content class # 5. Content class # 5 includes scenes that do not fit into any of the previous four classes.

It would be possible to classify the content based on thresholds, but this is a limited and less accurate method of evaluating larger amounts of data, and preferably a more complex and reliable method based on hypothesis testing is used. Each of the content classes described is determined by unique statistical motion characteristics and color parameters (see FIG. 2). Because of the unique statistical characteristics of well-defined content classes, it is not necessary to perform M-weighted hypothesis tests, and it is sufficient to separately formulate a null hypothesis (HO) for each content class based on these statistical characteristics. Since the distributions of examined parameters do not fit well into any of the known distribution functions, the sequence is classified as Content Class # 5. For the present hypothetical evaluation, a method that can work with empirical distributions (samples) is preferred. The most suitable method for this is not parametric and distribution-free: the Kolmogorov-Smirnov (KS) test, cf. Bosch K., "Statistical Pocketbook", Oldenbourg Wissensch. Vlg., (1998). The KS test is used to determine whether two underlying probability distributions differ, or whether an underlying probability distribution differs from a hypothesized distribution, in both cases using finite examples. The two-sample KS test is one of the most useful and general nonparametric methods of comparing two samples, as it is sensitive to differences in both the location and shape of the empirical cumulative distribution functions of the two samples. The Empirical Cumulative Distribution (ECDF) functions are obtained from the typical sequence group for each content class. In a practical test, model ECDFs were derived from a group of 142 typical sequences. Each content class is described by five model ECDFs (zero MV ratio, mean MV size, motion uniformity, horizontal mobility, green fraction) that correspond to the respective HO hypothesis. Furthermore, it is necessary to determine the maximum deviation (Dcc max) within a content class for all parameters (for each model ECDF). If Fn (x) is the model ECDF and F (x) is the ECDF of the examined sequence, then the maximum difference Dn between Fn (x) and F (x) can be written as follows:

Dn = maxjFn (x) F (x) || (2) The content class estimate is based on a binary hypothesis test within the first four content classes. The KS test compares the ECDFs of the examined sequence and all model ECDFs of the first four content classes. The KS test compares five ECDFs (with defined MV or defined color parameters) of defined content classes specified by the HO hypothesis with all five ECDFs of the examined content. If the obtained Dn for all parameters and the (first four) content classes is less than D ^ max for each parameter, then the examined sequence matches this content class.

If the ECDFs of the examined sequence do not match any of the first four content classes, the content classifier 1 (see Fig. 1) decides on the remaining content class # 5. Classifier 1 estimates the content on the sender side based on the original sequence.

2 are examples of model ECDFs as described above for the motion uniformity u (FIG. 2A) and the zero MV ratio z (FIG. 2B) with respect to the content classes # 1, # 2, respectively. # 3 and # 4 shown.

The performance of content classifier 1 was evaluated using two parameters. (1) Incorrect detection reflects the ratio of improper detection of a content class in the case that examined sequences belong to a different content class. (2) A good match reflects the ratio of a successful classification of sequences studied when the sequences under study belong to one of the first four classes. It should be noted that there are almost always cuts and no gradual changes in the test sequences. The scene change detector 3 was sensitive to gradual recording limits (dissolve, fade, erase). As can be seen from the following Table I, the achieved accuracy of the Content Classifier 1 is 98%, which is a satisfactory result for further quality estimates.

Table 1:

Content class Incorrect detection [%] Good agreement [%] 1 0 97 2 0 100 3 5, 6 92 4 0 100 Sequence number 786 98 For the tests, two groups of five video sequences each lasting 10 seconds and SIF were used Resolution selected. All sequences were coded with an H-264 baseline profile 1b. For subjective quality testing, the combinations of frame rate (FR) and bit rate (BR) shown in Table II were used. There were a total of 36 combinations. 9/19 Austrian Patent Office AT509 032B1 2014-02-15

Table II FR [fps] / BR [kbps] 24 50 56 60 70 80 105 5 Ne, Ca Vi Ne, Ca Ne 7.5 Ne, Ca Ne, Ca Vi Vi Ne, Sun, Vi 10 Ne, Ca Ne , Vi Vi Vi Ne, Sun, Vi 15 Ne Ne Ne Ne Ne, So, Vi (Ne-news; Ca-animation; So-foot ball; Pa-Panorama; Vi- Videoc ip) To obtain a MOS (middle Opinion value), the test was carried out on 36 subjects for two different groups of test sequences. The first group was used for metric performance and the second for metric performance evaluation. The training test group was conducted with 26 subjects and the evaluation test group with 10 subjects. The training and assessment tests were assembled from different groups of five video sequences. The chosen group varied in age (between 20 and 30), gender, education and experience with image processing.

The tests were carried out in accordance with the ITU-T recommendation, cf. ITU-T Recommendation P.910, " Subjective Video Quality Assessment Methods for Multimedia Applications " (Method for the subjective evaluation of video in multimedia applications) (1999) using the Absolute Category Rating (ACR) method, as it better mimics the streaming scenario in the real world. The subjects did not have the original sequence as a reference, resulting in a higher variance. Subjects rated the video quality using a five-point MOS scale (1 - very bad, 2 - bad, 3 - adequate, 4 - good, 5 - very good). According to experience with previous psychovisual experiments, the subjective results are slightly different when viewed on UMTS handsets or with PC monitors. Because of this experience, it seemed appropriate not to follow the ITU-T recommendation on this point only in this one case, and to mimic the real conditions of the UMTS service, all sequences on a personal digital assistant (PDA-Personal Digital Assistant) PDA VPA IV UMTS / WLAN displayed.

The eye relief from the PDA was not determined, but chosen by the subjects. However, it was found that it was pleasant for all people to use the PDA at a distance of 20-30 cm. At the beginning of the test series, the test subjects were offered three training sequences. The test sequences were presented in an arbitrary order, with the additional requirement that the same sequence (even if differentially degraded) did not appear in sequence. Two runs of each test were made. To avoid a learning effect, there was a half-hour break between the first and second passes. Further processing of the results discarded those sequences that were rated with an individual standard deviation greater than 1. Following this rule, 12.4% of the test results were excluded.

For the actual video quality estimation, a real-time video quality estimator 10 may be used as shown schematically in FIG. The estimation at the receiver side 11 must be based only on the compressed sequence without (uncompressed) original sequence, and the information about the content class is displayed in parallel with the video streaming, see path 12, the latter being signaled at 13 to reduce the complexity of the Reduce processing as much as possible. Such a measuring arrangement allows the continuous video quality measurement in real time on both sides, namely the user (receiver side 11) and the provider (transmitter side 14).

More specifically, Fig. 3 shows a content-based video quality estimation system 10 having a content classifier 1 as shown in Fig. 1. An encoder 15 performs the data compression to obtain a compressed video which is sent to the receiver 11, decompresses it from a decoder 16, and a metric module 17 is provided for receiving BR and FR data as well as content class data and for issuing information about the mean opinion value (MOS) (estimation).

Due to the limited processing capacity of the terminals, it is advantageous to identify low-complexity objective parameters. To minimize the complexity, the most suitable parameters are already provided: frame rate (FR) and bit rate (BR). These parameters FR, BR are the codec compression settings and are displayed during the initiation of the streaming session, where no computational complexity is required for the estimation since they are known by both the transceiver 14 and the receiver 11. Furthermore, the influence of these two parameters on one examined data set for each content class can be described separately. For this purpose, a known multivariate statistical method can be used, namely the Principal Component Analysis (PCA), cf. Krzanowski W.J., "Principles of Multivariate Analysis", Clarendon Press (1988). The PCA was performed to check the further applicability of the objective parameters BR and FR for metric execution. The PCA was performed separately for all content classes. In the test case, the first two components proved to be sufficient for modeling the data variance, see Table III, which shows the overall variability of the first two components for all content classes:

Table III

Sequence Variab. from PC1 [%] Variab. of PC2 [%] Content class 1 61,7 23,1 Content class 2 51,8 32, 9 Content class 3 54, 8 30,4 Content class 4 53,1 42,7 Content class 5 63, 5 28, 2 The PCA Results for the first two components ("Component 1" and "Component 2") are illustrated in Figure 4, and they show a sufficient impact of BR and FR on the record for all content classes.

The proposed low complex metric can be based on two objective parameters BR and FR for each content class as follows: MOS = f (BR, FR, Content_class) (3) Here, a common model is proposed for all content classes , Therefore, the model has linear and hyperbolic elements, see the following equation (4), and the coefficients A, B, C, D, E vary considerably for the content classes CC (they may also have zero values). On the other hand, a fairly good correlation was achieved with one offset and two non-zero coefficients, see Table IV below:

Table IV

Coeff. CC 1 CC 2 CC 3 CC 4 CC5 A 4.0317 1.3033 4.3138 1.8094 1.0292 B 0.0157 0.0337 0.0290 C -44.9873 0 -31.7755 0 0 D 0 0.0828 0.0604 0.0044 0 E -0.5752 0 0 0 -1.6115 (CC = Content class - Content class) 11/19 Austrian Patent Office AT509 032B1 2014-02-15 (4)

MOS = A + B BR + - ^ - + D FR + Jr BK rK

The metric coefficients were obtained by a linear regression of the proposed model with the training group (MOS values averaged over two runs from all 26 subjective scores for the respective test sequence). To evaluate the quality of the fit of our proposed metric, a (linear) Pearson correlation factor: r =, TxTy T (5) V (xx) (yy) and the Spearman rank correlation factor: r '= l-fi (xy); x ~ y) (6) N (N3-1) can be used; see also VQEG: " Final report from the Video Quality Experts Group on the validation of objective models of video quality assessment ", http://www.vaea.org/ (2000).

In this case, the vector x corresponds to the MOS average values of the evaluation group (averaged over two runs from all 10 subjective evaluations for the respective test sequence) for all coded sequences tested. The vector y corresponds to the prediction by the proposed metric. The dimension of x and y refers to N. The truncation of the subjective video quality estimate as compared to the subjective quality data is summarized in Table V below and shown in FIG. Correlations obtained in the motion group show a very good performance of the proposed metric for all content classes except Content Class # 3, which contains two- and three-dimensional animations. This feature increases the variability of MOS results within this content class and reduces metric pass performance.

Table V

Content Type CC 1 CC2 CC 3 CC 4 CC5 r 0,9277 0,9018 0,7559 0,9030 0,9307 r '0,9964 0,8863 0,8409 0,9812 0,9695 [0079] The perceptual quality metric described above Content based works well for the most common content types for mobile video streaming services. The proposed method allows for continuous quality measurement at both the transceiver and the receiver side because of its low processing complexity. The automatic content classification enables a video quality estimation within a content class. The proposed automatic content classification recognizes the most common content types with high accuracy. In addition, hypothesis-based classification is a universal statistical content classification method that provides almost unlimited possibilities for defining new content classes. It is therefore sufficient to design a content-dependent, low-complexity metric for each defined content type. The proposed metrics based on basic codec compression set parameters, on the one hand, have minimal complexity and, on the other hand, excellent predictive power. The present approach to video quality estimation provides a reliable method that can easily be extended.

The above description relates to the determination of the content class before performing the class-dependent quality estimation. However, it has been found that it is also possible to estimate the video quality without special content classification, namely by using a universal metric, which in turn is reference-free and based on content-adaptive parameters, to provide a content-dependent video quality estimate based on the Austrian Patent Office AT509 032B1 2014-02-15 as described above in connection with content classification; however, such motion characteristics determined by the amount and direction of movement between two scene changes may also be used for direct video quality estimation.

As already stated, the video quality perceived by users depends on the spatial and temporal characteristics of the video sequences, and most of the prior art techniques are based on spatial features. Here, the present quality estimate is primarily motion-based, and the proposed non-referenced universal metric is suitable for various video content types and data compression settings and is a potent means of estimating user perceived video quality for low-resolution video streaming services.

As already stated, the visual perception of video content by humans is determined by the character of the considered sequence, which is described in the prior art by spatial information. Such approaches come mainly from the quality estimation of still images. In the present technique, motion characteristics are preferably used to characterize the sequence. At low resolution and after applying compression, not only the speed of motion (which most affects the compression rate), but also the type and direction of movement (timing information) play an important role in user perception. Therefore, the present invention is based on motion characteristics of the video sequences for determining the perceived quality.

The advantage of temporal segmentation of video sequences and motion vectors is in the video quality estimation within two sections. Since the video sequences may contain different scenes - recordings with different characteristics - each sequence is first segmented by the dynamic threshold based scene change detection.

With regard to the extraction of sequence motion parameters, the static or dynamic character of a sequence is one of the main reasons for the differences in perceived quality. After examining various sequences, the following parameters can be chosen to represent the motion characteristics: [0085] FIG. Zero MV Ratio Within One Shot Z: Z is defined as the percentage of zero motion vectors within a shot. It is the proportion of the frame that does not (or only slightly) changes between two consecutive frames, averaged over all frames of a shot. It usually corresponds to the background when the camera is static within a shot.

[0087] Mean MV Size Within Shot N: This is the fraction of mean size of non-zero motion vectors within a shot, normalized to the screen width, expressed as a percentage. This parameter defines the amount of global movement.

[0089] Ratio of MV Deviation Within Shot S: S is defined as the ratio of the standard deviation of the motion vector within a shot to the mean MV size N within a shot, expressed in percent.

[0091]. Movement uniformity within a shot U: This is the percentage of motion vectors pointing in the predominant direction (the most common direction of MVs) within a shot. For this purpose, the granularity of the direction is 10 °.

Furthermore, it is advantageous to investigate the influence of these motion parameters as well as the bit rate (BR) on the examined content. A well-known multivariate (multidimensional) statistical method, namely Principal Component Analysis (PCA), can be used for this purpose. The PCA is performed to test for further applicability of the motion characteristics and BR for the metric design. In a test case, the first two components proved to be sufficient for a corresponding modeling of the data variance. The variability of the first component is 42.1% and the second component is 20.6%. The PCA results as illustrated in Figure 6 show sufficient influence of the parameters examined on the data set for all content classes.

In the above test, the frame rates were 5fps; 7,5fps; 10fps; 15fps and bit rates 24kbit / s, 50kbit / s, 56kbit / s, 60kbit / s, 70kbit / s, 80kbit / s, 105kbit / s used in combination. There were a total of 36 combinations, but some combinations were excluded where the resulting video quality was clearly inadequate. In order to obtain a MOS (mean opinion value), 36 test persons (training group with 26 and evaluation group with 10 persons) were worked on for two different groups of test sequences. The training and assessment tests were assembled from different groups of five video sequences. The tests were conducted in accordance with the ITU-T Recommendation using the Absolute Quality Rating (ACR) category method, as it better mimics the real-world streaming scenario.

A universal metric for all contents based on defined motion parameters and BR is now proposed. MOS = a + b-BR + cZ + d-Se-tf-N2 + g-ln (U) + h'S'N (7) The metric coefficients a, b, c, d, e, f, g and h can be obtained with the linear regression of the proposed model in a training group (MOS values averaged over two runs from all 26 subjective scores for a given test sequence). To assess the quality of fit of the proposed metric for the given data, a Pearson and Spearman Rank Correlation Factor, VQEG: " Final Report from the Video Quality Assessment Group ", http: // /www.vaeq.org/. (2000). The metric model was evaluated with MOS scores of the scoring group (averaged over two runs from all 10 subjective scores for each test sequence). The metric predictive power over the Pearson correlation score is 0.8025, and with the Spearman correlation 0.9982.

The correlations obtained in the evaluation group - cf. Figure 7 - shows a very good performance of the proposed metric for all contents.

Thus, the proposed motion-based video quality metric for mobile video streaming services allows continuous and reference-free quality measurement at both the sender / receiver and receiver sides.

In addition, due to the content-related dependency of the metric parameters, the proposed metric offers a good performance in the case of contents with significantly different content features. 14/19

Claims (20)

Austrian Patent Office AT 509 032 B1 2014-02-15 Claims 1. A method of estimating the perceptual quality of a video sequence in video streaming applications, wherein a motion estimation motion is used in the video sequence, and wherein video data comprises a bit rate and a frame rate are characterized in that predetermined sequence motion parameters are extracted from the video sequence and used along with at least one of bit rate and frame rate of the video data to perform the quality estimation. 2. The method according to claim 1, characterized in that for the estimation of the video quality between two sections scene changes are detected on the basis of a dynamic threshold. 3. The method according to claim 1 or 2, characterized in that the motion uniformity is used as a movement parameter. 4. The method according to any one of claims 1 to 3, characterized in that a defined as a percentage of zero motion vectors within a frame or a recording zero motion vector ratio is used as a movement parameter. A method according to any one of claims 1 to 4, characterized in that an average motion vector size, defined as a proportion of mean size of non-zero motion vectors within a frame normalized to the screen width and expressed in percent, is defined as Motion parameter is used. A method according to any one of claims 1 to 5, characterized in that the motion vector deviation ratio within a frame defined as the ratio of the standard deviations of the motion vector within a shot to the mean motion vector magnitude within a shot and expressed as a percentage is used as the motion parameter for the video quality estimate becomes. 7. The method according to any one of claims 1 to 6, characterized in that in the case of a system with a transmitter for transmitting video sequences and a receiver for receiving video sequences, the video quality estimation is performed at the receiver. 8. Method according to claims 3 to 7, characterized in that a nonlinear model for the video quality estimation is used. Method according to claim 8, characterized in that the quality estimate is based on the equation MOS = a + b-BR + c-Z + dSe + fN2 + gln (U) + h-SN, where MOS is the mean value of opinion, BR the Bit rate, Z the zero motion vector ratio within a shot, S the motion vector deviation ratio within a shot, N the mean motion vector magnitude within a shot, and U the motion uniformity within a shot and a, b, c, d, e, f, g, and h empirically determined are metric coefficients. A method according to any one of claims 1 to 8, characterized in that the content of the video sequence is evaluated on the basis of sequence movement parameters of the video sequence and one of a group of predetermined content classes is assigned to the video sequence, whereupon the quality estimate is made for that content class. 15/19 Austrian Patent Office AT509 032B1 2014-02-15 11. The method according to claim 10, characterized in that for determining the respective content class also color features are evaluated. The method of claim 10 or 11, characterized in that the motion parameters used to determine the content class are based on motion vector statistics. 13. The method according to any one of claims 10 to 12, characterized in that the quality estimate is based on the equation MOS = A + B-BR + -jfe + D-FR + -j%, where MOS is the mean value of opinion, BR the bit rate and FR the Are frame rates, and where A, B, C, D and E are empirically determined content class specific metric coefficients. A receiver (11) for receiving video sequences in video streaming applications, comprising video data at a bit rate and a frame rate, characterized by a metric module (17) adapted to perform a perceptual quality estimation based on predetermined ones , motion parameters extracted from the video sequence, and at least one of the bit rate and the frame rate of the video data. A system (10) comprising a receiver (11) according to claim 14 and characterized by a content classifier (1) adapted to evaluate the content of the video sequence and determine an associated content class from a group of predetermined content classes based on motion parameters of the video sequence is. 16. System according to claim 15, characterized in that the metric module (17) is set up for performing the quality estimation for the associated content class on the basis of the bit rate and the frame rate. 17. System according to claim 15 or 16, characterized in that the content classifier (1) comprises a scene change detector (3). A system according to any one of claims 15 to 17, characterized in that the content classifier (1) comprises a motion vector calculation unit (4) and an associated motion vector feature extraction unit (6). A system according to any one of claims 15 to 18, characterized in that the content classifier (1) comprises a unit (5) for extracting color features. A system according to any one of claims 15 to 19, characterized in that the content classifier (1) is arranged in a transmitter (14). 3 sheets of drawings 16/19