KR20050084940A - A method for generating a quality oriented significance map for assessing the quality of an image or video - Google Patents

Abstract

A method for generating a quality oriented significance camp for assessing the quality of an image or video, comprising the steps of extracting features of the image or video, determining a perceptual quality requirement of at least one extracted feature and integrating the extracted features and the perceptual quality requirement of the at least one extracted feature to form an array of significance level values, thereby generating the quality oriented significance map.

Description

A method for generating a quality oriented significance map for assessing the quality of an image or video}

Visual distortion metrics play an important role in monitoring the quality of broadcasted images / videos, controlling compression efficiency and improving image enhancement processes. There are generally two classes of quality or distortion assessment approaches. The first class is based on mathematically defined measurements such as, for example, widely used mean squared error (MSE), peak signal to noise ratio (PSNR), and the like. The second class is based on measuring visual distortion by simulating human visual system (HVS) characteristics.

In the first class of approaches, the definition of MSE is

Given by Wow Are the pixel values of the original image and the distorted image, respectively. The definition of PSNR is

to be. The advantages of the first class of approaches are that they are mathematically simple and the computational complexity is low. For this reason, the first class of approaches is widely adopted.

The second class of approach, however, targets perceptual results closer to human vision and thus better accuracy in visual assessment and information processing. However, due to the incomplete understanding of HVS and the delay in including physiological and / or psychological results in HVS, the performance of the second class of approaches is still not satisfactory.

There is physiological and psychological evidence that an observer who sees an image or video does not pay attention to all the visual information of the image or video but focuses only on certain areas. This visual attention information from the observer has been used in many uses of HVS, for example for the calculation of visual perceptual search processes, or for assessing the quality of an image or video.

Visual attention can be implemented by either a bottom-up process or a top-down process. In the bottom-up process, visual attention is based on stimuli from the visual features of the image / video, and a significance map for the image / video is formed based on these stimuli. Examples of visual feature based stimuli include lighting, color, movement, shape, and the like. In the top-down process, a significance map for an image / video is formed based on indications from other known information such as dictionary / active area knowledge or sound.

[1] discloses a method for measuring the distortion of an image by combining three factors, namely, loss of correlation, luminance distortion, and contrast distortion.

[2] proposes a non-reference quality matrix 100 as shown in FIG. The distorted image / video 101 is received by the artifact extraction unit 102 and the distribution of blurring and blockiness of the image / video 101 is detected. The determination unit 104 determines the characteristics of this blurring and block phenomenon and generates an output signal 105 representing the distortion value of the distorted image / video 101.

The methods in [1] and [2] are included in the first class of approaches and thus do not provide results close to human perception as compared to the second class of approaches.

[3] proposes a matrix 200 based on video decompression and spatial / temporal masking as shown in FIG. 2. The reference image / video 201 and the distorted image / video 202 are received by the signal decompression units 203 and 204, respectively. Corresponding decompressed signals 205, 206 are received by contrast gain control unit 207, 208 for spatial / temporal masking of decompressed signals 205, 206, respectively. The corresponding processing signals 209, 210 are processed by the detection and pooling unit 111 and an output signal 212 is generated that represents the distortion value of the distorted image / video 202.

In [4], a neural network is used to measure the quality of an image / video by combining a plurality of visual features as shown in FIG. The reference image / video 301 and the distorted image / video 302 are input to the plurality of feature extraction units 303 and the various features of the image / video 301, 302 are transferred in the plurality of feature extraction units 303. Extracted. Features extracted by neural network 304 are received and a distortion value 305 of distorted image / video 302 is generated.

[5] discloses a method for evaluating the perceived quality of a video by assigning different weights to several visual stimuli.

References [4] and [5] are not computationally efficient because they process the entire image or video the same, so that non-essential parts of the image / video are also processed.

In [6] several bottom-up visual stimuli are used to determine areas of high visual attention in the image / video. Features determined from this bottom-up visual stimulus are weighted and accumulated to form a significance map indicating areas of high visual attention. This method only determines bottom-up features, so the quality evaluation of the image / video is not very good. Moreover, areas of high visual attention do not always mean that these areas should be coded with high quality.

[7] discloses a method similar to [6] but using both bottom-up and top-down visual stimuli to determine areas of high visual attention in the image / video. The determined features obtained from bottom-up and top-down visual stimuli are integrated together using a Bayes network that must be trained prior to integration. As mentioned above, an area with high visual attention does not always mean that this area should be coded with high quality. Moreover, the use of Bayesian networks to integrate the features of image / video is complex because the Bayesian network needs to be trained before integrating the features.

Therefore, a more accurate and still powerful method of assessing the quality or distortion of an image or video is desired.

1 is a block diagram illustrating a general purpose non-reference matrix for measuring perceptual image / video distortion.

FIG. 2 is a block diagram showing Winkler's full reference matrix for measuring perceptual image / video distortion.

3 is a block diagram illustrating Yao's full reference matrix for measuring perceptual image / video distortion.

4 shows a general purpose system for monitoring the quality of video for a broadcast system.

5 shows a quality oriented significance map according to the present invention.

6 is a block diagram illustrating a general purpose system for generating a quality oriented significance map according to a preferred embodiment of the present invention.

7 is a view showing a specific embodiment of the quality-oriented significance map according to a preferred embodiment of the present invention.

8 is a block diagram showing a distortion matrix including a quality oriented significance map according to the present invention.

FIG. 9 is a block diagram illustrating a general purpose referenceless matrix including a quality oriented significance map according to the present invention as a general purpose referenceless measure for perceptual image / video distortion of a distortion matrix.

FIG. 10 is a block diagram showing a full reference matrix of a Winkler with a quality oriented significance map according to the present invention as a full reference matrix of the Winkler for measuring perceptual image / video distortion of the distortion matrix.

FIG. 11 is a block diagram showing Yao's full reference metric with a quality oriented significance map according to the present invention as Yao's full reference metric for measuring perceptual image / video distortion of the distortion metric.

It is an object of the present invention to provide a method which can improve the performance of existing methods of assessing the quality or distortion of an image or video.

This object is achieved by the features of the independent claims. Additional features are obtained from the dependent claims.

The present invention provides a method of generating a quality oriented significance map for assessing the quality of an image or video, comprising extracting features of an image or video, determining perceptual quality requirements of at least one extracted feature, and And incorporating features and perceptual quality requirements of the at least one extracted feature to form an array of significance level values to produce a quality oriented significance map.

In addition, at least one of the extracted features is used to determine perceptual quality requirements of the image / video based on the feature. In other words, the importance of the quality of the image / video perceived by the viewer is determined based on the extracted features.

The significance level values obtained by integrating the extracted features with the perceptual quality requirements of the at least one extracted feature form 3-D for the image and 4-D for the video. This array of significance level values is used as a quality oriented significance map for assessing the quality or distortion of an image or video.

It should be noted that certain areas of high visual attention in the image / video do not always correspond to the same high-quality areas of the image / video. In other words, a particular area of high visual attention in the image / video does not always require that such area of the image / video be encoded in high quality, and vice versa.

Since perceptual quality information is used to determine significance level values, the resulting significance map closely follows the perceptual quality requirements of the image / video. Therefore, a more accurate significance map for evaluating the quality of an image or video is obtained, compared to any of the prior art that uses a map and based only on visual attention.

The significance map generated by the present invention can be used for both the first and second class approaches in existing distortion metrics, thereby improving the accuracy of the image / video quality estimation process.

According to the present invention, features of an image or video are extracted using visual feature based information and knowledge based information. In other words, both bottom-up process (visual feature based) and top-down process (knowledge based) are used. These processes determine which features of an image / video cause visual attention, and consequently extract those visual attention features. Extracted features include motion, lighting, color, contrast, orientation, texture, and so on. Existing image / video descriptors, for example MPEG-7 descriptors, may be used.

According to the present invention, object motion in a video or sequence of images is separated into a relative motion vector and an absolute motion vector. Relative motion is object motion compared to background or other objects, and absolute motion is the actual motion of an object in an image or video frame. Based on the determined relative and absolute motion vectors, the quality level value of the object (pixels or region) is determined. The determined quality level value is integrated with other extracted features from the image / video to form an array of significance level values.

Object motion analysis can be separated into two stages: global motion estimation and motion mapping. Global motion estimation provides motion estimation of an image or video camera, and motion analysis extracts the relative and absolute motion vectors of the object.

It should be noted here that other features of the image / video can be used to determine quality level values of pixels or regions of the image / video. Examples of other features are face detection, human detection and texture analysis. The quality level values thus determined from these other features may be integrated with the quality level values obtained from the motion analysis to produce a quality oriented significance level.

According to a preferred embodiment of the present invention, all extracted features and determined quality level values of at least one feature are integrated to form an array of significance level values using a nonlinear mapping function.

When a nonlinear mapping function is used, it has the advantage of reducing computational requirements and simplifying the implementation. In addition, algorithms or systems for nonlinear mapping functions do not require training, as opposed to the Bayesian network used in the system described in [5].

It should be noted here that in variant embodiments other techniques similar to neural networks or fuzzy rules may be used to integrate the extracted features with the determined quality level values of the at least one feature.

According to another preferred embodiment of the present invention, the coupling effects as a result of the integration of the extracted features are taken into account when forming an array of significance level values. Due to the use of the combining effect, other extracted features that can be treated as significant effects are not integrated by adding them in a linear manner. Different combinations of extracted features result in different coupling effects.

Specifically, the quality-oriented significance map according to another preferred embodiment of the present invention is represented by the following equation

It can be obtained using the above formula, Is an element of the quality oriented significance map at scale (s), position (i, j) and time (t), Is the nth extracted feature, Is and A coupling coefficient representing the coupling effects of combining n is the index of the extracted feature; k is another index of the extracted feature such that 1 <k <N and k ≠ L; N is the total number of features extracted; g _l is a nonlinear joint mapping function defined by g ₁ (x, y) = min (x, y); And L equals L = arg max ( Represented by) Is the maximum value.

In a preferred variant embodiment of the invention, the integration of the extracted features is performed by applying a nonlinear mapping function to the sum of the weighted features.

Specifically, the quality-oriented significance map according to a preferred modified embodiment of the present invention is represented by the following equation

Is obtained using, where g ₂ is a nonlinear mapping function, which is

Wherein α is a parameter for providing nonlinearity and c is a constant.

According to a preferred embodiment of the present invention, the method for generating a quality oriented significance map further comprises a post processing step for processing the generated quality oriented significance map. The post processing step improves the quality of the generated significance map by removing any noise that may be present. The post-processing step may also be used for other tasks, including smoothing or enlarging the significance map and removing artifacts present in the significance map.

In particular, a Gaussian smoothing technique is used to remove impulse noise caused by errors during extraction of features according to a preferred embodiment of the present invention.

The above-described embodiments of the present invention apply not only to methods but also to apparatuses, computer readable media, and computer programs.

4 shows a general purpose system for monitoring the quality of video for a broadcast system.

The image / video source 401 is encoded in the encoder unit 402 and the encoded image / video 403 is transmitted over a wide area communication network (WAN) 404. The transmitted image / video 405 from the WAN 405 is received by the decoder unit 106 and decoded.

Distortion matrix unit 410 receives decoded image / video 407 from decoder unit 406 with / without reference image / video 408. The reference image / video 408 is generated from the image / video source unit 409, which is the same as the image / video source unit used to generate the original image / video 401. Alternatively, the reference image / video 408 may simply be extracted from the original image / video 401 encoded in the encoder unit 402.

In general, distortion metrics can be classified into full reference (FR), reduced reference (RR), and no reference (NR) models. The FR matrix performs a pixel-by-pixel and frame-by-frame comparison between the distorted image sequence and the corresponding undistorted corresponding portion. The reduced reference matrix calculates some statistics from the distorted image sequence and compares them with the corresponding stored statistics of the undistorted image sequence. Statistical data are selected and correlated by conventional regression analysis. Unreferenced metrics perform feature extraction on the distorted sequence without requiring any information from the undistorted image sequence to find artifacts such as MPEG block boundaries, pointed noise, or image blur.

The distortion matrix unit 410 compares both the reference image / video 408 and the decoded image / video 407 (for FR and RR) or analyzes artifacts of the decoded image / video 407 ( NR) to assess the quality of the decoded image / video 407. The output signal 411 generated by the distortion matrix unit 410 represents the quality of the decoded image / video 407.

The distortion matrix unit 410 may be implemented using the distortion matrix as described above (see FIGS. 1, 2 and 3).

According to the present invention, a quality-oriented significance map, in particular a hierarchical quality-oriented significance map (HQSM), is proposed. The HQSM is generated based on both visual attention and perceptual quality requirements of the image / video.

In the bottom-up process, a map of significance level values of pixels or regions of an image / video may be determined from visual features based on several rules:

1. The observer's eye position is not always fixed in areas of high visual attention, and eye movements follow areas of high visual attention;

2. Other features of the image / video are not added linearly to provide a cumulative effect;

3. The observer's eyes see the world outside the concentration or attention zone;

4. The selection of image / video features may be space based or object based; And

5. The integration and selection of the stimuli that cause visual attention depends on the hierarchical structure of the "winner-takes-all" process.

Note that the last rule 5) applies only to the same observer at any given moment. For a group of observers, the attention area can be represented by a statistical map. Even for a single observer, more significant regions than one significant region are obtained when the image / video is observed over a period of time. The significant regions may be represented by a statistical map.

For the top-down process, another map of significance level values of pixels or regions of the image / video may be defined using active regions or prior knowledge from other media. For example, the audio sound of an airplane will cause the observer to focus his attention on the plane object of the image / video.

The significance maps generated above are integrated to form the HQSM. The HQSM according to the invention is a three dimensional array for images, or a four dimensional array for video, as illustrated in FIG.

HQSM can be represented using the following equation:

Where M stands for HQSM, Represents the map elements of the HQSP at scale (s), position (i, j) and time (t), W _s is the width of one frame or image of the video, and L _s is the height of one frame or image of the video And N _t is the time interval of video (applicable only to video).

Map elements ( A high value of) represents a high level of significance of a pixel or region of the image / video, and a high weight should be assigned to the distortion measurement of its corresponding pixel or region and vice versa.

The creation of the HQSM includes three steps as shown in FIG. The visual features of the image / video 601 are extracted in the feature extraction unit 602 based on the following stimuli:

1. Visual attention stimuli, eg movement, lighting, color, contrast, orientation, texture, etc.

2. knowledge-based stimuli, eg face, person, shape, etc.

3. Custom stimuli

It should be noted here that existing image / video descriptors, such as MPEG-7 descriptors, may be integrated to extract features.

The extracted features 603 are received by the decision unit 604 for incorporating the extracted features 603 to generate an overall HQSM 605. The overall HQSM 605 is further processed by a post processing unit 606 that improves the quality of the overall HQSM 605 to produce the final HQSM 607 according to a preferred embodiment of the present invention.

7 is a diagram showing in detail the generation of the HQSM according to a preferred embodiment of the present invention. Other features extracted in accordance with a preferred embodiment of the present invention are summarized below.

Motion analysis

Object motion in a sequence of video or images may be separated into two vectors, a relative motion vector and an absolute motion vector. Relative movement is the movement of an object relative to the background or other objects. Absolute movement is the exact position moved within the image or video.

Motion analysis can be separated into global motion estimation for determining the motion of the camera used for the image / video and motion mapping for extracting relative motion vectors and absolute motion vectors.

Global motion estimation is

Where (ΔX, ΔY) is the estimated motion vector of the pixel or region (X, Y) of the video, C _f is the zoom factor, and (t _x , t _y ) is the translation vector It can be estimated using a three-parameter method.

Note that the estimated motion vectors ΔX and ΔY are also absolute motion vectors.

The three-parameter method is preferred because the three-parameter method is less susceptible to noise than other modeling methods, such as a six-parameter model or a four-parameter model.

The error of global motion estimation can be defined as follows:

The values of C _f , t _x and t _y can be obtained by minimizing the three equations as given below:

A mitigation algorithm can be used to determine the minimized values of C _f , t _x, and t _y , which are summarized in the following steps:

1. Select pixels or areas with large changes in the image / video;

2. Determine (C _f , t _x , t _y ) that satisfies Equation 4 among the selected pixels;

3. Using Equation 3, the error? Is obtained for every pixel.

4. Select the pixels of the image / video within the error range ([ε-Δ, ε + Δ]).

5. Repeat Step 2 and Step 3 until (C _f , t _x , t _y ) is less than the predetermined value.

After (C _f , t _x , t _y ) is obtained, the relative motion can be determined using the following equation:

The relationship between the attention level and the relative motion vector is a nonlinear monotonically increasing function. Attention levels increase as relative movement increases. When relative motion reaches a certain value, the attention level does not increase with any additional relative motion increase. Therefore, the relationship between relative motion vector and attention level

Can be represented by the formula, wherein x _r is Is a relative motion vector, where a and b are parameters where a> 0, b <1 and a · 10 ^b = 1.

Similarly, the relationship between attention level and absolute movement is a nonlinear function. As absolute movement increases, the level of attention increases correspondingly and then decreases. The relationship between the attention level and the absolute motion vector can be defined as follows:

Wherein x _a is Is the absolute motion vector defined by Are the parameters assigned to be

As can be seen from equation (7), when x = 1 / d, f _a (x) is the maximum, so c = de.

Thereafter, the overall movement attention level can be determined as follows.

The relationship between relative motion, absolute motion, attention level and perceptual quality level can be summarized as shown in Table 1 below.

Relative movement Absolute movement Attention level Quality level lowness lowness lowness lowness height lowness height height lowness height lowness lowness height height height middle

As can be seen in Table 1, an object with a high absolute motion attracts the viewer's visual attention. However, the observer will not care about the quality of the object. For example, an observer may follow a flying ball's movement in a video or sequence of images, but does not pay much attention to the shape (quality) of the flying ball. When the relative movement of the flying ball is high, the absolute movement is low, and the observer pays much attention to the shape (quality) of the flying ball.

It is important to note that the level of attention is not always the same as the perceived quality level. In addition to the level of visual attention, perceptual quality requirements are also used to form an array of significance maps according to the present invention, so HQSM is more accurate and powerful in assessing the quality of images / videos compared to any of the prior art.

Lighting mapping

High illumination or contrast in the area of the image / video usually produces high visual attention. For example, spotlight lighting on the stage draws the viewer's visual attention. Illumination can be estimated by applying a Gaussian smoothing filter to the image / video. Other lighting estimation means may be used.

Color mapping Of Skin color mapping

Color mapping is similar to lighting mapping except that differences in values between different regions of the image / video may be used to determine the value of the current pixel or region.

Skin color attracts visual attention in many situations and detection of skin color can be performed in the Cb-Cr active area. In particular, a lookup table may be used to assign possible color values to each pixel or region of the image / video. Skin color is detected when the pixel value falls within the range of 77 <Cb <127 and 133 <Cr <173.

Face detection

Face detection is the detection of facial regions from an image / video, where a human face is likely to cause a high visual attention of the observer. Skin color and shape information is useful for face detection.

Eye / mouth detection

In the face area, the eyes and mouth of the face usually attract higher visual attention than other parts of the face. Face detection and shape information can be used for eye / mouth detection.

Geometry Analysis Mapping

Shape analysis is useful for determining object shapes in images / videos that cause visual attention. Information about shape analysis is also useful for detecting other information such as face, eyes / mouth, metabolic subtitles and the like. Shape analysis can be performed by applying a Watershed algorithm to an image / video frame and dividing the image into smaller regions. Merge-division methods and shape description / clustering methods as mentioned in [9] can be performed to determine the shape of the objects in the image / video.

Human body detection

Human body detection is possible using information obtained from shape analysis, face detection and eye / mouth detection.

Caption detection

Subtitles in images / videos have important information and therefore have high visual attention. Subtitles can be detected using the method as disclosed in [8].

Texture Analysis Of Mapping

Texture negatively affects the overall value of the significance level value, and consequently the generated HQSM. In other words, the texture reduces the overall value of the map elements of the HQSM. Specifically, it represents the texture characteristics of the image / video. For

Since the negative effects of the texture of the image / video are taken into account, considering the texture features when forming an array of significance level values, the overall accuracy of the significance map is increased. Therefore, the HQSM generated by the present invention has a high accuracy compared to the significance map generated by any of the prior art.

Custom attention

In this feature, the significance level of some or all pixels or regions of an image / video is artificially defined based on other information such as audio, intentional attention to a particular object, and the like.

It should be noted here that only a few of the feature extracts are mentioned but the invention is not limited to specific feature extraction methods and that other features of the image / video can be further included in the method of generating HQSM according to the invention. will be.

After all the features of the image / video are extracted, they are integrated in decision unit 604. According to a preferred embodiment of the present invention, a nonlinear mapping function is used to integrate the extracted features.

The combining effects as a result of combining any pair of extraction features are different, and a model for integrating a pair of extracted features taking into account the combining effect is given by

In the above formula, Is an element of the quality oriented significance map; c ¹² is a binding coefficient representing the binding effect; Wow Is a pair of extracted features; n is the nth extracted feature; And g ₁ represents a nonlinear function.

This nonlinear function may preferably be defined as follows.

In another preferred embodiment of the present invention, three or more extracted features are integrated using the following equation.

In the above formula, Is the n th extracted feature, and c ^Lk is And A binding coefficient indicating the binding effect to combine; n is the index of the extracted feature; k is another index of the extracted feature such that 1 <k <N and k ≠ L; N is the total number of features extracted; And L is The maximum value of and is expressed as:

It should be noted from Equation 12 that only the coupling effect between the extracted feature having the maximum value and other extracted features is considered. Combined effects between other extracted features are ignored.

In a preferred variant embodiment of the invention, the integration of the extracted features is performed using the following equation:

Where w ₁ , w ₂ , ..., w _n are the weights of the extracted features and g ₂ is a non-linear mapping function.

This nonlinear mapping function is preferably as follows.

Where α is a parameter with a value of α = 2 that satisfies nonlinearity, and C is a constant with a value of C = 1 that allows the observer's eyes to see the world outside the concentration or attention zone.

In alternative embodiments, other techniques, such as neural networks, fuzzy rules, may be used to integrate the extracted features to form the significance map 605.

The significance map 605 generated from the integration of the extracted features is received by the post processing unit 606 which further improves the quality of the generated significance map 605 to form the final HQSM 607.

In post processing unit 606, a Gaussian smoothing filter may be applied to generated significance map 605 to remove impulse noises caused by errors in feature extraction process 602.

The HQSM generated by the present invention may be applied to both the first class of approaches and the second class of approaches. Specifically, the HQSM may be integrated into the MSE as given by

In the above formula, Is a modified MSE that incorporates HQSM.

The PSNR as a result of the modified MSE is consequently given as follows.

In the above formula, Is the modified PSNR value incorporating HQSM.

The HQSM can be applied to any existing distortion matrix after being generated in accordance with the present invention to improve the accuracy of distortion measurement or quality assessment.

8 shows how the generated HQSM 801 can be incorporated into an existing distortion matrix 802. It should be noted here that the HQSM 801 is separated from the processing of the image / video by the distortion matrix 802 and the outputs from the HQSM 801 and the distortion matrix 802 are integrated in the integration unit 803.

9, 10 and 11 show how the HQSM can be applied to the distortion matrix as shown in FIGS. 1, 2 and 3. Since the application of the HQSM to the distortion matrix is independent of the processing of the image / video by the distortion matrix, the HQSM may be applied to the distortion matrix at any stage of the quality assessment / distortion measurement process (as indicated by the dashed line).

Experiments are performed to determine the performance of the HQSM and the existing distortion matrix according to the present invention.

In these experiments, HQSM is generated using features extracted based on light mapping, motion analysis, skin color mapping and face detection. The generated HQSM is applied to the PSNR method, and the distortion metrics are disclosed in [1] (Wang's matrix) and [2] (Winkler's matrix). Two video sequences, labeled "half" and "autumn leaf," are used as test video sequences for evaluation of the quality of the video sequences.

The results of the above experiments are summarized in Table 2.

Distortion matrix PSNR HQSM based PSNR King's matrix HQSM-based king metrics Winkler's Matrix HQSM-based Winkler Metrics harp 0.8118 0.85 0.6706 0.6853 0.6912 0.7412 Fall leaves 0.1324 0.5441 0.9324 0.9265 0.8235 0.8647

As can be seen from the results in Table 2, the distortion matrix with integrated HQSM provides better performance in video quality evaluation. The only exception is the king's metric on the video sequence "autumn leaves".

The reason for this is due to the high Spearman correlation for the video sequence "autumn leaves". In addition, the value of the quality level for "autumn leaves" using the King's matrix is already very high (maximum is 1), and therefore the subjective grading of video sequences by a group of observers varies greatly in this case.

Therefore, the HQSM generated by the present invention can improve the performance of existing video quality estimation methods.

The above-described embodiments of the present invention apply not only to a method but also to an apparatus, a computer readable medium, and a computer program.

Although embodiments of the present invention have been mentioned so far, such embodiments are merely illustrative of the principles of the present invention. Other embodiments and configurations may be devised without departing from the spirit of the invention and the scope of the appended claims.

The following documents are mentioned herein.

[1] Z. Wang, A.C. Bovik, "A universal image quality index", IEEE Signal Processing Letters, Vol. 9, No. 3, March 2002, Pg. 81-84.

[2] Z. Wang, H. R. Sheikh and A. C. Bovik, "No Reference perceptual quality assessment of JPEG compressed images", IEEE International Conference on Image Processing, 2002.

[3] Stefan Winkler, "Vision Models and Quality Metrics for Image Processing Applications", ph.D. Thesis # 2313, Swiss Federal Institute of Technology, Lausanne, Switzerland, 2000.

[4] S. Yao, et al, "Perceptual visual quality evaluation with multi-features", submitted to IEE Electric Letters.

[5] WO 99/21173

[6] US Patent Publication No. 2000/0126891

[7] EP 1 109 132

[8] US 6243419

[9] Miroslaw Bober, "MPEG-7 Visual Shape Descriptors", IEEE Transaction on circuits and systems for video technology, Vol. 11, No. 6, June 2001.

Claims (15)

In the method for generating a quality-oriented significance map for evaluating the quality of an image or video, Extracting features of an image or video; Determining perceptual quality requirements of the at least one extracted feature; And Generating a quality directed significance map by integrating the extracted features with the perceptual quality requirements of the at least one extracted feature to form an array of significance level values. The method of claim 1, wherein the features of the image or video are extracted using visual feature-based information and knowledge-based information. 3. The method of claim 2, wherein absolute motion and relative motion are determined and used to determine a quality level value of a pixel or region of an image or video, wherein the determined quality level value is a perceptual quality requirement used to generate a quality oriented significance map. A method of generating a quality oriented significance map, characterized in that. The method of claim 2, wherein the extracted features and perceptual quality requirements of the at least one extracted feature are integrated to form an array of significance level values using a nonlinear mapping function. . 5. The method of claim 4, wherein the combining effects as a result of the integration of the extracted features are used when forming an array of significance level values. 6. The quality oriented significance map of claim 5, wherein Obtained using the above formula, Is an element of the quality oriented significance map at scale (s), position (i, j) and time (t); Is the nth extracted feature; c ^Lk is and Is a coupling coefficient representing the coupling effects of combining N; n is the index of the extracted feature; k is another index of the extracted feature such that 1 <k <N and k ≠ L; N is the total number of features extracted; And L is Indicated by And a maximum value of. 7. The non-linear coupling mapping function of claim 6 wherein Method of generating a quality-oriented significance map, characterized in that defined by. The method of claim 4, wherein the incorporation of the extracted features is performed to determine weights for each of the extracted features, add weighted extraction features, and apply a nonlinear mapping function to the accumulated features, thereby providing a visual significance level value. And a method for generating a quality oriented significance map. 10. The method of claim 8, wherein the quality oriented significance map is Obtained using the above formula, Is an element of the quality oriented significance map at scale (s), position (i, j) and time (t); Is the nth extracted feature; n is the index of the extracted feature; And g ₂ is a non-linear mapping function. 10. The nonlinear mapping function of claim 9 wherein As defined above, wherein α is a parameter for providing nonlinearity, and and c is a constant. The method of claim 1, wherein the generated quality oriented significance map is further processed in a post processing step to improve the quality of the generated quality oriented significance map. 12. The method of claim 11, wherein the post processing step is performed using a Gaussian smoothing technique. An apparatus for generating a quality oriented significance map for evaluating the quality of an image or video, the apparatus comprising: A feature extraction unit for extracting features of an image or video; A judging unit for determining the perceptual quality requirement of the at least one extracted feature; And Generating a quality oriented significance map comprising an integrated unit for integrating the extracted features with the perceptual quality requirements of the at least one extracted feature to form an array of significance level values, thereby generating a quality oriented significance map. Device. A computer readable medium having a program recorded thereon, wherein the program causes a computer to execute a procedure for generating a quality oriented significance map for evaluating the quality of an image or video. The procedure is Extracting features of an image or video; Determining perceptual quality requirements of the at least one extracted feature; And And incorporating the extracted features and perceptual quality requirements of the at least one extracted feature to form an array of significance level values, thereby generating a quality oriented significance map. A computer program element in which a computer executes a procedure to generate a quality oriented significance map for evaluating the quality of an image or video. The procedure is Extracting features of an image or video; Determining perceptual quality requirements of the at least one extracted feature; And And incorporating the extracted features and perceptual quality requirements of the at least one extracted feature to form an array of significance level values, thereby generating a quality oriented significance map.