WO2012162878A1

WO2012162878A1 - Method and device for determining saliency value of current block of image

Info

Publication number: WO2012162878A1
Application number: PCT/CN2011/074911
Authority: WO
Inventors: Xiaodong Gu; Ning Liao; Zhibo Chen
Original assignee: Technicolor China Technology Co Ltd
Current assignee: Technicolor China Technology Co Ltd
Priority date: 2011-05-30
Filing date: 2011-05-30
Publication date: 2012-12-06
Anticipated expiration: 2013-11-30

Abstract

A method and a device for determining a saliency value of a current block of an image are disclosed.　The described method comprises the steps of determining, block-wise for each block in the image, features, wherein the features of a given block comprise sub--ranges of overall value ranges of DCT coefficients into which the values of said DCT coefficients fall for at least one of said given block and at least one reference image block used for predictive encoding of the said given block, determining frequencies of occurrence, in the image, of the features determined for the current block and determining the saliency value of the current block using the determined frequencies of occurrence of the features determined for the current block. This approach has the advantage of computational efficiency in relying on DCT transform coefficient values only. Occurrence frequencies based analysis further provides stable results.

Description

METHOD AND DEVICE FOR DETERMINING SALIENCY VALUE

CURRENT BLOCK OF IMAGE

TECHNICAL FIELD The invention is made in the field of saliency value determination .

BACKGROUND OF THE INVENTION

Video delivery over noisy or lossy transmission paths, such as IP networks, leads to quality degradation. Video quality assessment models for IP network can be addresses at media-layer, at packet-layer, or at bitstream- layer. Bitstream- layer models can be based on parameters extracted from the transport stream in the bitstream with no or little decoding. Bitstream models have the additional advantage of lower data rates and thus lower bandwidth and processing requirements compared to metrics looking at the fully decoded video. Using such bitstream based metric, it is thus possible to measure the quality of many video streams in parallel. Unpublished PCT application 2010/0000603 proposes an example of such a bit stream model to assess video quality loss over IP network caused by packet loss during

transmission .

The effect of packet loss on the quality of a video is stronger when the packet loss occurred in a block comprised in an extremely salient area, while not being so strong in when the packet loss occurred in a block comprised in a medium salient areas and weak or even ignorable for packets lost in non-salient areas. United States Patent Application 2008152245 mentions a rule based bit stream domain region-of-interest (ROI) extraction using motion vectors and coefficient characteristics such that a ROI map may be formed by assigning, using a

predefined rule, to individual blocks 1 or 0 to designate ROI-blocks and non-ROI -blocks , respectively.

Summary of the Invention

The inventors recognized that, since the prior art ROI extraction is rule-based, it is not always stable. The inventors therefore propose a method for determining a saliency value of a current block of an image encoded according to H.264 according to claim 1 and a corresponding device according to claim 2.

Said method comprises determining, block-wise for each block in the image, features, wherein the features of a given block comprise sub-ranges of overall value ranges of DCT coefficients into which the values of said DCT coefficients fall for at least one of said given block and at least one reference image block used for predictive encoding of the said given block. The method further comprises determining frequencies of occurrence, in the image, of the features determined for the current block. And, the method also comprises determining the saliency value of the current block using the determined frequencies of occurrence of the features determined for the current block.

This occurrence frequencies based approach has the

advantage that it is computationally very efficient since only the DCT transform coefficients of the video stream are required. Further since occurrence frequencies are

analysed, the determined saliency value is more stable.

The features of further advantageous embodiments are specified in the dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

Exemplary embodiments of the invention are illustrated in the drawings and are explained in more detail in the following description. The exemplary embodiments are explained only for elucidating the invention, but not limiting the invention's disclosure or scope defined in the claims .

In the figures:

Fig. 1 shows a flow chart of an exemplary embodiment of the invention

EXEMPLARY EMBODIMENTS OF THE INVENTION

The invention may be realized on any electronic compress- image receiving or transmitting device comprising a

processing device correspondingly adapted. For instance, the invention may be realized in a set top box, a

television, a mobile phone, or a personal computer wherein this listing is exemplarily. According to the human activity analysis, image areas which are outstanding their surrounding are easy to catch a viewer's attention at first sight. Therefore, an exemplary embodiment of the proposed method comprises the following steps: Extracting representative features to which viewer's attention is sensitive, the feature extraction is based on blocks of predetermined size, e.g. 4x4 blocks or 8x8 blocks; generating histograms for the extracted features and statistically estimate the occurrence frequencies;

combining the statistical results for all the features and getting a value which identifies an attention level over the frame. Independent orthogonal transforms, such as the discrete cosine transform (DCT) help well in extracting content features. In bitstream, DCT coefficients can be parsed easily by de-entropy coding (a look-up dictionary

operation) . The DCT coefficients have a value range which is divided into sub-ranges. Each DCT coefficient has a value which falls in exactly one sub-range. These subranges into which the DCT coefficients of a block fall are considered the spatial features of the block.

Therefore, in the exemplary embodiment only the DCT

coefficients are required for determining the features representative of the spatial content of the frame. That is, none of the frame pixel is decoded for obtaining the spatial features, only fast de-entropy coding is required to get DCT coefficients.

In H.264, the DCT operations are applied on 4x4 blocks. Then there are 16 DCT coefficients for each block: DC= ACO, AC1, AC2 , AC15. The sub-ranges into which these

coefficients fall compose the 16 spatial features of the block: f1 ... f16.

For I-frames, i.e. frames encoded using intra-prediction without reference to another frame, the DCT coefficients can be determined easily, while for P-frames and B-frames determining the DCT coefficients requires recursion. That is, blocks of P-frames and B-frames use motion vectors (MV) to reference blocks from other frames, the referenced block being used as predictors. Since the referenced blocks are selected such that they most closely resemble the blocks referencing them, spatial features of a referenced block are a good approximation or prediction of spatial features of a block predicted.

Since a reference block needs not to be aligned with blocks into which the frame containing the reference block is partitioned, the reference block may comprise parts of up to four partitioning blocks. But attentiveness values are calculated only for partitioned blocks. Therefore, the spatial features of the reference block is then defined as a weighted average of the spatial features of all the partitioned blocks intersecting with the reference block, wherein the weighting is according to the size of the respective intersection.

If the reference frame is an I-frame, the approximate spatial features are accessible directly. But, if the reference frame is still a P- or B-frame, prediction of the spatial features requires recursion until reaching a reference frame which is an I-frame.

In an exemplary embodiment, for P- frames and B- frames length and direction of the motion vector of a block are used for determining additional temporal features of this block. The motion vector can be derived directly from bitstream. Motion vectors can be estimated based on larger size such as 16x16. In this case, each 16x16 block can be divided, for instance into sixteen 4x4 sub-blocks wherein each sub-blocks' motion vector is set to the motion vector of the 16x16 block. As with DCT coefficients, length and direction of the motion vector have a value range which is divided into sub-ranges wherein the number of sub-ranges may differ for length and direction and further differ from the number of sub-ranges into which the DCT coefficients' value range is partitioned. Each motion vector length falls in exactly one length sub-range and each motion vector direction falls in exactly one orientation sub-range. These sub-ranges into which motion vector length and direction of a block fall are considered the temporal features of the block .

Blocks in P- or B- frames can be encoded in INTRA-mode. Such blocks do not have a motion vector. In order to allow for comparability of INTRA-mode encoded blocks of P- or B- frames with not INTRA-mode encoded blocks of the same frame, INTRA-mode encoded blocks can be provided with a virtual motion vector derived from motion vectors of adjacent blocks. For instance, motion vectors of adjacent blocks can be used for calculating a virtual motion vector for INTRA-mode encoded blocks in P- or B- frames, e.g. by determining an average or a weighted average of the motion vectors of adjacent blocks.

More generally speaking, there are N blocks in a frame and F different block data components comprised in the bit stream description of each block. Each of these j=l...F different block data components has an individual,

component-specific value range R_j which is divided into sub- ranges wherein the number of sub- ranges may vary in particular for block data components of different type. The sub- range R_j (B±) in which the j^'-th block data components fall for block B± is considered the j^'-th feature of this block B±. In the following, #R_j (B±) denotes the total number of blocks of a frame having said j^'-th feature of block B±. #R_j (B_±) /N is called the weight of the j^'-th feature of block Bi .

The smaller #R_j (B±) , or #R_j (B±) /N, is the more unique is the occurrence of the j^'-th feature in block B_± with respect to the frame and therefore the more outstanding or salient is block B_± due to the j^'-th feature. That is, the more likely this feature catches viewer's attention. Therefore,

#R_j (Bi) /N is considered an exemplary measure for

determination of the contribution of the j^'-th feature of block Bi to the overall saliency or attentiveness of block Bi.

Since each feature's contribution to the saliency of a block can be considered independent from the contribution of other features, AV{B₁ ) =

)l N) ±S considered an exemplary measure for determination of the overall saliency of block Bi . In an exemplary implementation the following attentive value AV(Bi) of block B_± is determined:

AV{B, ) = - log(¾ (#R_J (5, )/N))=F* log(N) -∑ ^ log(#R,{B_t ))

In an exemplary application scenario of quality assessment for videos on lossy or noisy IP networks a few, e.g. 3 to 5, different attentive levels are sufficient. Therefore, in an embodiment the attentive values are normalized into a discrete set of c=5 different attentive levels. The

normalization operation can be implemented using

traditional classification algorithms resulting in:

An exemplary embodiment of the device for determining a saliency value receives, in a video bitstream, compressed data of a video sequence as an input .

The input bitstream first is passed to a parser module PARS, in which some low complexity pre-processing

operations are applied. For example, motion vectors are picked out and de-entropy coding is applied to get the DCT coefficients .

The output of the parser module PARS is passed to feature extraction module FEEX which, in this exemplary embodiment, extracts 16 spatial features and 2 temporal features for every 4x4 blocks of the frame.

The extracted features are output by module FEEX and received as input by a statistics module STAT. In

statistics module STAT, for each of these extracted

features a histogram is generated. The histograms are then used for determining an attentive weight for each feature from which an overall attentive weight is determined for each 4x4 block of the frame by multiplication of the weights of the features of each block. The block's

attentive weight is output to refinement module FINE in which the block's attentive weight is classified as

belonging to one of three different classes wherein each of the different classes is associated with a different saliency value. In another embodiment, the block's

attentive weight is classified either as belonging to a region-of -interest or as not belonging to a region-of- interest.

In the proposed scheme, a saliency map is extracted in linear time. It is of high computing efficiency which meets the requirement of bitstream- layer quality assessment model. Only a small additional space is required to store the DCT coefficients of the reference frame and the current frame .

Claims

Method for determining a saliency value of a current block of an image block-wise predictive encoded using DCT transform, comprising:

Determining, block-wise for each block in the image, features, wherein the features of a given block comprise sub-ranges of overall value ranges of DCT coefficients into which the values of said DCT

coefficients fall for at least one of said given block and at least one reference image block used for predictive encoding of the said given block,

determining frequencies of occurrence, in the image, of the features determined for the current block and determining the saliency value of the current block using the determined frequencies of occurrence of the features determined for the current block.

Method of claim 1, wherein the blocks of the image are encoded in a bitstream according to H.264 and the DCT transform coefficient values are parsed from the bitstream by de-entropy coding using a look-up

dictionary .

Method of claim 1 or 2 , wherein the features of the given block comprise sub-ranges of overall value ranges of length and direction of a motion vector referencing the at least one reference image block.

Method of claim 3, wherein, in case the given block is intra-coded, a motion vector of length Zero is assigned to the given block. Method of one of the claims 1-3, wherein, in case the given block is encoded in SKIP-mode, only DCT

transform coefficient values of the at least one reference block are used for determining the features of the given block.

6. Method of one of the claims 1-3, wherein, in case the given block is inter-encoded, DCT transform

coefficient values of both, the current block and of the reference image block, are used for determining the features of the given block.

Method of one of the claims 1-6, wherein the method comprises using the saliency value determined for the current block for classifying whether the current block belongs to a region-of-interest .

8. Device for determining a saliency value of a current block of an image block-wise predictive encoded using DCT transform, comprising:

Means for determining, block-wise for each block in the image, features, wherein the features of a given block comprise sub-ranges of overall value ranges of DCT coefficients into which the values of said DCT coefficients fall for at least one of said given block and at least one reference image block used for predictive encoding of the said given block,

Means for determining frequencies of occurrence, in the image, of the features determined for the current block and

Means for determining the saliency value of the current block using the determined frequencies of occurrence of the features determined for the current block .