WO2006123310A2

WO2006123310A2 - Video encoder using a dependency map

Info

Publication number: WO2006123310A2
Application number: PCT/IB2006/051582
Authority: WO
Inventors: Cécile DUFOUR; Nicolas Vanhaelewyn
Original assignee: Koninklijke Philips Electronics N.V.
Priority date: 2005-05-20
Filing date: 2006-05-18
Publication date: 2006-11-23
Also published as: WO2006123310A3

Abstract

The present invention, mainly applicable in streaming and broadcast for TV, video on demand and storing systems, relates to a video encoder (ENC) for encoding frames (FR) of a video signal before transmission, said video encoder (ENC) including a bit rate allocation unit (BRAU) for allocating a bit rate to encoded data in a encoding unit (ENU). According to the invention, this video encoder (ENC) further implements a dependency map calculation unit (DMCU) which calculates, for each data of said frame (FR), a corresponding dependency value (DV) taking into account Intra and Inter prediction mechanisms. This dependency value (DV) expresses on which degree said data will be used in further encoding, the dependency map (DM) being used by the bit rate allocation unit (BRAU).

Description

VIDEO ENCODER USING A DEPENDENCY MAP

FIELD OF THE INVENTION

The present invention relates to a video encoder for encoding frames of a video signal before transmission, said video encoder including a bit rate allocation unit for allocating bit rate to encoded data of said frame in a encoding unit.

BACKGROUND OF THE INVENTION

The quality of a digital video encoder is highly dependent upon the bit rate strategy that it applies. Different clues are taken into account to realise this bit rate allocation so as to optimise the video quality output by a video encoder. Some of these clues include texture and motion parameters, combined with psycho-visual elements.

Such a video encoder is known from the patent application US2003/0031128. This document discloses a method for performing adaptive intra refresh (it is here recalled that the word "intra" generally designates a mode according to which all the macroblocks of a frame are processed independently of any other frame, i.e. without any reference to another frame, whereas the word "inter" designates a mode according to which at least another frame is used as a reference, i.e. either a preceding frame, or a subsequent frame, or both). An intercode distortion value and an intracode distortion value are calculated as are an intercode bitrate and an intracode bitrate. Based on a comparison of the calculated intercode distortion value and the intracode distortion value, and on a comparison of the intercode bit rate and the intracode bit rate for each macroblock, a decision is made as to which predicted frame macroblocks are to be intracoded. According to this document, a distortion value is thus calculated from residuals but also from previous distortion data. A given bit rate is accordingly allocated to the data to be encoded in accordance with an optimization of an equation taking into account the distortion and the bit rate. Such a method implies complex calculations of distortion as, for example, in equation 22 presented in the document. Such an approach is probabilistic and presupposes the use of a concealment step in the decoder in order to evaluate the impact of the bit rate allocation. Moreover, it is known that video encoder process highly relies on the proper estimation of redundancies among consecutive images (by using motion estimation), or inside the image (spatial prediction). Each data is encoded to some extent relative to previously encoded data, and this dependency accumulates from one frame to the next. The quality of each encoded area hence propagates throughout the sequences.

SUMMARY OF THE INVENTION

Accordingly, it is an object of the invention to provide a video encoder able to take into account the impact of bit rate allocations on the quality of the video signal when received without engendering complex calculations of distortion. To this end, there is provided a video encoder implementing a dependency map calculation unit for calculating, throughout the time, for each data of said frame, a corresponding dependency value taking into account Intra and Inter prediction mechanisms, said dependency value expressing on which degree said data will be used in further encoding, said dependency map being used by said bit rate allocation unit.

Experiments show that the better is encoded an area in the first place, the better the compression performance on subsequent areas depending on it. On the contrary, an area which does not serve for further prediction is more appropriately encoded poorly, so as to save bit rate for other places. Hence the degree of which data depends on each other is an information a video encoder could benefit from in its bit rate allocation strategy, so as to find the best compromise between quality and bit rate. The invention enables the encoder to know the relative dependency of each encoded data. According to the invention, dependency maps are calculated independently of any distortion and just take into account facts about the prediction mechanisms including Intra encoding and Inter coding that will be realized using said data in further encoding. Thus, such a dependency map takes into account all prediction mechanisms that are likely to use said current data to be encoded. Such a dependency map does not require complex calculation as dependency value are straightforward obtained from what kind of prediction mechanisms are used. The single requirement is that the encoder work with some time delay as there is the need to know to which extend current data will be used in the future, before encoding it. Effectively, the use of dependency map affect the way data are encoded based on future dependency and the encoder needs to work with look-ahead mechanism (forward calculation) or double pass encoding loop (retroactive calculation). Advantageously, the encoder can be an offline encoder. As the bit rate allocation unit has access to said dependency map, it knows which data will be used in the future and can decide to allocate a more or less large bit rate for said current data.

Said data of a frame can be a macroblock, a pixel or a sub-pixel data. Said dependency value may be binary or be a discretized value in a given interval of values.

In an embodiment, said dependency map has the size of a frame. Said dependency value may be dependent on whether intra encoding of subsequent macroblocks are likely to use current macroblock data.

Said dependency value is dependent on whether Inter encoding of subsequent macroblocks are likely to use current macroblock data.

The invention also relates to a method for encoding frames of a video signal before transmission, said method including a bit rate allocation step for allocating a bit rate to encoded data in a encoding step, said method further including a preliminary dependency map calculation step for calculating, throughout the time, for each data of said frame, a corresponding dependency value taking into account Intra and Inter prediction mechanisms, said dependency value expressing on which degree said data will be used in further encoding, said dependency map being used by said bit rate allocation step.

The invention also relates to a computer program product comprising program instructions for implementing, when said program is executed by a processor, an encoding method as disclosed above.

The invention finds applications for instance in streaming/broadcast (film distribution), video on demand, storage systems (DVD) in which a video encoder as described above is advantageously implemented. BRIEF DESCRIPTION OF THE DRAWINGS

Additional objects, features and advantages of the invention will become apparent upon reading the following detailed description and upon reference to the accompanying drawings in which: - Fig. 1 illustrates a video encoder according to the invention,

- Fig.2 gives an example of data of a frame concerned by the invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, functions or constructions well known by the person skilled in the art are not described in detail since they would obscure the invention in unnecessary detail.

Fig.l shows a schematic diagram of a video encoder ENC according to the invention. Said encoder ENC includes an encoding unit ENU for encoding data of a frame FR according to a encoding decision CD received from a bit rate allocation unit BRAU. Said encoding decision CD is taken according to a dependency map DM calculated by a dependency map calculation unit DMCU. Said dependency map DM includes dependency value DV for each data of said frame FR. This calculation unit DMCU is advantageously linked to a memory MEM in order to store said dependency value DV.

The invention thus implements a building of a dependency map DM throughout the encoding process of any bitstream processed by a digital video encoder ENC. The system being double pass, the bitstream is generated at the second pass.

For each new frame to encode, the dependency map calculation unit DMCU first initialises an array having a predefined size. Said array is called the dependency map or the map in the following. Advantageously, the size is chosen in order to be the same as the size of the encoded frame. The size of said map DM may also be chosen to be a multiple or a fraction of the size of the frame. In a general case, any kind of size could be chosen. Such a choice depends on the granularity wished for said dependency map DM. Effectively, depending on the size of said map DM, each element of said map DM corresponds to a more or less extended amount of data in said frame. The dependency map DM can be indifferently stored in a compressed way or not.

Each element of the map DM corresponds to a discretized value from a set of values. Such set can be [0;l] or any set of values defined, for example, by the interval [0;255] or any other interval. For example, in the following, each element of said map DM is initiated at the lowest value and each element is then updated by multiplying by a factor or by adding a factor that takes into account Intra and Inter prediction mechanisms through the encoding process of a data.

Then, for each element of the dependency map DM to which are related corresponding data of the frame, the dependency map calculation unit DMCU calculates to which extent the corresponding data in said frame will be used in further encoding. In order to realize this function, said dependency map calculation unit DMCU receives an encoding information CI from said encoding unit ENU. Such encoding information CI includes information about whether Intra encoding of subsequent macroblocks are likely to use current macroblock data, about whether

Inter encoding of subsequent macroblocks are likely to use current macroblock data.

Several corresponding dependency factors or values are defined as following: α describes a dependency value or factor corresponding to further Intra coding using current data. Several factors can be defined for the different possible dependencies encountered in MPEG-2, MPEG4, H.264.

Thus in the following 0C_dC represents the DC dependency (the DC dependency applies to one fixed macroblock in MPEG-2 (the previously decoded macroblock), while is applies to one macroblock out of two in MPEG-4), α_ac represents the AC dependency (none in MPEG-2, some in MPEG-4 if AC flag is set to one), OCpJ_x represents the pixel dependency encountered only in H.264 (up to 9 predictions directions). β describes a dependency factor corresponding to further Inter coding using current data. Texture and motion dependencies are distinguished and the overall dependency is the sum of the both, with potential weighting factors w_tex and w_mv. Thus β_mv represents the motion dependency when a vector used for Inter encoding of subsequent macroblocks using current data is based on a prediction involving neighbouring vectors (for example, up to 3), β_res represents the texture dependency that concerns P and B frames in MPEG-2 and MPEG-4 and multiple frame reference in H.264. β_res depends also on the residual by being lower when residual is small because sending some texture hides errors.

As illustrated on figure 2a, a macroblock BL₄ of the frame at time t is spread over various pixel (for example 16*16 pixels) and, as illustrated on this figure, can be partly used in further encoding of another macroblock BL₄₊₁ of the next frame displaced in an Inter prediction mechanism. When such a prediction occurs, as represented on figure 2a, the displaced macroblock BL₄₊₁ does not generally recover a macroblock of the frame FR at time t. Thus data DA from the macroblock BL₄ will have a dependency value DV that will differ from the values of other data of the macroblock BL₄.

Thus, as illustrated on figure 2b, the dependency map DM for each block or each group of pixel, as data DA, or each pixel or each sub-pixel estimation inside a macroblock BL₄ can lead to different dependency value DV. This fact is schematically illustrated by only two dependency values FV(BL₄-DA) and FV(DA) for the data of the macrobloc BL₄. Moreover it is noted that the level of granularity of the dependency map DM is defined by its size regarding the size of the frame. On figure 2, the size of the dependency map DM is chosen to be equal to the size of the frame FR. Consequently the granularity of the dependency map DM can be the pixel. The dependency map DM can be calculated in two levels or in multilevel. In such a multilevel approach, the invention assumes that some element of the dependency map DM cannot be affected to a 0 or 1 value and some intermediate discretized levels are defined.

Such intermediate levels are met when data of the frame is likely to be used in further encoding of data having an intermediate corresponding dependency value

DV, when data of the frame is likely to be used in further encoding of data which do not have the same dependency value DV. In this last case, the dependency of the current data is calculated as the pro rate of the dependency value DV on which it relies upon. Examples of the calculation of a dependency value DV for data that will be used in further intra encoding and for data that will be used in further Inter encoding are given below for an image labelled t in MPEG-4 and for dependency value updated by addition of a factor. In the following equations, "+=" is a C language sign that means the accumulation of contributions in the future.

For further intra encoding using current data with coordinates [i j] and DC encoding against the upper neighbour and AC pred flag set to 1, the dependency value DV is:

DV(ij,t)+=α_dc+α_ac+DV(ij+l ,t) with [i,j] refering to the coordinates in the image: for example, the second macroblock of size 16*16 of an image has a pixel coordinates of [16,0].

For further Inter encoding using current data with coordinates [ij], residual R and motion vector MV:

DV(ij,t)+=W_teχ*DV_teχ(i,j,t)+W_mv*DV_mv(ij,t)

Where DV_teχ(i,j,t)+=β_res(MV,R)+DV_teχ(i+MV_x,j+MV_y,t+l) And DV_mv(ij,t)+=β_mv+DV_mv(i- 1 j- 1 ,t)+DV_mv(ij- 1 ,t)+DV_mv(i+l j- 1 ,t). Practically, the dependency value DV of a data is a sum of the dependency values of zones that are dependant on this data.

The bit rate allocation unit BRAU is free to act upon the dependency map FM of each incoming frame. It can apply some strategies among the following: lowering of the bit rate allocated for the encoded data that will not be used in further encoding of subsequent macroblocks, increasing of the bit rate allocated for encoded data that have a high dependency value, signifying that these data will be used in a large extent in further encoding, modification in an encoding decision, for example, choosing an Intra refresh instead of an Inter coding, skip decision (decide to encode or not the residual for a macroblock), filtering out those data which have little impact on subsequent data, realizing unequal error protection (UEP) that give a more important priority on data having high dependency value, realizing forward error correction (FEC) that retransmits parts of the bitstream having high dependency values.

The invention enables a digital video encoder to be capable of improving video compression performance, and/or improving the overall Quality of Service on a video transmission chain, by making use of the dependency map to improve its bit rate allocation strategy. It results in an increase of visual quality, when compared to encoder not equipped with the invention and working at the same bit rate, or in a decrease in bit rate, when compared to an encoder not equipped with the invention working at the same visual quality level.

The invention finds numerous applications among encoders with at least an allowed time delay of one frame. Particularly encoders realizing an offline encoding are concerned. For example, this is the case for content sourcing by DVD authoring. Moreover, applications where a lowering of the bit rate may be advantageous are particularly concerned by the invention. Thus, streaming and broadcast for TV, video on demand, storing systems (DVD) are particularly concerned.

The invention provides the possibility to have a high granularity in terms of resolution and depth as the resolution in the calculation of the dependency map DM can reach sub-pixel accuracy and as the depth can reach few bits representation. Moreover the tracking using the dependency map DM is temporal and takes into account prediction capability of the video standards.

It is to be understood that the present invention is not limited to the aforementioned embodiments and variations and modifications may be made without departing from the spirit and scope of the invention as defined in the appended claims. In the respect, the following closing remarks are made.

There are numerous ways of implementing functions of the method according to the invention by means of items of hardware or software, or both, provided that a single item of hardware or software can carries out several functions. It does not exclude that an assembly of items of hardware or software or both carry out a function, thus forming a single function without modifying the method of processing the drift frequency in accordance with the invention.

Said hardware or software items can be implemented in several manners, such as by means of wired electronic circuits or by means of an integrated circuit that is suitable programmed respectively.

Any reference sign in the following claims should not be construed as limiting the claim. It will be obvious that the use of the verb "to include" and its conjugations do not exclude the presence of any other steps or elements besides those defined in any claim. The article "a" or "an" preceding an element or step does not exclude the presence of a plurality of such elements or steps.

Claims

1. A video encoder (ENC) for encoding frames (FR) of a video signal before transmission, said video encoder (ENC) including a bit rate allocation unit (BRAU) for allocating a bit rate to encoded data in a encoding unit (ENU), said video encoder (ENC) further implementing a dependency map calculation unit

(DMCU) for calculating, throughout the time, for each data of said frame, a corresponding dependency value (DV) taking into account Intra and Inter prediction mechanisms, said dependency value (DV) expressing on which degree said data will be used in further encoding, said dependency map (DM) being used by said bit rate allocation unit (BRAU).

2. A video encoder as claimed in claim 1, wherein said data of a frame is a macroblock.

3. A video encoder as claimed in claim 1 , wherein said data of a frame is a pixel.

4. Video encoder as claimed in claim 1, wherein said data of a frame is a sub-pixel data.

5. A video encoder as claimed in claim 1, wherein said dependency value is binary.

6. A video encoder as claimed in claim 1, wherein said dependency value is a discretized value.

7. A video encoder as claimed in claim 1, wherein said dependency map has the size of a frame.

8. A video encoder as claimed in claim 1, wherein said dependency value is dependent on whether intra encoding of subsequent macroblocks are likely to use current macroblock data.

9. A video encoder as claimed in claim 1, wherein said dependency value is dependent on whether inter encoding of subsequent macroblocks are likely to use current macroblock data.

10. A method for encoding frames of a video signal before transmission, said method including a bit rate allocation step for allocating a bit rate to encoded data in a encoding step, said method further including a preliminary dependency map calculation step for calculating, throughout the time, for each data of said frame, a corresponding dependency value taking into account Intra and Inter prediction mechanisms, said dependency value expressing on which degree said data will be used in further encoding, said dependency map being used by said bit rate allocation step.

11. A computer program product comprising program instructions for implementing, when said program is executed by a processor, an encoding method as claimed in claim 10.

12. A video on demand system, recording system, streaming/broadcast system including a video encoder as claimed in one of claims 1 to 9.