EP1913781A2

EP1913781A2 - Method of encoding and decoding video images with spatial scalability

Info

Publication number: EP1913781A2
Application number: EP06778218A
Authority: EP
Inventors: Edouard Francois; Jérome Vieron; Nicolas Burdin
Original assignee: Thomson Licensing SAS
Current assignee: Thomson Licensing SAS
Priority date: 2005-08-12
Filing date: 2006-08-10
Publication date: 2008-04-23
Also published as: US20100150229A1; WO2007020230A3; FR2889778A1; WO2007020230A2

Abstract

The invention relates to a method of encoding and decoding video images with spatial scalability. The inventive method comprises the following steps consisting in: encoding (5) a low-resolution image, by performing a calculation of a local or reconstructed decoded image, in order to supply an encoded low-resolution image; oversampling (6) the reconstructed image in order to supply a prediction image; and encoding (7) a higher resolution image, comprising a difference calculation with the prediction image in order to supply residues. The invention is characterised in that the method also comprises a step consisting in selecting or calculating filter coefficients to be used for oversampling and a subsequent step consisting in encoding said coefficients such that they can be transmitted to the decoder with the other encoded data. The invention is suitable for hierarchical encoding with spatial scalability.

Description

METHOD FOR ENCODING AND DECODING VIDEO IMAGES WITH SPACE SCALABILITY

The invention relates to a device and a method for coding and decoding video images with spatial scalability, more particularly to a first low resolution image and at least a second higher resolution image from the low image. resolution, the first and second images having a common video part. The domain is that of hierarchical coding with spatial scalability and extended spatial scalability (ESS).

Spatial scalability is the ability to scale information to make it decodable at multiple levels of resolution and / or quality. More specifically, a data stream generated by the coding device is divided into several layers, in particular a base layer and one or more improvement layers. These devices make it possible in particular to adapt a single data stream to variable transport and visualization conditions. For example, in the particular case of spatial scalability, the portion of the data stream corresponding to the low resolution images of the sequence can be decoded independently of the portion of the data stream corresponding to the high resolution images.

The hierarchical coding with spatial scalability makes it possible to code a first portion of data called the base layer, relative to the low resolution format, and from this base layer a second portion of data called the improvement layer, relative to the high resolution format. The additional data relating to the improvement layer is generally generated according to a method comprising the following steps:

coding of the low resolution image and possibly local decoding of this coded image to obtain a reconstructed image,

scaling or upsampling the reconstructed low resolution image, for example by interpolation and filtering, to obtain a high resolution prediction image, and

- Difference, pixel by pixel, of the luminance values of the source image and the prediction image to obtain residuals relating to the enhancement layer, when the inter-layer coding mode is selected. Thus, the coding of the high resolution image exploits the low resolution image scaled as a prediction image. The method is also applied to chrominance images if they exist.

On the decoder side, the decoding method performs the reverse operations:

decoding the low resolution image to obtain a reconstructed image,

the addition, pixel by pixel, of the residues relating to the enhancement layer to the luminance values of the prediction image.

The various decoding operations are normative, in particular the decoder performs the operations of over-sampling the images of the base layer with predefined filters in the specification of the standard.

Coding operations are not normative. However, the local decoding operations performed by the coder for the computation of the reconstructed image must preferably be similar to those performed by the decoder, to obtain the same reconstructed image from which the residues are calculated, thus avoiding any problem of drift at the decoding level.

Also it is preferable that the sub-sampling operations of the high resolution image, in the case where it is used to obtain the low resolution image to be encoded, correspond to the operations of over-sampling the reconstructed image. low resolution so that the high resolution prediction image thus obtained and from which the residuals for the enhancement layer are calculated, can be as faithful as possible to the high resolution source image. The coefficients of the analysis filters must be adapted to those of the synthesis filters.

As a result, optimization of the filters operated at the encoder is not feasible. Indeed, such an optimization would result in a deterioration of the quality of the images due to the exploitation of different oversampling filters at the encoder and the decoder or because of the exploitation of uncorrelated oversampling and undersampling filters. . One of the aims of the invention is to overcome the aforementioned drawbacks. For this purpose, the subject of the invention is a method for coding video images with spatial scalability, coding a first low resolution image and at least a second higher resolution image from the low image. resolution, the first image having a common video portion with the second image, comprising

a coding step of the low-resolution image performing a calculation of a local or reconstructed decoded image, to provide an encoded low-resolution image, a step of oversampling the reconstructed image to provide a prediction image,

a coding step of the higher resolution image comprising a calculation of difference with the prediction image for providing residues, characterized in that it also comprises a step of selecting or calculating filter coefficients to be used; for oversampling then a coding step of the coefficients to be transmitted to the decoder with the other coded data.

According to a particular implementation, the coefficients of the filters are a function of the video content of an image, a plane of a sequence of images or a sequence of images.

According to a particular implementation, the coefficients of the oversampling filters are a function of the desired bit rate or quality of the high resolution image. According to a particular implementation, the method comprising a step of downsampling a higher resolution source image to provide the low resolution image to be encoded, is characterized in that the coefficients of the sub-sampling filters are function coefficients of the oversampling filters. According to a particular implementation of this method, the coefficients of the oversampling filters are a function of the video content of an image, a plane of a sequence of images or a sequence of images.

According to a particular implementation, the coefficients of the oversampling filters are a function of the levels and profiles used for the coding. The invention also relates to a method for decoding video images from a data stream comprising a base layer for encoding a low resolution image and at least one enhancement layer for encoding a video image. higher resolution image from residue, including

a step of decoding the low resolution image to provide a reconstructed low resolution image,

a step of over-sampling the reconstructed low-resolution image to provide a prediction image; a step of decoding the high-resolution image including an addition of residues to the prediction image, characterized in that also includes a step of decoding filter coefficients transmitted in the data stream for calculating the filters used for the oversampling. According to a particular implementation, the data stream comprising at least two enhancement layers, the method comprises a step of decoding at least a first and a second set of filters to respectively perform a first filtering for the oversampling from the low resolution image to a higher resolution image and a second filtering to oversampling the higher resolution image into a higher resolution image.

The invention also relates to a data stream comprising a base layer relating to a low resolution image and an upper layer relating to a higher resolution image, characterized in that it comprises a data field comprising values of digital filters to be exploited for oversampling the low resolution image, to provide an exploited prediction image, with the upper layer data, for decoding the high resolution image.

The idea is therefore to allow the use of so-called proprietary filters, by adding in the syntax of the data flow elements or fields describing the filters to be exploited for the over-sampling at the decoder level. With the transmission, in the stream, of some additional data relating to the filters and / or coefficients of the filters, the encoder can adapt the oversampling filters, the decoder being able to reproduce the same oversampling operations as the encoder, thus avoiding temporal drift phenomena.

For example, the filters used can be selected based on the video content of the image sequence, a clip or image plane. They can also be chosen according to the targeted spatial resolution, precise filters for high resolutions, more approximate for low resolutions. Another criterion may be the complexity of the display devices at the decoder, simpler filters then being implemented when addressing decoders of low computing power.

It is also possible to arbitrate between complex oversampling filters and thus a better quality of the prediction image for the calculation of the improvement layer residues giving a better compression ratio and simpler oversampling filters. reducing the treatment time.

FIG. 1 schematically represents a scalable decoding circuit according to the invention.

The bitstream received by the scalable decoder is transmitted to a demultiplexing circuit 1 which separates the data relating to the base layer or low layer and those relating to the improvement layer or high layer. The base layer data is transmitted to a low resolution decoder 2 which conventionally decodes the base layer information to output a low resolution image. The reconstructed images of the low-resolution decoder are transmitted to an over-sampling circuit 3

This circuit receives, for example from the demultiplexing circuit 1 or the central processing unit not shown in the figure, the data relating to the digital filters to be configured to perform the oversampling and filtering operations. The image thus oversampled and filtered or prediction image is then transmitted to the high resolution decoder 4 on a second input, the first input receiving the data relating to the enhancement layer from the demultiplexer 1. The residues are added to the prediction image for outputting a high resolution image. FIG. 2 represents a part of the scalable coding scheme according to the invention.

A low-resolution image is transmitted on the input of a low-resolution coder 5 which performs an encoding of this image to provide, on a first output, compressed data constituting the base layer or low layer, data transmitted to a multiplexer 8 On a second output, a reconstructed image from the local decoder, a decoder allowing, in known manner, reconstructing the coded images to calculate predicted images in order to take advantage of the temporal correlation, is transmitted to an oversampling circuit 6. This circuit filters and over-samples the reconstructed image from digital filters to provide a prediction image. This circuit is connected to a first processing circuit, not shown in the figure, which calculates the coefficients of the digital filters to be implemented, to transmit them to the oversampling circuit.

The prediction image calculated by the circuit 6 is transmitted, on a first input, to a high resolution encoder 7. This encoder receives, on a second input, the data relating to the high resolution source image. The encoder calculates, among other things, in a known manner, the residue which is the difference between the high resolution image and the prediction image to output data corresponding to the improvement layer or high resolution layer. This data is transmitted to the multiplexing circuit 8 which performs the multiplexing with the data of the base layer to provide the data stream or bitstream output of the encoder. The multiplexing circuit also comprises a second processing circuit, also not shown in the figure, whose function is to configure the flow of data transmitted by the coders according to a particular syntax. According to the invention, the syntax includes fields, for example at the sequence level or the slice level (slice in English according to the MPEG standard), allocated to these filters. Thus, the coefficients calculated by the first processing circuit are transmitted to the second processing circuit to be inserted into the appropriate fields in order to be transmitted to the decoder via the data stream.

Of course, the processing circuits may be the same central unit or be arranged differently, for example in an encoder, the calculation of the filters and the integration of the data of the filters in the stream being made at a layer level. The described part of the coding device receives as input at least two image sequences, one in low resolution format and one in high resolution format. These two sequences can for example be provided by a content creator. The encoding device may also include a subsampling module that can directly generate the low resolution image sequence from the source high resolution image sequence. This device then receives as input a single sequence of images in high resolution format.

According to this device, an improvement of the invention consists in calculating, from the coefficients of the digital filters exploited by the oversampling circuit 6, the digital filter coefficients exploited by the subsampling circuit of the high source image. resolution to obtain the low resolution image or vice versa. For example, from the analysis filters determined for the sub-sampling, it is possible to calculate the complementary synthesis filters for the oversampling according to the approach described in the document R. Ansari, CW Kim, and M. Dedovic. . Structure and design of two-channel filter banks derived from a triplet of half band filters. IEEE Transactions on Circuits and Systems II, 46 (12): 1487-1496, Dec. 1999. " The reverse approach, consisting of starting from a synthesis filter and deriving the analysis filter, as in the above document, can also be adopted.

The low-resolution and / or high-resolution coding circuits are for example of the H264AVC type, or its SVC extension, which stands for Scalable Video Coding, which makes it possible to address the scalable coding. The filters used for oversampling and / or undersampling are, for example, polyphase linear filters, for which the coefficients are a function of the position of the pixels to be interpolated.

In general, it is thus possible to exploit oversampling and / or subsampling filters depending on the content of the image, a complex filter being for example set up for a highly textured image, a simple filter for a homogeneous image. The type of filter used may be a function of the texture of the image, for example by avoiding the use of Lanczos filters for highly textured images, filters which generate aliasing (aliasing according to the English name) which is very unpleasant. penalizing for compression. Filters may vary between two clips if the video content changes, simple filter for a foreground corresponding to a scene with strong movement because in this case the texture is generally not very precise and its details are not perceived by the eye, complex filter for the following shot with a scene not very eventful, because the texture is then much better visible and must therefore be correctly oversampled. An analysis or pre-analysis of the image or the sequence of images can be exploited for the calculation of the filters.

According to an implementation example, the complexity of the filters used for the over-sampling of the low resolution reconstructed image is a function of the available resources of the receiver operating at the decoder. For example, a low resolution layer and a higher resolution layer correspond to data to be displayed respectively on a mobile phone screen and on an organizer screen also called personal digital assistant. It is then possible to lighten the oversampling calculations by simplifying the filtering, which saves the resources of the organizer. This to the detriment of a slight drop in quality of the image to display. For example, at a given profile and level combination (of the English profile / level, as defined in the MPEG-2 or MPEG-4 AVC standards), we associate a type of filter or coefficients of filter.

According to another example of implementation, the oversampling filters are different according to the exploited layers. By adding, in the previous example, a high-resolution layer for display on a television screen, simplified filters are used for oversampling the low resolution image to obtain a higher resolution image. for display on an organizer, more complex filters are exploited for the oversampling of the higher resolution image to obtain a high resolution image for display on a television. According to another exemplary implementation, the oversampling filter at the encoder, and thus also at the decoder, is selected from a compromise image quality or compression ratio / time or processing capacity. Since the reconstructed image is more faithful to the high-resolution source image when operating complex filters requiring a longer processing time, the quality of the high-resolution image is improved and the coding cost is decreased. For example, it is considered that the filters used are linear filters, possibly polyphase, separable, filtering in two dimensions can be done by filtering separately in one dimension and then in the other. Thus the filters are mono-dimensional. The following process, based on the resolution of the images, can be implemented:

- if over-sampling converts from English-language Common Intermediate Filter (CIF) format (176 columns by 144 lines) to CIF (Common Intermediate Filter) format (352 columns by 288 lines), the The over-sampling filter used is the following: monophase filter {1/2; 1/2}

- if the oversampling converts from CIF format to 4CIF format (704 columns by 576 lines), the over-sampling filter used is the following: single-phase filter

{1/32; -5/32; 20/32; 20/32; -5/32; 1/32} This filter is the one used by default in the current version of the SVC standard being defined

- if the oversampling converts from QCIF or CIF format to any higher format having a horizontal and vertical size ratio other than 2: a 4-polyphase filter based on the Lanczos filter described in the following table (the coefficients must be divided by 128):

- if the oversampling converts from 4CIF format to any higher format (for example 72Op format - 1280 columns by 720 lines), the over-sampling filter used is the following: 6-polyphase filter based on the Lanczos filter described in the following table (the coefficients must be divided by 32):

This solution makes it possible to process 4 levels of spatial scalability, QCIF, CIF, 4CIF, greater than 4CIF, with ad hoc filters according to the level of spatial resolution and the inter-layer size ratio considered. According to one particular implementation of the invention, the syntax relating to the SVC standard, which uses predefined filters, is modified.

This SVC syntax is described for example in the Joint Video Team (JVT) of ISO / IEC MPEG & ITU-T VCEG document (ISO / IEC JTC1 / SC29 / WG11 and ITU-T SG 16 Q.6), 15th meeting, Busan , KR, April 16-22, 2005.

The following parameters and fields are added in the syntax of the bitstream as follows: - at the sequence level: a variable, named load_coef. This variable can take the following values: load_coef = 0, in which case the default oversampling technique applies / oac / _coe / = 1, ad hoc coefficients are encoded in the bit stream at the sequence level and s then apply to all images in the load_coef = 2 sequence, ad hoc coefficients are encoded in the bit stream at the image level and then apply to the image with which they are associated

If / oac / _coe / = 1, the filter coefficients are encoded in the sequence level syntax. The syntax described in Table 1 below (level syntax sequence) illustrates how these coefficients can be encoded. - at the image level: If load_coef = 2, the coefficients of the filters are coded in the syntax at the image level. The syntax described in Table 2 below (slice level syntax) illustrates how these coefficients can be encoded. We can reuse the coefficients of the previous slice, without recoding, by signaling it in the bit stream (same_coef_as_previous_slice = 1). The syntax is actually described at the level of a "slice" or "slice" in English which, according to the standard, is a continuous series of macroblocks. Here the description is limited to a slice consisting of all the macroblocks of the image and thus assimilated to an image, a solution usually adopted. But it is of course possible, according to this syntax, to use different filters for particular areas of the image corresponding to slices. The following tables are an example of syntax for the data flow. The syntax description method of the bitstream uses the "C code" convention. It corresponds to the method used in the description of the MPEG or H264 standards, it is thus found, to give examples, in documents such as ISO / IEC 13818-2 or else JVT-O202 entitled "Joint Scalable Video Model JSVM 2, 15th meeting, Busan, KR.

Table 1 corresponds to the syntax to be added to that relating to the sequence level, the filters can then be renewed at each sequence.

Table 2 corresponds to the syntax to be added to that relating to the slice level, the filters can then be renewed at each slice of the image.

Table 2

Tables 3 and 4 describe this syntax more explicitly:

Table 3

Table 4

By default, we consider that a technique of oversampling, and if necessary, the corresponding coefficients, is available. In the case of non-transmission of proprietary coefficients in the bit stream, this technique applies.

In the coding / decoding process, the changes are directly related to the syntax changes. Only the part relating to the texture is considered:

If load_coef = 0, the default technique for over-sampling texture applies.

Otherwise if load_coef = λ, the over-sampling of texture of the low resolution images is carried out with the coefficients of filter coded / decoded at the level sequence coef_seq.

Otherwise, if load_coef = 2, the over-sampling of texture is done on each low resolution image with the coded / decoded filter coefficients at the coefjpic image level, applying to this image.

The coefficients of the oversampling filters can be calculated by the encoder. They can also be selected by the encoder from a set of filters preset and stored in a memory of the encoder. It is thanks to the transmission of the parameters of the filters to the decoder that it is possible to adapt the filtering to the coding. The filters used in the decoding for the oversampling are thus determined during the coding.

An extension of the proposed approach is to standardize the decoder with a set of predefined filters. In addition, ad hoc filters may also be indicated in the syntax, as described in the invention, and stored by the decoder. Thus the decoder at any time has a set of potential filters, stored in memory and indexed by a number. The indexes of the filters to be used can then be sent to the decoder, indicating which filters it should use, without the need to explicitly send the coefficients of these filters. If at any given time new filters are to be used, they are encoded in the syntax with their associated index.

Claims

A method for encoding video images with spatial scalability encoding a first low resolution image and at least one second higher resolution image from the low resolution image, the first image having a common video portion with the second image, comprising

a coding step (5) of the low-resolution image performing a calculation of a local or reconstructed decoded image, to provide an encoded low-resolution image,

an oversampling step (6) of the reconstructed image to provide a prediction image,

a coding step (7) of the higher resolution image comprising a calculation of difference with the prediction image for providing residues, characterized in that it also comprises a step of selecting or calculating filters to be used for oversampling and then a coding step of the coefficients to be transmitted to the decoder with the other coded data.

2 coding method according to claim 1, characterized in that the coefficients of the filters are a function of the video content of an image, a plane of a sequence of images or a sequence of images.

Method according to claim 1, characterized in that the coefficients of the oversampling filters are a function of the desired bit rate or quality of the high resolution image.

4. The method as claimed in claim 1, including a subsampling step of a higher resolution source image for providing the low resolution image to be encoded, characterized in that the coefficients of the subsampling filters are a function of the coefficients of the filters. over-sampling.

Method according to claim 4, characterized in that the coefficients of the over-sampling filters are a function of the video content an image, a plane of a sequence of images or a sequence of images.

6 Process according to claim 1, characterized in that the coefficients of the oversampling filters are a function of the levels and profiles used for the coding.

Method for decoding video images from a data stream comprising a base layer for encoding a low resolution image and at least one enhancement layer for encoding a higher resolution image from residues, comprising

a step of decoding (2) the low resolution image to provide a reconstructed low resolution image,

an oversampling step (3) of the reconstructed low resolution image to provide a prediction image,

a decoding step (4) of the high resolution image comprising an addition of residues to the prediction image, characterized in that it also comprises a step of decoding filter coefficients transmitted in the data stream for the calculation of filters used for oversampling.

The method according to claim 7, the data stream comprising at least two enhancement layers, characterized in that it comprises a step of decoding at least a first and a second set of filters to respectively perform a first filtering for oversampling the low resolution image to a higher resolution image and a second filtering for oversampling the higher resolution image to a higher resolution image.

A data stream comprising a base layer relating to a low resolution image and an upper layer relating to a higher resolution image, characterized in that it comprises a data field comprising values of digital filter coefficients intended to be exploited for the oversampling of the low resolution image, to provide an exploited prediction image, with the data of the upper layer, for the decoding of the high resolution image according to the method of claim 7.