GB2455979A - Video processing using quincunx sub-sampling - Google Patents

Abstract

A method of video signal processing, for a picture sequence, comprises: diagonally low pass filtering 1 input video; performing quincunx sub-sampling 2 (i.e. in a shape corresponding to the dots on the five of a die - see Figure 5), and processing the quincunx samples to provide output video which is rectangularly sampled with reduced spatial resolution and increased picture rate. Also independently claimed is an analogous video encoding method, wherein the quincunx sub sampling reduces input sample rates by a factor of two, and quincunx samples are re-ordered so the signal appears as a conventional, rectangularly-sampled signal to a subsequent encoder. Further independent claims relate to: analogous decoding steps, additionally interpolating the quincunx video signal to produce a video signal at original resolution; and an encoder in which quincunx video samples are retained or discarded on an even or odd functional basis, and samples are split 3 into rectangular samples, forming child pictures (Figure 7) for interleaving 5 into a video stream at double the original rate.

Description

-1-2455979 VIDEO ENCODING, DECODING AND PROCESSING This invention relates to encoding of video and to pre-processing of video to improve the efficiency of compression encoding. Aspects of the invention also include complementary post processing and decoding.

In one aspect, the present Invention provides a method of processing video which comprises a sequence of pictures, the method comprising the steps of diagonally low pass filtering the input video; performing quincunx sub-sampling; and processing the quincunx samples to provide output video which is rectangularly-sampled with a reduced spatial resolution and an increased picture rate.

In another aspect, the present invention provides a method of encoding video which comprises a sequence of pictures, the method comprising the steps of diagonally pre-filtering the video, performing quincunx sub-sampling to reduce input sample rates by a factor of two, processing and re-ordering the quincunx samples so that the signal appears to a subsequent encoder as conventional, rectangularly-sampled video signal, and encoding the modified video signal at reduced spatial resolution and increased picture rate.

In yet another aspect, the present invention provides a method of decoding video which has been so encoded, the method comprising the steps of decoding the modified video signal at reduced spatial resolution and increased frame rate, processing and re-ordering the modified video signal so as to reconstruct a quincunxially-sampled video signal, and interpolating the quincunx video signal to produce a conventional video signal at the original video resolution.

In still another aspect, the present invention provides a method of encoding and decoding video which comprises a sequence of pictures, the method comprising the encoder steps of diagonally pro-filtering the video; performing quincunx sub-sampling to reduce input sample rates by a factor of two, processing and re-ordering the quincunx samples so that the signal appears to a subsequent encoder as conventional, rectangularly-sampled video signal, and encoding the modified video signal at reduced spatial resolution and increased picture rate; the method further comprising the decoder steps of: decoding the modified video signal at reduced spatial resolution and increased frame rate; processing and re-ordering the modified video signal so as to reconstruct a quincunxially-sampled video signal; and interpolating the quincunx video signal to produce a conventional video signal at the original video resolution.

One example of the invention provides a video processor, comprising a diagonal filter which suppresses spatial frequencies (fh, f) which satisfy (fIJfH) +(flfv) >F; a quincunx sub-sampler in which samples c of the video picture are retained if 1+] is even, and discarded if 1+1 is odd (or vice-versa); and a picture splitter in which the quincunx samples are split into two groups of rectangular samples forming child pictures CI and C2 such that CI consists of samples P'j with p,, = C2,.2j and C2 consists of samples q with q,j = c21+1,21+1; and a picture interleaver in which the pairs of child pictures are interleaved so that a single video stream is created at double the original picture rate.

Another example of the invention provides a video encoder system, comprising a diagonal filter which suppresses spatial frequencies (f,,, f) which satisfy (ff/fH) +(fjfv) >F for some constant value F�=1, where fH and f are the maximum horizontal and vertical spatial frequencies of the original input video; a quincunx sub-sampler in which samples c, of the video picture are retained if 1+] is even, and discarded 1ff +j is odd (or vice-versa); and a picture splitter In which the quincunx samples are split into two groups of rectangular samples forming child pictures Cl and C2 such that Cl consists of samples P/i with Pi.j = c2,2J and C2 consists of samples q, with q,j = c+1,21+1; a picture interleaver in which the pairs of child pictures are interleaved so that a single video stream is created at double the original picture rate; and a video encoder in which the sequence of child pictures is encoded by means of conventional video coding standards or techniques (including H264, MPEG2, H261, H263, Dirac) devised for rectangularly sampled video.

Still another example of the invention provides a complementary video decoder system.

comprising a video decoder in which the sequence of child pictures is decoded according to the conventions of the compression system, so that the inter picture C2 of each child picture pair is reconstructed by reference to CI in accordance with the bit stream, wherein the Cl reconstructed samples are p,1 and the C2 reconstructed samples are q,; a picture de-interleaver which extracts a pair of child pictures Cl and C2 from decoded picture stream; a picture combiner which forms a single rectangularly sampled picture at the original picture resolution by combining the two child pictures and inserting zeros to create samples c such that: C2,,21 = Pi C2j+12jO (7) C21,2j+1 = 0 = q, and a diagonal interpolation filter which provides interpolated values for the zeroed samples.

In one embodiment, the pre-processing reduces the input sample rate via quincunx subsampling, re-ordering the samples so that the signal appears to the encoder as a conventionally-sampled video stream of one quarter spatial resolution and double the original picture rate. Each original picture (frame or field) is mapped to two output pictures of the reduced resolution, produced at the same sample time. This process can be integrated with the motion compensation facility of an encoder so that one picture of each output pair can be predicted from the other motion vectors. This is highly effective sInce both output pictures actually derive from the same input picture. Additional reversible quincunxial filtering can be applied to increase efficiency and reduce any aliasing introduced. After decoding, the pairs of co-timed video pictures may be reconstructed into a single picture of the original resolution by merging the two pictures and interpolating the missing samples.

Aspects of this invention accordingly provide novel techniques for pre-processlng video pictures via filtering and quincunx sampling, which discard "high diagonal" spatial frequencies, which are relatively invisible to the human eye, in order to increase compression efficiency in subsequent encoding. Examples may produces compression gains of approximately 30% and reduce computational load in the video coder and decoder by half.

The invention will now be described by way of example with reference to the accompanying drawings, in which: Figure 1 shows a conventional rectangular grid sampling positions Figure 2 shows a spatial frequency domain of signals supportable by a rectangular sampling grid, up to Nyquist horizontal and vertical frequencies fH, fv.

Figure 3 shows excess bandwidth of a square-sampled signal over an isotropic frequency domain. The hatched area represents high diagonal frequencies which are less visible.

Figure 4 shows spatial frequency response of a signal after diagonal low-pass filtering (50% cut shown).

Figure 5 shows Quincunx sample positions.

Figure 6 shows Quincunx samples decomposed into two interleaved rectangular sample arrays Figure 7 shows splitting a quincunx-sampled picture into two child pictures.

Figure 8 shows GOP prediction structures. a) A conventional (12,3) GOP structure. b) A GOP structure optimised for coding quincunx child sub-pictures.

Figure 9 shows an example of reversible quincunx filtering.

FIgure 10 shows spatial frequency bands produced by half-band low and high-pass filtering on quincunx-sampled signals. The hatched area represents the low-pass frequences (first child picture); the remaining shaded area represents the high-pass frequencies (second child picture after motion compensation).

Figure 11 shows an embodiment of the preprocesing and encoder operation.

Figure 12 shows an embodiment of the post-processing and decoder operation.

Conventional digital video signals comprise samples located at rectangular sample points (Figure 1).

A rectangular sample structure supports the representation of signals within a corresponding rectangular spatial frequency domain (FIgure 2). On the other hand, the human eye is approximately equally sensitive to detail at all orientations, giving a circular frequency response. Given sufficient vertical and horizontal resolution, therefore, a rectangular sample grid supports high diagonal spatial frequencies which are significantly less visible (Figure 3).

Ideal isotropic (circular) low-pass pre-filtering can therefore be used to remove spatial frequencies that are relatively invisible to the human eye. In practice, diagonal low-pass filtering is used instead. This produces a triangular spatial frequency response (Figure 4), and works better in conjunction with video coding techniques. For example, MPEG-2 uses diagonal block coefficient scanning and run-length coding, and diagonal filtering will produce very small or zero coefficients for all trailing coefficients in a DCT block.

However, diagonal low-pass filtering has an additional benefit not previously exploited in pre-processing techniques: if the cut-point is set appropriately, the original rectangular sample structure can be sub-sampled in a quincunx structure. To avoid allasing, the filter must remove at least half of the spatial spectrum, cutting at or below the maximum horizontal and vertical frequencies fH and fv.

In quincunx sampling, samples are located along diagonal lines at 45 degrees to the image boundaries (Figure 5). When the diagonally filter cut point is set to 50%, quincunx subsampling then causes the signal to be critically sampled with respect to its spatial spectrum. Quincunx sampling is therefore the optimal representation of a diagonally-filtered two-dimensional signal. Given ideal filtering the discarded samples can be perfectly reconstructed by interpolation according to the usual Nyquist principles.

Video filtered and subsampled in this way has two advantages: firstly, relatively invisible spatial frequencies have been discarded; and secondly the sample rate has been reduced by half. Both of these features should in principle significantly aid video compression. Unfortunately, the quincunx sample structure is not a standard video sample structure supported by any conventional video encoding technology or standard.

This invention is concemed in certain aspects with treating these filtered, quincunxial samples so that they may be presented as conventional video signals to a video encoder which operates in a standard way.

One embodiment of this Invention encompasses a pre-processing method for * diagonally pre-filtering video signals, * performing quincunx subsamping to reduce input sample rates by a factor of 2, * processing and re-ordering the quincunx samples so that the signal appears to a subsequent encoder as a conventional, rectangulaily-sampled video signal, and * encoding the modified video signal at reduced spatial resolution and increased picture rate.

Another embodiment encompasses a post-processing method for: * decoding the modified video signal at reduced spatial resolution and Increased frame rate, * processing and re-ordering the modified video signal so as to reconstruct a quincunxially-sampled video signal, * interpolating the quincunx video signal to produce a conventional video signal at the original video resolution.

The reodering of quincunx picture signals may be achieved by considering the samples shown in Figure 5 as two offset interleaved rectangular sample arrays. This is shown in Figure 6 where one sample array is marked with a cross and the other with a circle.

Splitting the quincunx samples from a picture into these groups produces two sub-sampled child' pictures with vertical and horizontal dimensions half that of the original input video (Figure 7).

For example, a standard definition 720x576 pixel picture component will produce two 360x288 pixel picture components after quincunx sub-sampling and re-ordering.

This basic structure of diagonally pre-filtering, quincunx sampling, and child picture splitting may be used as a pro-process to any video encoding process, without modifying that encoder except to configure the picture dimensions and frame rate correctly to the new values. However, two further enhancements may also be considered.

In the first enhancement, an encoder is configured to use one child picture to predict another by motion compensation. That is, at least one of the reference pictures for one of the child pictures will be the other child picture in a pair. Since the two child pictures will be highly correlated, deriving as they do from the same original picture, motion compensation will be highly effective.

A constant vector field representing the displacement of the samples in one child picture from the samples in the other, may be used. The ideal constant vector field would have value (1/2,1/2) or (-112,-112), since each sample is located in the centre (halfway-point) of 4 samples from the other child picture. However, an the (0,0) field may also be effective.

Atypical Group of Pictures (GOP) motion-prediction structure for video coding uses: intra pictures, known as I pictures, which are coded without reference to other pictures; forward-predicted pictures, known as P pictures, which are predicted from previous pictures in a sequence (the prediction is forward in time); and bi-directional pictures, known as B pictures which are predicted from both previous and subsequent pictures.

Figure 8a shows a conventional GOP where P pictures occur every third picture and I pictures every 12. The arrows point from the picture to its reference picture. Subscripts refer to the picture number in presentation (display) order.

Figure 8b shows a GOP structure optimised for coding quincunx child sub-pictures. In this structure, P pictures occur half as often (as the picture rate is doubled). The first child picture of each pair (even picture numbers) is predicted from child pictures of the same type (i.e. another even picture number). Even second child picture (odd picture numbers) is always a B picture and is always predicted from the first and the next I or P picture.

In the second enhancement, additional reversible filtering may be used to modify the samples in the two child pictures.

The child pictures are sub-sampled versions of the quincunxially-sampled picture, and as such may be aliased. This will mean that they will not code as efficiently as they might.

Aliasing may be reduced by using one child picture to modify the other in a so-called lifting scheme. For example, if P'j represent the samples in one child picture and qij those in the other we might modify the samples as follows: p,, = Pi +( q.ij.i + qj.ij + + q,)/4 (1) so that one child picture is a low-pass filtered picture (Figure 9) using a half-band Nyquist quincunc filter. This will reduce the bit rate in subsequent encoding of the low-pass child picture.

Note that this filtering is completely reversible provided that sufficient dynamic range is maIntained for the resulting values.

My chain of filtering by which one child picture is modified by means of some combination of values from the other child picture may be applied, and will equally be reversible.

The configuration by which the first child picture is low-pass filtered by this means is partlculaily suitable In conjunction with the first enhancement (section 2.2.1), by which each second child picture is motion compensated from the first. If a constant motion field is used in conjunction with the half-band filter low-pass filter defined in (1) for the first child pIcture, the motion compensated residual will represent the result of applying a half-band high-pass filter (unity minus a half-band low-pass filter) to the qulncunx samples, sub-sampling at the second child picture sample points. This will assist in compression of the second child picture also.

A preferred embodiment of the encoder pre-processing operation is illustrated In Figure 11 and a preferred embodiment of the decoder post-processing operation is illustrated in Figure 12.

In Diagonal Filtering I a low-pass filter is applied which suppresses spatial frequencies (fh, f) which satisfy (ft/f,.,) +(f/fv) >F (2) for some constant value F�=1, where fH and f are the maximum honzontal and vertical spatial frequencies of the original input video. The passband may be adjusted adaptively by the Encoder Control 7.

In Quincunx Subsampling 2, samples c of the video picture are retained if i+j is even, and discarded if I + j is odd (or vice-versa).

In Picture Splitter 3, the quincunx samples are split into two groups of rectangular samples forming child pictures CI and C2 such that CI consists of samples p,j with p,,1=c21,21 (3) and C2 consists of samples q,.j with = c21+121+1 (4) In Reversible FIlter 4, the child picture samples are filtered to reduce aliasing so that Cl is a low-pass filtered version of the original picture: p,1 = p,j +( q'-ii + + + q,j)14 (5) In Picture Interteaver 5, the pairs of child pictures are interleaved so that a single video stream is created at double the original picture rate.

In Video Encoder 6, the sequence of child pictures is encoded by means of conventional video coding standards or techniques (H264, MPEG2, H261, H263, Dirac) devised for rectangularly sampled video.

Encoder Control 7, sets the video encoder parameters to be consistent with the pre-processed picture sizes and picture rates, configures the encoder motion prediction structure so that each second child picture C2 is always predicted from the first child picture CI, and may set the passband of Diagonal Filter I according to the prevailing bit rate or quality in the Video Encoder 6.

In Video Decoder 8, the sequence of child pictures is decoded according to the conventions of the compression system, so that the inter picture C2 of each child picture pair is reconstructed by reference to CI in accordance with the bit stream. The Cl reconstructed samples are p, and the C2 reconstructed samples are q,,1.

Picture De-interleaver 9 extracts a pair of child pictures Cl and C2 from decoded picture stream, for processing by the Inverse Reversible Filter 10. This reverses the anti-allasing pre-fllter by the recipe: = Pi.j -( q,.ij.i + q,11 + q,,1.1 + qij)/4 (6) Picture Combiner 11 forms a single rectangularly sampled picture at the original picture resolution by combining the two child pictures and inserting zeros to create samples c such that: C2L2J = Pij C21+1,2j=O (7) 021.2j+1 = 0 C2,+1,2j+1 = qij This picture Is filtered in Diagonal Interpolation Filter 12 to provide interpolated values for the zeroed samples in (7).

It will be understood that feature of the described embodiments can be preformed in hardware or in software or in combinations thereof. Steps can be performed on a video signal in real time or in non-real time or performed on a video file stored in any appropriate manner.

Claims (22)

1. A method of processing video which comprises a sequence of pictures, the method comprising the steps of diagonally low pass filtering the input video; performing quincunx sub-sampling; and processing the quincunx samples to provide output video which is rectangularly-sampled with a reduced spatial resolution and an increased picture rate.

2. A method according to Claim 1, wherein the step of processing the quincunx samples to provide output video comprises re-ordering the samples to form a conventionally-sampled video stream of one quarter spatial resolution and double the original picture rate.

3. A method according to Claim I or Claim 2. wherein each original picture is mapped to two output pictures of the reduced resolution, produced at the same sample time.

4. A method according to Claim 3, wherein one of said output pictures is modified using information form the other.

5. A method according to Claim 4, wherein a pixel in one said output picture is summed with pixel data from corresponding, neighbouring pixel locations in the other.

6. A method according to Claim 4 or Claim 5, in which said modification Is reversible.

7. A method according to any one of Claims 4 to 6. whereIn, If p,,1 represent the samples in one said output picture and q, those in the other, the samples are modified as: p1, = p,

8. A method according to Claim 7, wherein: py = p, +( q11,11 + qi-ij + + q,1)14

9. A method of encoding video which comprises a sequence of pictures, the method comprising the steps of diagonally pre-filtering the video, performing quincunx sub-sampling to reduce input sample rates by a factor of two, processing and re-ordering the quincunx samples so that the signal appears to a subsequent encoder as conventional, rectangularly-sampled video signal, and encoding the modified video signal at reduced spatial resolution and increased picture rate.

10. A method according to Claim 9, in which each input picture is mapped to two output pictures of reduced resolution, the encoder being configured to predict one of said output pictures from the other.

11. A method according to Claim 10, wherein one of said output pictures is predicted from the other by motion compensation using a constant vector field, preferably of (1/2,1/2) or (-1/2,-1/2).

12. A method of decoding video which has been encoded by a method according to any one of Claims 9 to 11, the method comprising the steps of decoding the modified video signal at reduced spatial resolution and increased frame rate, processing and re-ordering the modified video signal so as to reconstruct a quincunxially-sampled video signal, and interpolating the quincunx video signal to produce a conventional video signal at the original video resolution.

13. A method of encoding and decoding video which comprises a sequence of pictures, the method comprising the encoder steps of diagonally pre-filtering the video; performing quincunx sub-sampling to reduce input sample rates by a factor of two, processing and re-ordering the quincunx samples so that the signal appears to a subsequent encoder as conventional, rectangularly-sampled video signal, and encoding the modified video signal at reduced spatial resolution and increased picture rate; the method further comprising the decoder steps of: decoding the modified video signal at reduced spatial resolution and increased frame rate; processing and re-ordering the modified video signal so as to reconstruct a quincunxially-sampled video signal; and interpolating the quincunx video signal to produce a conventional video signal at the original video resolution.

14. A method according to Claim 13, in which each input picture is mapped to two output pictures of reduced resolution, wherein one of said output pictures is modified in a reversible filtering step in the encoding using information from the other and wherein a reverse filter step is conducted in the decoding.

15. A video processor, comprising a diagonal filter which suppresses spatial frequencies (fh, f) which satisfy (f,,/fH) +(fjfv) >F; a quincunx sub-sampler in which samples c of the video picture are retained if I i-j is even, and discarded if I 1 is odd (or vice-versa); and a picture splitter in which the quincunx samples are split into two groups of rectangular samples forming child pictures Cl and C2 such that Cl consists of samples P'.j with Pi,j = C2j,2j and C2 consists of samples qij with q,.j = c2,+i,21+i; and a picture interleaver in which the pairs of child pictures are interleaved so that a single video stream is created at double the original picture rate.

16. A video processor according to Claim 15, further comprising a reversible filter in which the child picture samples are filtered to reduce aliasing so that Cl is a low-pass filtered version of the original picture: = p,,1 +( qi-ij.i + q,.ij + q,,1.1 + qij)14)

17. A video encoder system, comprising a diagonal filter which suppresses spatial frequencies (fh, f) which satisfy (f,,ffH) +(fjfv) >F for some constant value F�=1, where f and f are the maximum horizontal and vertical spatial frequencies of the original Input video; a quincunx sub-sampler in which samples c,j of the video picture are retained if! +j is even, and discarded if / +j is odd (or vice-versa); and a picture splitter in which the quincunx samples are split into two groups of rectangular samples forming child pictures Cl and C2 such that Cl consists of samples P',j with p,,1 = c21 and C2 consists of samples q,j with q, = C21+1,2j+1; a picture interleaver in which the pairs of child pictures are interleaved so that a single video stream is created at double the original picture rate; and a video encoder in which the sequence of child pictures is encoded by means of conventional video coding standards or techniques (including H264, MPEG2, H261, H263, Dirac) devised for rectangularly sampled video.

18. A video encoder system according to Claim 17, further comprising a reversible filter in which the child picture samples are filtered to reduce ailasing so that Cl is a low-pass filtered version of the original picture P = p,j +( q,111 + + q,,11 + q,j)/4).

19. A video encoder system according to Claim 17 or Claim 18, the encoder motion prediction structure is configured so that each second child picture C2 is always predicted from the first child picture Cl.

20. A video encoder system according to any one of Claims 17 to 19, wherein the encoder serves to set the passband of the diagonal filter according to the prevailing bit rate or quality in the encoder.

21. A video decoder system adapted to decode video encoded in a video encoder system according to any one of Claims 17 to 19, comprising a video decoder in which the sequence of child pictures is decoded according to the conventions of the compression system, so that the inter picture C2 of each child picture pair Is reconstructed by reference to Cl in accordance with the bit stream, wherein the Cl reconstructed samples are p and the C2 reconstructed samples are q; a picture de-interleaver which extracts a pair of child pictures Cl and C2 from decoded picture stream; a picture combiner which forms a single rectangularly sampled picture at the original picture resolution by combining the two child pictures and inserting zeros to create samples c,j such that: C21,2j = c2!+1,21 0 (7) C2,2j+1 = 0 = and a diagonal interpolation filter which provides interpolated values for the zeroed samples.

22. A video decoder system according to Claim 21, further comprising an inverse reversible filter which serves to reverse the anti-aliasing pre-f liter.