GB2324220A - Video data compression system - Google Patents

Abstract

A video signal is sampled and digitised (42) and wavelet transformed (44). A tree processor (46) extracts from each frame trees of correlated image areas in the sub bands of the transferred image. For the current frame, only those trees (if any) which have changed compared to the previous frame or frames are subsequently encoded. The trees are encoded according to a quantization table selected from a set of such tables. The selection is based on that proportion of the trees in a frame which have changed. The larger the number of changed trees the coarser the quantization.

Description

DATA COMPRESSION The present invention relates to a data compression system. Illustrative embodiments of the invention concern the use of Wavelet Transforms and Trees into which the coefficients produced by the transform are organised.

Wavelet Transforms and Trees are Known. For a better understanding of the background to the present invention, reference will now be made to Figures 1 and 2 of the accompanying drawings in which: Figure 1 is a schematic diagram of a frame of image information subject to a wavelet transform over 4 scales; and Figure 2 is a conceptual diagram of a tree structure; Referring to Figure 1, consider a CIF image comprising 352 x 288 pixels for 625 lines per frame at 25 frames per second and a block of 16x16 = 256 pixels forming part of the image. A wavelet transform is repeatedly applied to the CIF image horizontally and vertically and the transformed image is subsampled, in known manner. In the example of Figure 1, the wavelet transform is applied over 4 scales and the invention is illustrated by way of example with reference to Figure 1.

However, other numbers of scales are possible.

The transformed image comprises a 2-dimensional array of spatial frequency components which are within, for example, thirteen sub-bands 0-12. In Figure 1 spatial frequency generally increases horizontally from left to right and vertically from top to bottom. The image is subsampled by 2 horizontally and vertically in bands 10, 11 and 12. Thus, the original 16x16 block is reduced to 8x8. The image is subsampled by 4 horizontally and vertically in bands 7, 8, and 9, so the original 16x16 block is reduced to 4x4. The image is subsampled by 8 horizontally and vertically in blocks 4, 5 and 6 so that original 16x16 block is reduced to 2x2. It is subsampled by 16x16 in bands 0, 1, 2 and 3 so the block is reduced there to a single pixel. Each of the bands 0-12 represents spatial frequency components of the whole original image in a particular range of frequencies. The contents dfhe bands are highly correlated for typical video images.

The bands 0-3 having the lowest frequencies usually contain most of the image information. Bands 10, 11, and 12 having the highest frequencies usually contain little image information.

Because of the correlation of the image information in the bands, one coefficient a in band 0 tends to be correlated with one coefficient b c d in each of bands 1, 2 and 3 respectively. In turn, coefficient b is correlated with a group e, f, g of 2x2 coefficients in each of bands 4, 5 and 6 and soon to bands 10, 11 and 12.

In this way, a tree structure can be conceptualised as shown in Figure 2. For a 16x16 = 256 pixel block of the original untransformed image, a corresponding tree structure as shown in Figure 2 also having 256 coefficients may be identified. Thus a tree is a set, or block, of coefficients selected from the sub-bands in accordance with the tendency of the coefficients to be correlated one with another.

It is known from US-A-5 321 776 and US-A-5 131 670 (Shapiro) to provide a video data compression system in which a still image is subject to a wavelet transform and trees of correlated image information are extracted from the transformed image. Shapiro deals only with still images and so its application to video does not take advantage of inter frame redundancy.

According to one aspect of the present invention there is provided a video compression encoder comprising: an input for receiving an image signal, an output for outputting a compression encoded image signal, transform means for producing from each picture of the image signal an array of spatial frequency coefficients representing the image in a plurality of frequency bands, means for extracting from each picture of the transformed image signal trees comprising sets of coefficients representing correlated image areas in the said bands, means for comparing corresponding trees from successive pictures and for determining which trees in one picture which have changed compared to the corresponding trees in a previous picture,and means for selecting trees for output and compression encoding in dependence upon the comparison.

Another aspect of the present invention provides a video compression decoder for use with the encoder of said one aspect comprising means for compression decoding trees, means for storing decoded trees, means for replacing stored trees by newly decoded corresponding trees, and means for inverse transforming the stored trees.

"Picture" means field or frame. For convenience reference is made hereonafter to frames.

In a preferred embodiment of the invention a 2-dimensional array of spatial frequency component representing the image in a plurality of 2D frequency bands is produced.

Preferably the transform means performs a wavelet transform.However, it is possible to use other transforms provided the transformed data is appropriately ordered.

Preferably, the powers of corresponding trees from different pictures are compared to determine which trees have changed. 'Power' in this context may mean the sum of the squares of the coefficients Pi of a tree: SPi2 Other measures of degree of change could be used such as the sum of the absolute values of the coefficients Pi of the trees.

In accordance with one inventive aspect of the present invention, the trees of a current frame of the image are compared with the corresponding trees of the preceding frame, or frames, and those trees which have changed significantly are encoded and transmitted to the decoder thus taking advantage of inter-frame redundancy.

Preferably, if the power of the tree of the current frame exceeds that of the corresponding tree of a preceding frame or frames by a threshold amount Thl, the preceding tree is replaced by the current tree. Thus, the number of trees to be encoded in each frame may be reduced because only trees which have changed significantly need to be encoded and transmitted.

One embodiment encodes, for a picture, only those trees which have changed compared to a previous picture. The amount of data to be encoded is thus reduced providing more scope for compression. Furthermore, encoding (and thus decoding) of individual trees is an essentially intra-picture process and is not dependent on preceding or succeeding pictures, reducing the effect of decoding errors on picture quality. Yet further an error in one tree will not affect subsequent pictures. As pictures change it is unlikely that errors will continue to occur in the same tree or, if they do, not in the same location of a tree.

In accordance with another inventive aspect of the present invention, trees are encoded and output in descending order of degree of change or power difference. In an embodiment of this aspect of the invention, all the trees of a current picture or frame are listed in descending order of power difference between the tree of the current frame and the corresponding tree from the previous frame or frames. The trees with largest power difference are those which have changed most and thus must be encoded to correctly represent the current frame. The trees with lowest power difference are those which have changed least (or are unchanged) and thus are those which least need encoding to represent the current frame. The trees are encoded and transmitted to the decoder in the list order.

In one embodiment of this aspect of the invention, only trees having a power difference greater than the threshold are encoded and transmitted. In another embodiment of this aspect, the trees are encoded and transmitted in the listed order until a desired bit rate is achieved. Thus in this other embodiment at least some trees whose power difference is less than the threshold may be encoded and transmitted.

In this context bit rate is bits/frame or bits/field. If many trees have changed only those changed trees are likely to be encoded within the preset bit rate. If only a few trees have changed it is likely a large proportion, or all, of the trees forming the picture, will be encoded.

In accordance with a further inventive aspect of the invention, the proportion of the number of trees having power differences above a threshold Th2 to the total number of trees in the frame is established and the quantization applied to the trees in the frame is chosen according to the proportion.

Preferably, a plurality of precalculated quantization tables are used and a selection is made from the precalculated tables according to the proportion. The quantization table to be used is chosen on the basis of the target bit rate for a frame, and the proportion of the trees above the threshold Th2.

In practice, the threshold Th2 is chosen to provide, with the appropriate quantization tables, a desired bit rate, an example being approximately 2Mbits/second for a typical image. Some images may allow a larger number of trees than indicated by the threshold to be encoded, others fewer. The trees in a frame are quantized in accordance with the chosen quantization table and encoded in descending order of power difference until the bit rate reaches the target of 2Mbits/second.

Consider an image frame which has for example 352 x 288 = 101376 pixels and known as a CIF image. The trees of the transformed image have the same number of coefficients. At a frame rate of 25 frames/second and a target bit rate of 2Mbit/second, a frame is encoded to about 0.8 bits/pixel (2Mbit/s / 101376 x 25) if all the trees of a frame are transmitted. If fewer than all the trees are transmitted the number of bits/pixel varies in inverse proportion to the fraction of the trees transmitted. For example if 'M2 the trees are transmitted the number of bits per pixel is 2 x 0.8 = 1.6. Thus if the proportion of trees to be sent to the decoder is 1H2 the total number of trees in a frame, and the target bits/pixel for a frame is 0.8, the trees to be actually sent are quantized by a table giving 1.6 bits/pixel.

The quantization is coarsened for increasing numbers of trees whose degree of change exceeds the threshold. In this way the bit rate can be set and the picture quality is maximised for the degree of change.

In embodiments of the invention, syntax data is inserted into the bit stream sent to the decoder. The syntax data identifies: a) the position in the image of a tree if the trees are transmitted in order of power difference instead of image position; b) the quantization applied to the tree if different quantizatons are appied to different trees as discussed above; c) by means of a STOP symbol the place in a tree after which all coefficients are zero; and/or d) by means of a JUMP symbol the place in a subband after which all coefficients are zero.

In an embodiment of the invention not only are the quantization tables chosen in accordance with the said further inventive aspect but also runlength and/or Huffman encoding tables. Thus for each quantization table there is a corresponding Runlength and Huffinan table.

For a better understanding of the present invention and to show how the same may be carried into effect, reference will be made, by way of example, to the accompanying drawings in which: Figure 3 is a schematic diagram of a list of trees; Figure 4 is a schematic block diagram of an illustrative encoder according to the invention; Figure 5 is a schematic diagram of a 1-dimensional store storing a tree structure; Figure 6 is a schematic block diagram of an illustrative decoder according to the invention; and Figure 7 is a schematic diagram of an image frame transformed over 4.5 scales.

An illustrative encoder will now be described with reference to Figure 4. The encoder transmits compressed image data to the decoder of Figure 6.

A video signal representing an image is produced by a camera 40. The video signal comprises a luminance component (Y) and two colour components (C). Figure 4 illustrates only the luminance channel. The colour component channels are similar.

The video signal is sampled by way of example according to the known 4:2:2 sampling system and digitized in an A to D converter 42 to represent a digital CIF image having 352 Y pixels horizontally and 288 Y pixels vertically. The digitised Y component of the image is wavelet transformed in a wavelet transform circuit 44 to produce for each frame a 2 dimensional array of coefficients as shown in Figure 1.

A tree extractor 46 extracts the trees from the wavelet transformed data. The extractor reorders the data in known manner to extract the trees. The trees are then stored in a first, current, tree store 460 of a tree proceesor 48 . A current frame of tree coefficients is stored in the frame store 460 of the tree processor 48. The processor 48 has a second, previous, frame store 462 storing trees from previous frames. The trees are stored in both stores in a predetermined order of position in the image. A control processor 480 controls storage and read-out. The control processor 480 stores, in a list store 482, a list of the trees in the store 462 together with the power differences established for those trees in the manner described below.

The current trees are read out of the frame store 460 by the control processor 480 in the same predetermined order of image positions. Each tree is stored in a 1 dimensional store 464 as shown in Figure 5. The store 464 has 256 coefficient positions corresponding to the set of 256 coefficients forming one tree when 4 scales are used as in Figure 1. The corresponding old tree is read out from the old tree section of store 462 into a similar one dimensional store 466. A power comparator 468 compares the sum of the squares Spi2 of the coefficients current tree with the corresponding sum of the old tree. If the current tree power exceeds the old tree power by the preset threshold Thl, the current tree is listed in the list store 482 overwriting the corresponding old tree. When the current tree is transmitted to the decoder, the current tree is placed in the store 462 overwriting the corresponding old tree. The reason for first listing the current tree and updating the store 462 only when the current tree is sent to the decoder is to ensure that the contents of the store 462 of the encoder and of a corresponding store (662) of the decoder are the same.

A listed tree may not be sent if the target bit rate is reached before the listed tree is sent. Thus the store 462 holds a frame of trees some unchanged from previous frames and some new trees from the current frame, assuming the current frame differs from the preceding frame.

The difference of the powers of the corresponding current and old trees is also determined by the power comparator 468 for storage in the list store 482.

The control processor 480 determines the percentage of the stored trees, relative to the total number of stored trees, for which the power difference exceeds the predetermined threshold Th2 shown in Figure 3. On the basis of that proportion one of a plurality of quantization tables is chosen for quantizing the trees as they are read out of the store 462 thereby setting the bit rate. The control proceesor 480 uses the list to read out the stored trees in descending order of power difference e.g. from the top to the bottom of Figure 3. Because the trees are read out in an order of descending power difference unrelated to their position in the image, the process control inserts syntax data into the bit stream at a syntax inserter 470 or multiplexer, the syntax data indicating the positions in the store 462 from which the trees are read out.

It will be appreciated that although for clarity of description two thresholds Thl and Th2 have been referred to in practice the thresholds Thl and Th2 are the same.

Referring to Figure 4, the trees are quantized by a quantizer 490 according to the selected quantization table. Syntax data is inserted into the bit stream defining the selected table to allow correct decoding in the decoder. The quantized trees are then encoded in an encoder 492, and fed to a circuit 494 which determines the bit rate.

Normally the quantization table has been chosen to give a bit rate lower than the target bit rate. In this illustrative embodiment, the listed trees are fed to the quantizer 470 and encoder 492 in the listed order until the target bit rate (as measured by block 494) is reached. Thus for some frames not only the trees above the threshold are coded but also trees below the threshold are coded.In an alternative embodiment only trees above the threshold are coded.

The encoder 492 uses known loss-less coding techniques such as Runlength coding and Huffman Coding.

The process control 480 includes in the syntax data two codes shown symbolically in Figure 5.

i) End of tree or STOP symbol indicating for a tree that all subsequent coefficients in the tree are zero; and ii) End of sub-band or JUMP symbol indicating that, for a sub-band, all subsequent coefficients in that sub-band are zero: processing will then continue from the next sub-band. One sub-band might be zero but a subsequent sub-band might have significant information in it.

Referring to Figure 6, the decoder comprises a syntax extractor 670, an entropy encoder 692 and a dequantizer 690 complementary to the coder 492 and quantizer 490. A tree processor 648 comprises a tree store 662 and a process control 680. The tree store 662 corresponds to the store 462 and stores the received trees at the same addresses at which they were stored in store 462. The trees received by the decoder are those which have been encoded in the coder together with the addresses of those trees. The syntax extractor 670 extracts the addresses and the process control 680 stores the trees at the addresses overwriting any previously stored trees.

Thus store 662 stores a set of trees representing a wholly updated image. The set stored in store 662 is identical to the set stored in store 462 because the set stored in store 462 is updated only when a tree is sent to the decoder and thus store 662.

An inverse transform circuit 644 transforms the wavelet coefficients back to pixels of the time/amplitude domain. The pixels are converted to analog form in a D/A converter 642 and displayed on a display 640.

The encoded signal produced by the encoder may be transmitted in a signal transmission system such as an ATM (Asynchronous Transfer Mode) network and/or be stored on a suitable recording medium e.g. digital video tape.

The decoder of Figure 6 receives the encoded signal from the transmission system and/or the recording medium.

The store 462 usually stores at least some trees unchanged from a previous frame or frames. If the image changes only slowly the trees may be stored for long periods, e.g. greater than 1 second. It may be desirable to ensure that all trees stored in the store 462 are regularly replaced e.g. every 1/2 second or every second.

In the preferred embodiment of the invention, each quantization table comprises the same number of quantization coefficients as there are image coefficients in a tree. Thus for the example given above where a tree has 256 image coefficients, there are 256 quantization coefficients in 1:1 correspondence with the image coefficients. The quantization coefficients are determined experimentally using a set of representative image sequences to provide optimum fidelity at the target bit rate.

Different sets of quantization tables are provided for the luminance component and colour components respectively. Likewise, different Huffman and Run-Length coding tables are used in conjunction with different quantization tables.

In the above described embodiment, the trees are quantized one at a time in the listed order. In an alternative embodiment all the listed trees are quantized an the quantized trees are then sent in the listed order to the coder 492.

The invention has been illustrated with reference to the embodiment of Figures 4, 5 and 6. Various modifications may be made to the embodiment. For example, the target bit rate of 2Mbits/sec is only an example. Other bit rates may be used.

Although the embodiment uses a 4:2:2 sampling system, other sampling systems such as 4:1:1 could be used. Although a 2D array of transformed coefficients has been described, the invention could be applied to 1-D or 3-D arrays for example.

Although the invention has been described with reference to a transform applied over 4 scales as shown in Figure 1, the transform could be applied over other numbers of scales which numbers need not be integers. For example it could be applied over 3 scales or over 5 scales in which case a block size and a tree comprises 32x32 = 1024 coefficients. In a currently preferred embodiment the transform is applied over 4.5 scales as shown in Figure 7 and the block size and tree size is 32x16 coefficients.

Although the invention has been described in relation to a CIF image it can be applied to any image size.

The camera 40, A/D converter 42 and wavelet transformer 44 may be provided by suitable discrete systems. The functions of blocks 48, 470, 490, 492 and 494 may be implemented using a suitably programmed computer of sufficient power and processing speed. The corresponding parts of the decoder may be similarly implemented.

The invention has been illustrated with reference to processing frames. It may be applied to fields instead of frames. Herein and in the claims fields and frames are referred to generically as "pictures".

Claims (18)

1. A video compression encoder comprising: an input for receiving an image signal, an output for outputting a compression encoded image signal, transform means (44) for producing from each picture of the image signal an array of spatial frequency coefficients representing the image in a plurality of frequency bands, means (46) for extracting from each picture of the transformed image signal trees comprising sets of coefficients representing correlated image areas in the said bands, means (490, 492) for comparing corresponding trees from successive pictures and to determine which trees in one picture which have changed compared to the corresponding trees in a previous picture, and means for selecting trees for output (48), and compression encoding (490,492), in dependence upon the comparison.

2. An encoder according to Claim 1, wherein the encoding means applies a selected one of a plurality of different quantizations to the selected trees, the applied quantization being selected according to the proportion of trees relative to the whole number of trees in a picture for which the degree of change is greater than a preset amount.

3. An encoder according to claim 2, further comprising means for providing syntax data indicating the selected quantization.

4. An encoder according to Claim 1, 2 or 3, comprising means for storing trees of a picture, and means for listing the stored trees in order of degree of change.

5. An encoder according to claim 4, wherein all stored trees are replaced within a predetermined time interval.

6. An encoder according to Claim 4 or 5, wherein the trees are listed in descending order of degree of change.

7. An encoder according to claim 4, 5, or 6 wherein the selecting means selects the listed trees in descending order of degree of change.

8. An encoder according to claim 7, comprising means for measuring the bit rate, the listed trees of a picture being selected in the listed order until the measured bit rate reaches a preset level.

9. An encoder according to Claim 1, 2, 3, 4 or 5, wherein a tree in one picture is selected if the difference in degree of change of that tree and the corresponding tree from a previous picture exceeds a threshold.

10. An encoder according to Claim 6, wherein only trees for which the said degree of change exceeds the said threshold are selected.

11. An encoder according to any preceding Claim, wherein the comparing means compares the powers of the said corresponding trees.

12. An encoder according to any preceding claim, further comprising means for providing syntax data indicative of the positions of the selected trees in the image.

13. An encoder substantially as hereinbefore described with reference to Figure 4 together with Figures 1 to 3 and 5 optionally as modified by Figure 7.

14. A video compression decoder for use with the encoder of any preceding claim comprising means for compression decoding trees, means for storing decoded trees, means for replacing stored trees by newly decoded corresponding trees, and means for inverse transforming the stored trees.

15. A decoder according to claim 14, wherein trees of one picture are received in descending order of degree change relative to the corresponding trees previously stored in the storing means together with syntax data indicating the positions of the trees in the image, and further comprising means for extracting syntax data, the replacing means being responsive to the syntax data to store the trees in the storing means at locations indicated by the syntax data.

16. A decoder according to claim 14 or 15, wherein the encoded trees are quantized in the encoder according to a selected one of a plurality of quantizations and include syntax data indicating the selected quantization, and further comprising means for dequantizing the decoded trees in accordance with the quantization indicated by the syntax data.

17. A decoder substantially as hereinbefore described with reference to Figure 6.

18. A compression and decompression system comprising an encoder according to any one of claims 1 to 13 and a decoder according to any one of claims 14 to 17.