EP1163800A1

EP1163800A1 - Image compression and decompression

Info

Publication number: EP1163800A1
Application number: EP00912762A
Authority: EP
Inventors: Julian Hodgson
Original assignee: Imagination Technologies Ltd
Current assignee: Imagination Technologies Ltd
Priority date: 1999-03-22
Filing date: 2000-03-22
Publication date: 2001-12-19
Also published as: WO2000057649A1; GB2348334A; JP2002540685A; GB9906603D0

Abstract

The present invention compresses image data using a predetermined compression technique such as the Microsoft S3 compression scheme. The compressed image is then decompressed and difference values derived between the original image and the decompressed image. The thus derived difference values are then compressed for used in sub-different correction of the decompressed image and are transmitted or stored along with the compressed image data.

Description

IMAGE COMPRESSION AND DECOMPRESSION

This invention relates to image compression and decompression and is particular-.y applicable co compression and decompression tecnmques of types similar to the S3 compression tecnnoiogy licenced by Microsoft. Image compression consists of making a new representation of the image aata in digital form that takes up less storage space tnan the origina . Many compression techniques are referreα to as being lossy because they do not recover all tne original data but instead make some approximation to this.

Often, the pixels tnat represent an image are stored in what is known as RGB format. This means tney are built from three multi bit numioers which respectively represent the amount of red green and blue colours to be applied to a particular pixel. Anotner colour format is known as YUV. In this colour is encoded such that the brightness of the image is represented by the Y value ana can be used to display black and white images whilst the colour is contained in the U and V values. These colour formats can be grapnically represented using an orthognal coordinate system to define what s known as colour space.

In some cases translucency data is included with image data. This information is used when one image is shown against another image or a background and the translucency or alpha information is then used to combine one image with the otner. This is particularly useful when blending tne edge of one image into anotner. A special type of translucency <nown as punch-tnrough can be used to switch on and off translucency wnilst full variable translucency can be used for objects such as perspex which might have fractional or variable alpha values .

In most compression techniques blocks of pixels are operated on by the compression technique. In this patent application we will usually oe referring to a 4x4 pixel blocκ although the invention may of course apply to any block size.

The S3 compression technology referred to aoove provides a technique of lossy image compression. It contains tnree different compression scnemes <nown as DXT1, DXT2, ana DXT3. The first of these is in principle an opaque metnod of compression in that its main role is to compress RGB pixels. It does cater for a particular type of translucency known as "punch tnrougn" mch is used when translucency or alpha value is either on or off. The otner two compression schemes DXT2 and DXT3 are used oniy for translucency information where the alpha value is between 1 and 0. DXT1 operates by first splitting the image into blocks of 4x4 pixels. Colours of each of the pixels m the block are analysed and the two points furthest apart in colour space are derived from this. Each of these end points is then represented by a 16 bit colour value in format RGB565 (meaning 5, 6, and 5 bits to each of the R, G and B channels respectively) An index of two bits per pixel is then used to assign a colour value a_ong the line joining the two end points in colour space tc eacn of the pixels m the 4x4 bloc . The possible values of the index for eacn pixel are 00, 01, 10, and 11. 00 represents one end point ana 11 the other. The two other values represent intermediate points along the line ,/nιch are derived oy interpolation during decompression.

The cnoice of representative colours need not be the two furthest apart values . More complex scnemes which iterate to reduce errors could be used, and /e gntings based on e.g. colour perception ana other factors could be applieα.

Index values are assigned to each oixel at the compressor oy determining which of tne four possible colour values corresponds most closely to tne original colour value of that pixel. Thus, DXT1 provides a scheme whereby a DIOCK of 4x4 pixels wnich each originally were representeα by, for example, a 24 bit RGB colour value have their image data compresseα to two 16 bit colour values a a an index of two bits per pixel thereoy comprising a total of 64 oits.

Puncn through translucency is dealt with m DXT1 by having or._y one intermediate colour value in colour space and using the fourth index value to determine whether or not the translucency is on or off for a particular pixel. Thus, there is a reduction in the number of available colours for the 4x4 pixel block and the quality of the representation of colour in the block is thereby reduced when puncn-through is selected.

DXT2 is used purely for translucency. It operates merely by restoring the alpha information at a lower precision. Therefore, it does not work particularly well on alpha layers that are stored at high precision.

DXT3 is another compression scheme for storing the alpha data. It does this m a similar manner to the colour method described above. For each 4x4 block two 8 bit alpha values are stored along with a 3 bit per pixel index to select between the two 8 bit alpha values and six interpolated values oetween these two extremes.

Problems with image quality can arise in schemes such as DXT1 described above wnen an image block contains an edge between two different colours. As the colour data for the block is generated by only two representative colours it is only possible to represent graduations between the two colours and not graduations of tne two colours. Hence the scheme does not compress tne image so well ana in some cases this can result in the decompressed image having a blocky appearance. This is generally not acceptable .

Preferred embodiments of the present invention seek to reduce the blockmess of scnemes such as DXT1 and to improve the visual quality of the image after compression and decompression by reducing visual artefacts as well as by reducing more easi±y measured criteria sucn as tne least squares error.

Embodiments of the invention can aiso be applied to the otner DXT compressions scnemes descnoeo above which are used for translucency information.

The invention is designed with more precision in the appended claims to wnicn reference should now be made.

A preferred embodiment of the invention will now be described in detail by way of example with reference to the accompanying drawings m which:

Figure 1 is block diagram of a compression unit embodying the invention; ana

Figure 2 is a block diagram of a decompression unit embodying the invention. The embodiment descπoed comprises an improvement to the quality of the DXT1 image compression. The basic scheme as described above compresses RGB colour data for a 4x4 pixel block into two RGB565 representative colours totalling 32 bits and a 2 bit index per pixel indicating which interpolated colours to use . This makes another 32 bits giving 64 bits m all.

In Figure 1 which shows a olock diagram of the compression stage it can oe seen that the original pixels a enter the unit and are compressed according to the Dxtl scheme in the S3 compression unit 2 which outputs a compressed approximation layer f for each block.

In this embodiment the unit 2 also outputs the compression approximation layer at output b to be supplied to a decompression unit 4. This decompresses the compressed approximation layer in the same manner as it would be decompresseα normally. The output of this c is the data whicn would be used to αispiay tne original image in decompressed form. This data and tne original pixel data a is fed to a subtraction unit 6. This _s used to derive a difference Detween the original pixels and the decompresseα pixels.

Tnus, for eacn pixel t ere will be a difference for each of the RC3 values ana this is proviaed at an output d. Th s difference value then passes to a transformation unit 8. This is optional but it will be used in a preferreα emoodiment The purpose of this transformation unit is to determine tne range of error values for each 4x4 blocκ and to apply a scaling factor to them to ensure that t.ney can subsequently be compressed and transmitted with tne compressed -. age data in a form which will give reasoπaoie accuracy en decompression. The scaling factor applied to the block s provided at an output g.

The other output of the transformation unit is the scale values themselves. These are passed to another DXT1 compression unit which compresses the difference pixel data in a manner similar to that of the approximation layer. Two representative difference colours are selected to represent the extremes in colour space of all the difference pixels in the clock and are stored in RGB565 format. A 2 bit per pixel index is then used to indicate which out of these two end points and two intermediate points on tne line joining these end points s.nould be assigned to each pixel in the clock. Thus, a compressed error layer h s output by unit 10 comprising a further 64 bits in addition to tne 64 bits of the compressed approximation layer. The compressed approximation layer, tne compressed error layer, and the scaling factor for each block can then be stored or transmitted as required for subsequent decompression. The system may be simplified by omitting the transformation unit and sending unsealed data.

The numoer cf Pits used for the error layer could be greater or fewer than that used for the compressed approximation layer. The bits used to represent the scaling factor are included m the approximation and error layers. The first bit comes from the ordering of representatives in tne approximation layer, e.g. ascending order denotes 1 and desceⁿding denotes 0. Similarly the second oit comes from t"e ordering of representatives in the error layer. The image data is decompressed in the unit of Figure

2. This operates in the reverse manner to the compression

The approximation layer f passes to a DXT1 decompression unit 12 Tnat provides a decompressed output k. At the same t me, the error layer for the equivalent block passes to a DXT1 decompression unit 14 which provides decompressed errors l for each pixel in the block. The scaling factor g passes to the transformation unit 16 which performs the reverse scaling of the transformation unit 8 on tne scaled decompressed pixels to provide the unsealed decompressed errors j (this is omitted if unsealed data is sent) . The decompressed pixel data k and decompressed error data j is then combined for each pixel m an addition unit 18 to give decompressed and error adjusted pixels at output 1.

The scaling factor stored can scale the error values by a fixed amount per block. This enables the method to dynamically change the range of errors stored m the different block h thereby enabling maximum resolution to be given to each error. The system could be modified to provide a different scaling factor for each of the RG and B values.

In this particular implementation, two bits per block are used to specify whicn scaling is used out of a set of eg. 16, 32, 64 and 128, which we refer to as the scaling set. The source of these pits is discussed oelow.

For example, -_f the original RGB values m the pixel are (76, 87, 129) and tne approximation layer produces a pixel with value (62, 92, 116) after decompression, then the difference pixel fias a value of (6, 5, -13) . As the maximum absolute difference for each colour component is less than 16, it is appl cacle to choose the range 16 from the scaling set to represent all of the errors. The range represented by the number 16 can represent any value between -15 and 15, and nence the value 15, or in the general case the value (range -1) is added to tne difference values to give a scaled difference pixel with values ( 21 , 20, 2) thus, each component is in the range 0- 31 or 0- 'range -1) . The numoer is then scaled so that it occupies tne highest possible range of, in this case, 8 bits. In this case, each component will be multiplied by 8 for scaling purposes. Thus, the scaled and translated difference pixel is (168, 160, 16) . With all 16 of the transformed difference pixels in the block being found they are then compressed using the DXT1 method.

Other scaling sets and methods for selecting scaling functions can be used, e.g. maximums, average, or some other measure.

The reverse process follows in the decompression unit. If the decompressed difference pixel of the same pixel as above is (175, 155, 20), then the scaled back pixel is given by (21, 19, 3) . This then gives the decompressed difference pixel as (6, 4, -12) . This can then be subtracted from the decompressed values of the original pixel, namely (62, 92, 116) to give (56, 88, 128) . Which is much closer to the original pixel value of (56, 87, 129) than the pixel value in the approximation layer .

It is possible that some values may exceed the largest possible difference value which here s 127. In such a case the errors would be clamped to be within this range.

The two bits that are used to choose the scaling factor from the scaling set are chosen in the same way as the S3 DXT1 compression scheme chooses oetween regular and punch through blocks . This involves placing tne two representative values in either ascending or descending order to indicate whether or not a scaling factor is present, as explained above. Tne compression unit snown in Figure 1 could be modifieα so that only one DXTl compression unit is requireα. This wouiα involve tne use of a feedback mechanism to provide tne compressed error layer from compress_on unit 2 as well as tne compressed approximation layer. Similarly, m decompression it is possible to use only one αecompression unit oy providing suitable multiplexing between the error approximation layers and a decompression unit and tne transformation unit 16. The compression scheme can in an alternative embodiment be used with puncn through translucency. In this, one value of the mαex represents a punch through value. Tne fact that this value exists in the index is represented by the order of the two representative colours or the two error values. If the first representative colour or error value storeα is a larger value than the second then one of the index values represents punch through. If the representative colours or error values are m the opposite order then all tne index values are used to represent points on tne line in colour space.

When this is used then the reduction in quality is much less than in the basic DXTl compression scheme because of the error correction process provided in the embodiment of the invention. In such an arrangement, the t/o bits used for the scaling function are then provided by using RGB 555 instead of RGB 565 for the approximation error layer representatives. This frees up an additional 2 bits for use as the scaling function. Clearly, the punch through bit could be one of the b ts released by RGB 555, as desired in a particular application.

In another embodiment of the invention two layers are again useα for transmission. The first of tnese consists of four pixels which are stored in colour format RGB 444 which takes up total storage of 48 bits. These pixels each represent the average colour in each of the four 2x2 blocks m the 4x4 block. These average pixels are stored explicitly after truncation to RGB format. The second layer tnen consist of an error or difference layer which is adαeα to eacn of the average pixels for correction to a closer approximation to eacn of the original pixel values. A scaling s applied to eacn 2x2 sub bloc to bring all the errors in each sub blocκ: into the same range in a manner similar to that described aoove .

In another embodiment, an error layer is added to the DXT2 or DXT3 schemes for compressing the alpha or translucency value. The method by wnich this is done is analogous to the method described aoove . In this, an error layer is added to eacn DXT3 compression scheme. As descnoεd above this scheme produces two 8 bit alpha values for each sub block with an index of 3 oits to a pixel to represent intermediate alpha values which are generated from the representative values. This is modifieα such that the approximation layer consists of two 8 bit alpha values followed by an index of 2 bits per pixel _nstead of three. The error layer then consists of two 8 bit difference representatives and an mαex of 2 bits per pixel to point to a total of 4 interpolated difference values generated from the two representative alpha "alues. The error per olock can then ce scaled based en the average or maximum error per blocκ and truncated before compression to give better resolution in blocks with small errors. As n the opaque case, 2 bits resulting from ordering of the representatives can be used to specify a scaling value from a scaling set of size 4.

Tne invention can be used on pixels whicn are stored in other colour formats such as YUV. When this is done it can be helpful to split the pixels up into different groups as motivated by the sections of colour components. Pixels stored in YUV format can be representeα -ismg three schemes. Firstly, an approximation of the Y cr.annel is provided using two representatives in a manner similar to the alpha only compression method above ana a 2 bit index per pixel. Then a second approximation layer for the U and V channels is provided again using two \ representatives and a 2 oit index per pixel. Finally, the Y approximation layer _s corrected with the layer consisting of an error ir Y. In this example the UV layers not corrected an -.t is deemed less important although this could be provided in another embodiment of the invention.

Finding the best t to representatives or end points for the pixel error and approximation layers need not necessarily be done by cnoosmg two furthest apart points in colour space. Some form of training and iteration can be used as follows.

Firstly, the best t-vo representatives or end points for the pixels are founα without considering the interpolated points between them. This is done by taking the two furthest apart pixels m colour space and then performing a small number of generalised Lloyd algorithm iterations to find a better pair of representatives. These are the initial configuration or end point values. Alternatively, we could simply start with the furthest of pixels.

Another way to find the end point codes is to do some multivariate analysis on the pixels, and find the mean and principal direction of tne data by taking the eiganvalues of the covariates matrix of the points. At this point, the codes can be accepted and the next stage Joegms .

Alternatively, a second stage of iteration can be performed by doing a nu oer of iterations where the best fit two end point codes are improved upon. This is done to satisfy a least squares condition with respect to the nearest mterpretted codes to the pixels at tnat given iteration. Finally, the iteration could include some simulated annealing to fry and escape the local minima instead of just accepting smaller errors.

Other arrangements could be implementeα such as providing an approximation of YUV representatives corrected only with a layer consisting of errors or differences in Y. This type of image compression and decompression is typically used in video graphics such as eg computer games for use m the home ana supplied on some form of machine readable αata carrier along with software relating to the game. It can be also oe used in larger purpose built arcade type video games. It could also be used m realtime video transmissions.

In the games type applications the compression of data is performed before the software is sold. Thus, the compression unit such as the S3 unit would be modified m accordance with an embodiment of the invention either in hardware to provide a unit as shown in figure 1 above or, if all compression is provided in software this would be done by modifying the software used to compress images using S3 technology.

Claims

1. A method of compressing a digital image data comprising the steps of: compressing tne image data using a predetermined compression technique; decompressing tne thus compressed image; deriving difference values between the original image ana the decompressed image; compressing the thus derived difference values for use m subsequent correcting of the decompressed image ; and providing compressed image data and compressed difference values for decompression.

2. A method according to claim 1 in wnich the difference values are compressed using the same compression method as the image.

3. A method according to claim 1 and 2 in which the difference values nave a scaling factor applied to them prior to compression and //herein the scaling factor is also provided for decompression.

4. A method according to claim 1, 2, or 3 in which the image data comprises colour data.

5. A method according to claim 1, 2, 3 or 4 m which the image data comprises translucency αata.

6. Apparatus for compressing digital -.mage data comprising; means for compressing the image data using a predetermined compressing technique; means for αecompressing the compressed image data; means for deriving a difference value from the original image data and tne decompressed image αata; means for compressing the thus deπveα difference values; and means for providing the compressed image data and compressed difference values for subsequent decompression.

7. Apparatus according to claim 6 in whicn the means for compressing the difference values uses the same compression technique as the means for compressing the image data.

8. Apparatus according to claim 6 and 7 including means for scaling the difference values prior to compression and wherein the scaling factor is also provided for decompression.

9. A method for decompressing compressed digital image data comprising the steps of: decompressing the compressed image data using a predetermined decompressing technique; decompressing compressed difference values associated with the compressed image data; and correcting the decompressed image data with the decompressed difference values.

10. A method according to claim 9 in which the compressed image data and difference values are both decompressed using the same decompression technique.

11. A method according to claim 9 and 10 n which the difference values are scaled difference values and the method further includes the step of applying a reverse scaling factor to the decompressed difference values.

12. Apparatus for decompressing compressed digital image data comprising: means for decompressing the compresseα image data according to a predetermined decompression technique; means for decompressing compressed difference values associated with the compressed image data; and means for correcting the decompresseα image data with tne decompressed difference values.

13. Apparatus according to claim 12 in which the means for decompressing the image dat and the means for decompressing the difference values both use the same decompression technique.

14. Apparatus according to claim 12 or 13 m which the difference values are scaled difference values and further comprising means for applying a reverse scaling factor to the scaled difference values.

15. A computer program product comprising image data compressed according to the method of claim 1, 2, 3, 4, or 5.

16. A machine readable data carrier comprising image data compressed according to the method of claim 1, 2, 3, 4, or 5.

17. A computer program product comprising a set of instructions to configure a computer to compress digital image data according to the method of claim 1, 2, 3, 4, or 5.