AU2003201369B2 - Image Compression in Retained-mode Renderer - Google Patents

Description

S&F Ref: 627652

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT

ORIGINAL

Name and Address of Applicant: Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha 30-2, Shimomaruko 3-chome, Ohta-ku Tokyo 146 Japan Gregg Richard Kelly Spruson Ferguson St Martins Tower,Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Image Compression in Retained-mode Renderer ASSOCIATED PROVISIONAL APPLICATION DETAILS [33] Country [31] Applic. No(s) AU PS1276 [32] Application Date 20 Mar 2002 The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5815c -1- IMAGE COMPRESSION IN RETAINED-MODE RENDERER Technical Field of the Invention The present invention relates generally to image compression and, in particular, to a method of compressing image data. The invention also relates to a computer program product including a computer readable medium having recorded thereon a computer program for compressing image data.

Background Computer generated images are typically made up of many differing components or graphical elements which are rendered and composited together to create a final image.

In recent times, an "alpha channel" (also known as a "matte", an "opacity channel", or simply "opacity") has been commonly used in generating images. The alpha channel contains information regarding the transparent nature of each element. The alpha channel is stored alongside each instance of a colour, so that, for example, a pixel-based image with opacity stores an alpha (or opacity) value as part of the representation of each pixel.

An element without explicit alpha channel information is typically understood to be fully opaque within some defined bounds of the element, and assumed to be completely transparent outside those bounds.

An expression tree offers a systematic means for representing an image in terms of its constituent elements and which facilitates later rendering. Expression trees typically comprise a plurality of nodes including leaf nodes, unary nodes and binary nodes. Nodes of higher degree, or of alternative definition may also be used. A leaf node, being the outer most node of an expression tree, has no descendent nodes and represents a primitive constituent of an image. Unary nodes represent an operation which modifies the pixel data coming out of the part of the tree below the unary operator. Unary nodes include such operations as colour conversions, convolutions (blurring etc) and operations such as red-eye removal. A binary node typically branches to left and right 627652.doc -2subtrees, wherein each subtree is itself an expression tree comprising at least one leaf node. Binary nodes represent an operation which combines the pixel data of its two children to form a single result. For example, a binary node may be one of the standard "compositing operators" such as OVER, IN, OUT, ATOP and alpha-XOR, examples of which and others are seen in Fig. 7.

Several of the above types of nodes may be combined to form a compositing tree 800, as shown in Fig. 8, representing an image where a representation of each portion of the image is retained at each of the nodes 801). For the compositing tree 800, the result of the left-hand side of the tree 800 is retained at node 803, and may be interpreted as a colour converted image being clipped to spline boundaries. A path region represented by these spline boundaries is stored at the node 801 of the tree 800 and the resulting region representing the clipped image is stored at node 803. The clipped image region is composited with a second image which is stored at node 805.

Although the non-transparent area of a graphical element may of itself be of a certain size, it need not be entirely visible in a final image, or only a portion of the element may have an effect on the final image. For example, assume an image of a certain size is to be displayed on a display. If the image is positioned so that only the top left corner of the image is displayed by the display device, the remainder of the image is not displayed. The final image as displayed on the display device thus comprises the visible portion of the image, and the invisible portion in such a case need not be rendered.

Another way in which only a portion of an element may have an effect on a final image is when the portion is obscured by another element. For example, a final image to be displayed (or rendered) may comprise one or more opaque graphical elements, some of which obscure other graphical elements. Hence, the obscured elements have no effect on the final image.

627652.doc -3- Many conventional compositing models consider each node of a compositing tree to be conceptually infinite in extent. Therefore, to construct the final image, such a conventional system applies a compositing equation at every pixel of the output image.

Interactive frame rates of the order greater than 15 frames per second can be achieved in such a system, since pixel operations are quite simple and can be highly optimised. These highly optimised pixel operations can be fast enough to produce acceptable frame rates without requiring complex code. However, this is certainly not always true in a compositing environment.

The per-pixel cost of compositing can be quite high since an image is typically rendered in 24-bit colour Red, Green, Blue (RGB)) in addition to an 8-bit alpha channel, thus giving 32 bits per pixel. Each compositing operator has to deal with each of the four channels. Therefore, an approach of completely generating every pixel of every required frame when needed is highly inefficient.

The basic shape of operands to compositing operators in most current rendering systems is a rectangle or bounding box, regardless of the actual shape of the object being composited. Thus, it is a relatively simple operation to write an operator which composites within the intersection area of two bounding boxes. However, as a bounding box typically does not accurately represent the actual bounds of a graphical element, such an operation can result in a lot of unnecessary compositing of completely transparent pixels over completely transparent pixels. Furthermore, when the typical make-up of a composition is examined, it can be noticed that areas of many graphical elements are completely opaque. This opaqueness can be exploited during the compositing operation.

However, these areas of complete opaqueness are usually non-rectangular and so are difficult to exploit using compositing arguments described by bounding boxes. If irregular regions are used for exploiting opaque objects when compositing, then these regions can then be combined in some way to determine where compositing should occur.

627652.doc -4- Furthermore, if any such region is known to be fully transparent or fully opaque, further optimisations are possible.

Most conventional rendering systems fail to exploit similarities in a composition of graphical elements between one frame and the next. Since it is rare for every graphical element of an image to change from frame to frame, large areas of a compositing tree typically remain unchanged. Thus, it is only necessary to render those components of an image that have been altered by a particular action. If incremental changes are made to a compositing tree, then only a reduced amount of updating is necessary to affect the change.

Many current graphical systems use what is known as an 'immediate mode' application program interface (API). This means that for each frame to be rendered, a complete corresponding set of rendering commands is sent to the API. However, sending the complete set of rendering commands is somewhat inefficient in a compositing environment since, as discussed above, large sections of the compositing tree typically remain unchanged from one frame to the next, but are completely re-rendered anyway in an immediate mode API. On the other hand, a 'retained mode' API provides a complete compositing tree on a per-frame basis. In this case, a user provides an initial compositing tree, and then modifies the tree for each frame in order to effect a change. Changes which can be made to the tree include geometrically transforming part or all of the tree, modifying the tree structure (unlinking and linking subtrees), and modifying attributes (eg: colour) of individual nodes. Note that such modifications may not necessarily mean that the tree structure, for example as seen in Fig. 8, will change where only the attributes of an individual node have been modified.

A retained mode rendering system uses the combination of a number of techniques and assumptions to provide high quality images and high frame rates. Firstly, a retained mode rendering system uses irregular regions to minimise per-pixel 627652.doc compositing. For example, if one graphical element is on top of another, then pixel compositing is only needed inside the area where the two elements intersect. Having the ability to use irregular regions gives the ability to narrow down areas of interest much more accurately. Secondly, if an opaque graphical element is composited with another element using the OVER operator, for example, then the opaque graphical element completely obscures what the element is composited onto (inside the opaque element's area). This is a very useful property since it means that no expensive pixel compositing is required to achieve the output pixel within the area of overlap since the pixel value is the same as that at the equivalent point on the opaque element. Opaque elements induce similar behaviour in most of the compositing operators described above. Thirdly, in a retained mode rendering system it is often necessary to repeatedly redraw the compositing tree as graphical elements of the tree are updated. Therefore, an assumption is made that in the transition from one frame to the next, only part of the compositing tree will change.

This particular property can be exploited by caching away expensive-to-generate information regarding a composition so that the cached information can be re-used in a subsequent redraw operation from one frame to the next. Examples of expensive-togenerate information are regions of interest boundaries of areas of intersection between graphical elements etc); pixel data representing expensive composites ete); and topological relationships between graphical elements.

Caching away expensive-to-generate information regarding a composition of graphical elements, in order that the information can be reused in subsequent redrawing operations, can greatly improve processing and thus redrawing speed. However, if too much memory is used for storing the information, the speed of redrawing can actually be decreased due to increased memory paging.

One known method for compositing graphical objects in a retained mode rendering system, uses image data compression and decompression techniques in order to 627652.doc -6reduce the amount of memory required for storing images and in particular expensive-togenerate information. However, conventional compression techniques, such as the Joint Photographics Expert Group (JPEG) standard, tend to be too slow and thus, the processing speed lost due to compressing and decompressing images can outweigh any advantage gained through performing such compression and decompression.

Accordingly, an improved compression method for use within a rendering system, and in particular a retained mode rendering system, is clearly needed.

Summary of the Invention It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

According to one aspect of the present invention there is provided a method of compositing a plurality of graphical objects according to a predetermined compositing operation, at least one of said graphical objects being in the form of a compressed representation, said compressed representation comprising one or more coded pixel runs each representing one or more pixel difference values, each of said coded pixel runs having an associated byte code indicating a type for the associated coded pixel run, said method comprising the steps of: decompressing at least one coded pixel run of said compressed representation by: examining the byte code associated with said coded pixel run to determine the type of said coded pixel run; and determining pixel values for said coded pixel run, depending upon the type of said coded pixel run, using the pixel difference values represented by said coded pixel run; and compositing at least a portion of the pixel values determined for said compressed representation with pixel values for at least another one of said graphical objects, 627652 in accordance with said predetermined compositing operation, to determine one or more runs of composited pixel values representing the composited graphical obj ects.

According to another aspect of the present invention there is provided an apparatus for compositing a plurality of graphical objects according to a predetermined compositing operation, at least one of said graphical objects being in the form of a compressed representation, said compressed representation comprising one or more coded pixel runs each representing one or more pixel difference values, each of said coded pixel runs having an associated byte code indicating a type for the associated coded pixel run, said apparatus comprising: decompressing means for decompressing at least one coded pixel run of said compressed representation by: examining the byte code associated with said coded pixel run to determine the type of said coded pixel run; and determining pixel values for said coded pixel run, depending upon the type of said coded pixel run, using the pixel difference values represented by said coded pixel run; and compositing means for compositing at least a portion of the pixel values determined for said compressed representation with pixel values for at least another one of said graphical objects, in accordance with said predetermined compositing operation, to determine one or more runs of composited pixel values representing the composited graphical objects.

According to still another aspect of the present invention there is provided a computer program product including a computer readable medium having recorded thereon a computer program for compositing a plurality of graphical objects according to a predetermined compositing operation, at least one of said graphical objects being in 627652 -8the form of a compressed representation, said compressed representation comprising one or more coded pixel runs each representing one or more pixel difference values, each of said coded pixel runs having an associated byte code indicating a type for the associated coded pixel run, said program comprising: code for decompressing at least one coded pixel run of said compressed representation by: examining the byte code associated with said coded pixel run to determine the type of said coded pixel run; and determining pixel values for said coded pixel run, depending upon the type of said coded pixel run, using the pixel difference values represented by said coded pixel run; and code for compositing at least a portion of the pixel values determined for said compressed representation with pixel values for at least another one of said graphical objects, in accordance with said predetermined compositing operation, to determine one or more runs of composited pixel values representing the composited graphical objects.

Other aspects of the invention are also disclosed.

Brief Description of the Drawings One or more embodiments of the present invention will now be described with reference to the drawings, in which: Fig. 1 is a schematic block diagram of a general-purpose computer upon which arrangements described can be practiced; Fig. 2 is a flow diagram showing a method of compressing image data; 627652 Fig. 3 is a flow diagram showing a method of decompressing image data; Fig. 4 shows two intersecting graphical elements; Fig 5 is a flow diagram showing a method of compositing the graphical elements of Fig. 4; Fig. 6 shows an example of a compressed image data output stream; Fig. 7 shows the result of a variety of compositing operators useful with the described arrangements; Fig. 8 is an example of a compositing tree; Fig. 9 shows an image containing a number of overlapping objects and a corresponding compositing tree; Fig. 10 shows regions formed by combining two circles with non-grid-aligned regions; and Fig. 11 shows regions formed by combining two circles with grid-aligned regions.

Detailed Description including Best Mode Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

A method 200 of compressing image data is described below with reference to Fig. 2. In the method 200, an image is compressed on a scanline basis utilizing a scanline table. The method 200 is particularly suitable for images which consist of regions of uniform or blended colour and supports transparency and partial transparency in images, by utilising an alpha (or opacity) channel associated with each graphical element of an image. The method 200 results in compressed image data, which is configured as a stream 627652 of bytes representing one or more portions of an original image. The compressed image data requires much less memory storage space than the original image data. The method 200 is particularly advantageous in a retained mode rendering system, as described above.

As will be explained in detail below, the configuration of the compressed image data output stream, resulting from the method 200, comprises a scanline table followed by a series of pixel difference runs. Each pixel difference run represents one or more differences between adjacent input pixel values of a corresponding input pixel scanline.

The preferred pixel format is "Red, Green, Blue, Alpha (RGBA), at 32-bits per pixel".

However, any suitable pixel format can be used in the method 200.

A method 300 of decompressing image data, which was compressed according to the method 200, is also explained below with reference to Fig. 3. Further, a method 500 of compositing graphical elements, using the methods 200 and 300, is explained below with reference to Fig. The methods described herein are preferably practiced using a general-purpose computer system 100, such as that shown in Fig. 1 wherein the processes of Figs. 2, 3 and may be implemented as software, such as an application program executing within the computer system 100. In particular, the steps of the methods 200, 300 and 500 are effected by instructions in the software that are carried out by the computer. The instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part performs the compression method and a second part manages a user interface between the first part and the user. The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer from the computer readable medium, and then executed by the computer. A computer readable medium having such software or computer program 627652.doc 11 recorded on it is a computer program product. The use of the computer program product in the computer preferably effects an advantageous apparatus for compressing image data.

The computer system 100 comprises a computer module 101, input devices such as a keyboard 102 and mouse 103, output devices including a printer 115 and a display device 114. A Modulator-Demodulator (Modem) transceiver device 116 is used by the computer module 101 for communicating to and from a communications network 120, for example connectable via a telephone line 121 or other functional medium. The modem 116 can be used to obtain access to the Internet, and other network systems, such as a Local Area Network (LAN) or a Wide Area Network (WAN).

The computer module 101 typically includes at least one processor unit 105, a memory unit 106, for example formed from semiconductor random access memory (RAM) and read only memory (ROM), input/output interfaces including a video interface 107, and an I/O interface 113 for the keyboard 102 and mouse 103 and optionally a joystick (not illustrated), and an interface 108 for the modem 116. A storage device 109 is provided and typically includes a hard disk drive 10 and a floppy disk drive 111. A magnetic tape drive (not illustrated) may also be used. A CD-ROM drive 112 is typically provided as a non-volatile source of data. The components 105 to 113 of the computer module 101, typically communicate via an interconnected bus 104 and in a manner, which results in a conventional mode of operation of the computer system 100 known to those in the relevant art. Examples of computers on which the described arrangements can be practiced include IBM-PC's and compatibles, Sun Sparcstations or alike computer systems evolved therefrom.

Typically, the application program is resident on the hard disk drive 110 and is read and controlled in its execution by the processor 105. Intermediate storage of the program and any data fetched from the network 120 may be accomplished using the semiconductor memory 106, possibly in concert with the hard disk drive 110. In some 627652.doc 12instances, the application program may be supplied to the user encoded on a CD-ROM or floppy disk and read via the corresponding drive 112 or 11, or alternatively may be read by the user from the network 120 via the modem device 116. Still further, the software can also be loaded into the computer system 100 from other computer readable media.

The term "computer readable medium" as used herein refers to any storage or transmission medium that participates in providing instructions and/or data to the computer system 100 for execution and/or processing. Examples of storage media include floppy disks, magnetic tape, CD-ROM, a hard disk drive, a ROM or integrated circuit, a magneto-optical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 101.

Examples of transmission media include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including email transmissions and information recorded on websites and the like.

The methods 200, 300 and 500 may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of Figs. 2, 3 and 5. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

Referring to Fig. 2, the method 200 of compressing image data representing an input image, begins at step 201, where the processor 105 allocates space in memory 106 for a scanline table. The scanline table is preferably configured as an array of 4-byte entries for storing an offset value representing the start of compressed data for each scan line of the image. The scanline table is stored at the start of a compressed data buffer, which is configured in memory 106 for storing the resulting compressed image data output stream, such that the layout of the resulting compressed image data output stream is a scanline table followed by the compressed image data. The scanline table is 627652.doc 13particularly useful for facilitating subsequent decompression of the image, as will be explained below.

In the following steps 203 to 229, of the method 200, the input image data is processed one scanline at a time. The offset for a resulting compressed scanline corresponding to an input scanline is stored at the start of the compressed data buffer. The corresponding scanline is then smoothed, by determining 32-bit pixel differences, modulo 2 2 between adjacent pixels of the input image data. The smoothed scan line is then searched for runs of zeroes and small differences in adjacent pixel values. As will be explained in detail below, the small differences are compared to predetermined small pixel difference values stored in a "small difference table" configured within the memory 106. Based on this comparison, a byte code is determined for the resulting compressed output stream.

The small difference table is preferably configured for 128 entries. Non-zero difference values are looked up in a pre-configured hash table, which contains indices into the 128-entry small difference table. The hash function is (69621 x A), modulo 225' and is chosen for maximum efficiency, where X represents a 32-bit pixel difference value. The difference values in the small difference table correspond to adjacent pixels in an input stream of pixel data, which change in value by a small amount.

The small difference table contains 128 of the most commonly occurring differences. The difference values of the small difference table are those which arise from a difference of 0, +1 or +2 in each of the RGB colour channels of the adjacent pixels and in addition to the differences which arise from having a difference of I, I or 2 simultaneously in all four channels. This generates 128 distinct difference values as the value zero is excluded. If the pixel format is RGBA, where A represents the alpha 627652.doc -14channel, on a little endian computer architecture, for example, the differences equate to a 32-bit integer X as follows: r g x 28+ b x 2 6 modulo 2 32 where r, g, b represent the red, green and blue colour difference components of a pixel and range over the values 0, and a x (1 28 216 224) modulo 232, where a represents the alpha component of a pixel difference and ranges over the values +2.

Table 1, below, shows 'Multiplier' and 'Modulus' hash function values, and corresponding count values entries in the small difference table) for four example difference value implementations.

Table 1 Multiplier Value Modulus (little/big endian) Count Difference Value 69621 225/325 128 5x5x5x1 16645254 231 124 5x5x5x1 16645254 164/ 80 3x3x3x3 16645254 329/ 134 3x3x3x5 The first row of Table I shows the 'Multiplier Value' and little endian 'Modulus' values for the hash function, f(X) (69621 x X) mod 225 described above. For this hash function and a corresponding count of one-hundred and twenty-eight one-hundred and twenty-eight entries in the small difference table), there are five corresponding difference values which arise from a difference of 0, +2 in each of the RGB colour channels of the adjacent pixels and in addition to the differences which arise from having a difference of 1, 1 or 2 simultaneously in all four channels. There is also one difference value for the alpha channel.

The second row of Table 1 shows the 'Multiplier Value' and little endian 'Modulus' values for a hash function, f(X) (16645254 x X) mod 231. For this hash function and a corresponding count of one-hundred and twenty-four one-hundred 627652.doc 15 and twenty-four entries in the small difference table), there are five corresponding difference values 0, for each of the RGB colour channels and one difference value for the alpha channel.

The third row of Table I shows the 'Multiplier Value' and little endian 'Modulus' values for a hash function, f(X) (16645254 x X) mod 164. For this hash function and a corresponding count of eighty eighty entries in the small difference table), there are three corresponding difference values 0, for each of the RGB colour channels and three difference values 0, for the alpha channel.

Similarly, as seen in the fourth row of Table I, for a count of one-hundred and thirty-four and a hash function, f(X) (16645254 x X) mod 329, there are three corresponding difference values 0, for each of the RGB colour channels and five difference values for the alpha channel..

If the difference between two or more adjacent input pixels is zero, then a byte representing a run of zero-valued pixel differences is stored in the compressed data buffer to be output in the compressed output stream.

If the difference between two adjacent input pixels is found in the small difference table, then the difference between the two adjacent input pixels is stored in the compressed data buffer as a byte code in the compressed output stream. The byte code corresponds to the difference value in the small difference table. For example, if the difference value selected from the small difference table is then the index corresponding to that difference value is stored in the compressed output stream. If the difference value is not found in the small difference table then the actual pixel difference value is stored, unchanged, in the compressed data buffer as a run of pixel literal values.

Accordingly, the compressed image data resulting from the method 200 is composed of a series of pixel runs each representing one or more pixel differences.

627652.doc 16- Further, the first pixel in a run of consecutive pixel literals is preceded by a byte code indicating the length and type of the run. Three types of byte codes for the runs are defined, where the byte value range of a byte code determines the byte code type as follows: Type 1: byte value, 0 to 127, indicates an index into the small difference table; (ii) Type 2: byte value, 128 to 191 indicates a run of (byte value 127) zero differences; (iii) Type 3: byte value, 192 to 255 indicates a run of (byte value 191) literal differences. The type 3 byte code is followed by literal pixel difference values which are 4 x (byte value- 191) bytes long.

Again referring to Fig. 2, if the processor 105 determines that there are more input scanlines left to be processed, at step 203, then the method 200 proceeds to step 205. Otherwise, the method 200 concludes. At step 205, the offset corresponding to a next scanline is added to the seanline table. At the next step 207, if the processor 105 determines that the method 200 has reached the end of a current scanline then the method 200 returns to step 203. Otherwise, the method 200 proceeds to step 209, where the processor 105 determines a difference value representing the difference between a current input pixel and the previous input pixel. When determining the pixel difference value for the first input pixel of a scanline, there is no previous input pixel, so the previous input pixel value is set to zero.

If the difference value determined by the processor 105 is equal to zero, at step 211, then the method 200 proceeds to step 221. Otherwise, the method 200 proceeds to step 213. At step 221, if the processor 105 determines that a previous byte code stored in the compressed data buffer, for the current input scanline, was a zero run byte type 2) having an associated count value of less than sixty four, then the method 200 proceeds to step 225, where the previous zero run byte code is incremented by one. The count value 627652.doc 17associated with a type 2 byte code indicates the number of pixels in the 32-bit zero value pixel run which is represented by the type 2 byte code. Otherwise, the method 200 proceeds to step 223 where a zero run byte code is stored in the compressed data buffer, by the processor 105, with a count of one indicating one 32 bit zero value pixel run. After step 223, the method 200 returns to step 207.

The method 200 continues at the next step 213, where if the processor 105 determines that the pixel difference value determined at step 209 is in the small pixel difference table, then the method 200 proceeds to step 227. Otherwise, the method 200 proceeds to step 215. At step 227, the processor 105 outputs a byte code corresponding to the index in the small differencc table of the calculated pixel difference value. The byte is output to the compressed data buffer and the method 200 returns to step 207.

If the processor 105 determines, at step 215, that the previous byte code stored in the compressed data buffer was a literal difference byte type 3) having an associated count value of less than the value sixty four, then the method 200 proceeds to step 217 where the literal run byte code is incremented by one and the method 200 proceeds to step 219. Otherwise the method 200 proceeds to step 229. The count value associated with a type 3 byte code indicates the number of four byte literal pixel values represented by the type 3 byte code. At step 229, the processor 105 stores a literal difference value byte code with a count of one, in the compressed data buffer and the method 200 proceeds to step 219. At step 219, the processor 105 outputs four bytes whose value is equal to the pixel difference determined at step 209. After step 219, the method 200 returns to step 207 and the remaining scanlines are processed for the image in the same manner.

If the volume of compressed data resulting from the method 200 is not sufficiently less than the volume of the original input data then the compressed data can be discarded. This decision can be made on the basis of a threshold value compression).

627652.doc 18 Fig. 6 shows an example of a compressed image data output stream 600 for an image which is W pixels wide by H scanlines high. The output stream 600 comprises a scanline table 601 which is configured to store H offsets. The table 601 is followed by H compressed scanlines 603, which comprise one or more byte codes depending on the number of pixels, W, and on the pixel runs of a corresponding scanline. For example, the first compressed scanline 605 may comprise six byte codes 607. The first byte code 609 may be a type 1 byte code having a value of three, representing the third entry in an associated small pixel difference table. Therefore, the difference value between zero and the first pixel of the first input scanline is equal to a corresponding value stored in the third entry of the small pixel difference table.

The third byte code 611 of the byte codes 607 may be a type 3 byte code having a value equal to 'l93'indicating that the byte code 611 is followed by two 32 bit literal pixel difference values. Therefore, the value of the 32 bit 4 byte) code 613 indicates the actual difference value between the second and the third pixels of the first input scanline.

The byte code 615 of the byte codes 607 may be a type 2 byte code having a value of '129'. Therefore, byte code 615 represents two zero pixel differences such that the difference between both the fifth and sixth, and the sixth and seventh pixels of the input pixel stream is zero, indicating a flat colour pixel run.

Referring to Fig. 3, the method 300 of decompressing image data, representing an image which was compressed according to the method 200, begins at step 301 where an initial compressed scanline is located using a scanline table contained in a compressed image data buffer containing the compressed image. The method 300 decompresses scanlines associated with a rectangular portion of the compressed image. The rectangular portion typically includes the intersection of graphical elements representing a part of an image and is determined by drawing a bounding box, as known to those in the relevant 627652.doc 19art, around the region representing the intersection of the graphical elements. However, such a rectangular portion can contain the entire image. The manner in which such a rectangular portion is selected in a retained mode rendering system, utilising the method 300, will be explained in detail below with reference to Figs. 4 and 9.

At the next step 302 of the method 300, the processor 105 sets a nominal current pixel value stored in memory 106 to zero. Then at the next step 303, the processor 105 examines a first compression byte code of the initial compressed scanline to determine which of the three types of pixel run type 1, type 2 or type 3 as described above), the compression byte code represents. If the processor 105 determines that the compression byte code is a type I byte code, at the next step 305, then the method 300 proceeds to step 309. Otherwise, the method 300 proceeds to step 307.

At step 309, the processor 105 converts the type 1 compression byte code to a pixel difference value by using the byte code as an index into the small difference table.

At the next step 310, the pixel difference value determined at step 309 is added to the current pixel value stored in memory 106. Then at the next step 312, the processor 105 outputs the current pixel value to an output frame buffer configured within memory 106 and the method 300 proceeds to step 317. Thus, the pixel value stored in the frame buffer at step 312 is an uncompressed representation of the type 1 compression byte code At step 307, if the processor 105 determines that the compressed byte code is a type 2 byte code, then the method 300 proceeds to step 314. Otherwise, the byte code is type 3 byte code and the method 300 proceeds to step 313. At step 313, any 4 byte codes following the type 3 compressed byte code are stored in memory 106 as literal pixel difference values. Then at the next step 316, the processor 105 adds each of the literal pixel differences stored in memory 106, at step 313, to the current pixel value and outputs the current pixel value to the output frame buffer. Following step 316, the method 300 proceeds to step 317.

627652.doc At step 314, the processor 105 writes one or more 32 bit pixel values to the output frame buffer. The value represented by each of the 32 bit pixel values is equal to the current pixel value. The number of pixels written to the ouput frame buffer is indicated by the type 2 byte code.

[f there are any more compressed bytes to be processed for the current scanline, at step 317, then the method 300 returns to step 303. Otherwise, the method 300 proceeds to next step 323. At step 323, if any more compressed scanlines are required to be processed then the method 300 returns to step 301. Otherwise, the method 300 concludes.

The image represented by the decompressed image data in the frame buffer can be subsequently rendered on the printer 115 or display 114.

A method 500 of compositing graphical elements will now be explained by way of example, with reference to Figs. 4, 5 and 9. The method 500 uses the compression method 200 and decompression method 300 and will be explained with reference to a retained mode rendering system as described above.

Referring to Fig. 4, when two (or more) graphical elements rectangular shaped object 401 and object 402) are composited together, and at least one of those graphical elements is an image object 401) compressed according to the method 200, any portion of the compressed image object 401 which is required to complete the rendering of the object 401 and 402, can be decompressed prior to a required compositing operation over in this example) being performed. The decompression is performed by determining a smallest rectangle 403 or bounding box (shown by phantom lines in Fig. 4), which includes the intersection region 405 of graphical objects 401 and 402. A person skilled in relevant art would appreciate that in the case of the region 405, such a bounding box 403 would substantially follow the boundary of the intersection region 405.

The smallest rectangle 403 can be determined, by the processor 105, by traversing a compositing tree representing the composition of the two graphical objects 627652.doc -21 401 and 402 and then selecting the intersection region 405 which is stored at the parent node of the two leaf nodes representing the graphical objects 401 and 402. For example, Fig. 9, shows an image 900 comprising the graphical objects 401 and 402, and three other overlapping graphical objects 901, 903 and 905. Fig. 9 also shows a compositing tree 910 corresponding to the image 900. For ease of explanation, the objects 401, 402, 901, 903 and 905 are also given the labels D, C, E, B and A, respectively, as indicated in Fig. 9.

The smallest rectangle 403 can be determined, by the processor 105, by traversing the compositing tree 910 corresponding to the image 900 and then selecting the region 405 which is represented by the node 907 of the compositing tree 910. A bounding box 403 is then determined, by the processor 105, for the region 405 stored at node 907. The manner in which such a region 405 is represented and stored at the node 907, in the methods 200, 300 and 500 described herein, will be explained in further detail below.

In the present example, in order to composite the objects 401 and 402 of the image 900, the scan lines (not shown) in object 401, which intersect the rectangle 403, are decompressed, according to the method 300, into a temporary buffer configured within memory 106. In addition, if object 402 is a compressed image then the scan lines in object 402, which intersect rectangle 403, are decompressed, according to the method 300, into another temporary buffer configured within memory 106. A required compositing operation is then performed using the data defined by the decompressed scanlines stored in the temporary buffers and the temporary image buffers are subsequently discarded. If the result of the compositing operation is an image which is required for a subsequent redraw operation, then the resulting image is also compressed into a compressed image buffer, according to the method 200, and the compressed image buffer is subsequently discarded after the redraw operation.

Accordingly, only the scanlines which are required to complete the rendering of the image 900 are decompressed into a temporary buffer. For example, if any of the 627652.doc 22 objects 401,402, or 903 are opaque (determined from an alpha channel associated with these objects 401,402 and 903) then there is no requirement to decompress any scanlines associated with any region containing the intersection of these opaque objects 401,402 and 903, and the objects 901 and 905, since any such region is hidden in the final image.

Decompression of such hidden regions would only reduce the processing advantages gained by the methods 200 and 300 described herein, since processing would be performed on regions not required to render the image 900.

The regions which are required to complete the rendering of an image and are thus required to be decompressed into a temporary buffer, can be selected by traversing a compositing tree the compositing tree 910) corresponding to the image and then using an alpha channel associated with such regions and associated child graphical elements to determine whether the region is visible in the final rendered image.

Referring to Fig 5, the method 500 of compositing the objects 401 and 402 begins at step 501, where the processor 105 analyses the objects 401 and 402 to determine the overlap (or intersection) region 405 and then draws a bounding box around the region 405 to determine the smallest rectangle 403. At the next step 503, if the processor 105 determines that the object 401 is a compressed image then the method 500 proceeds to step 504. Otherwise, the method 500 proceeds to step 505. At step 504, the processor 105 allocates a first image buffer within memory 106. Also at step 504, the scan lines in object 401 which intersect rectangle 403 are decompressed according to the method 300, and decompressed data corresponding to the scan lines are stored in the first image buffer.

As described above, the smallest rectangle 403 can be determined, by the processor 105, by traversing the compositing tree 910 corresponding to the image 900, selecting the intersection region 405, represented by the rectangle 403, stored at node 907 of the compositing tree 910, and then determining a bounding box for the selected region.

627652.doc 23 At the next step 505, if the processor 105 determines that object 402 is a compressed image, then the method 500 proceeds to step 506. Otherwise, the method 500 proceeds to step 507. At step 506, the processor 105 allocates a second image buffer within memory 106. Also at step 506, the scan lines in object 402, which intersect the rectangle 403, are decompressed according to the method 300 and are stored in the second image buffer.

The method 500 continues at the next step 507, where if the result of the composition of the objects 401 and 402 is an image, then the method 500 proceeds to step 508. Otherwise, the method 500 proceeds to step 509. At step 508, the processor 105 allocates a third image buffer, within memory 106, large enough to store the uncompressed image data corresponding to the rectangle 403. At step 509, the processor 105 performs the comrnpositing operation object 401 over object 402) using the decompressed image data stored in the first and second image buffers. The over operation uses alpha channels associated with both of the objects 401 and 402 to determine the resulting image data representing the composition. At the next step 511, the first and second image buffers are discarded. The method 500 concludes at the next step 513, where if the result of the compositing operation performed at step 509 is an image, then the image is compressed, according to the method 200, using the decompressed data contained in the third image buffer. The resulting compressed output stream can be written to a compressed image buffer configured within memory 106, and the third image buffer can be discarded. The remaining objects 901, 903 and 905 can also be composited according to the method 500, as described above for objects 401 and 402.

Although the method 500 was described above with reference to the over operation, the method 500 is independent of the compositing operation associated with a smallest rectangular region the rectangle 403) to be rendered, since the scanlines associated with the smallest rectangular region are decompressed prior to the compositing 627652.doc 24 operation being performed on the data associated with the smallest rectangular region 403.

When storing regions representing graphical objects at the nodes of a compositing tree, there is a trade-off between how closely region boundaries follow graphical object boundaries and the benefits obtained. If region boundaries follow object boundaries very closely, a lot of work is usually involved in creating the region boundaries and in performing intersections and differences of regions. However, if region boundaries are too approximate, the region boundaries may either include large areas that are outside an objects' boundaries, resulting in too much unnecessary compositing, or the region boundaries may fail to include large areas of an object.

Region boundaries can be limited to sequences of horizontal and vertical segments. The segment size can be chosen so that there is neither too much detail so that the region operations are overburdened, nor too much approximation to result in wasted compositing.

One method to improve the efficiency of region operations is to choose as many as is practical of the horizontal and vertical segments of substantially all region boundaries to be in phase. The horizontal and vertical segments are preferably chosen from the horizontal and vertical lines of the same grid. The grid need not be regularly spaced, nor have the same spacing horizontally and vertically.

Choosing the horizontal and vertical segments from the horizontal and vertical lines of the same grid improves the efficiency of region operations compositing operations) by seeking to keep all region boundary detail to the level of detail contained in the underlying grid. Without constraining the majority of region boundary segments to a grid, region operators such as difference and intersection tend to produce a lot more fine detail. For example, in Fig. 10, two circles 1001 and 1002 are shown with respective regions 1003 and 1004 that are not grid-aligned. These circles are overlapped yielding 627652.doc 25 difference regions 1005 and 1007, and intersection region 1006. In Fig. 11, the same circles 1001 and 1002 have regions 1013 and 1014 that are aligned to grid 1010. These circles are overlapped yielding difference regions 1015 and 1017 and intersection region 1016. It can be seen in the example of Figs. 10 and 11 that the grid-aligned regions yield less detailed results at the expense of slightly less efficient region coverage. Regions 1005, 1006 and 1007 togethcr contain a total of sixty segments, while regions 1015, 1016 and 1017 together contain only fifty-two.

Accordingly, a smallest rectangular region of an image to be decompressed in the methods described above, is preferably determined according to a grid the grid 1010), in the manner described above. For example, as seen in Fig. 11, the smallest rectangular region 1020 (shown shaded in Fig. 11) representing the intersection region 1016 is determined according to the grid 1010. Alternatively, in a further arrangement of the present invention, the region 1025 represented by the thick dark lines surrounding the intersection region 1016 of the circles 1001 and 1002 can be determined and subsequently decompressed, in accordance with the method 300, in order to perform a compositing operation on the circles 1001 and 1002. The region 1025 is aligned to the grid, although the region 1025 is not rectangularly shaped.

In one arrangement of the present invention, the data model used to represent regions the intersection region 405) corresponding to the nodes of a compositing tree the compositing tree 910) consists of a region description and a proxy. Each region of a composting tree has at least the following associated data: A Region Description: A low-level representation of the boundaries of the region. The region descriptions of all the regions are mutually exclusive (nonintersecting, non-overlapping); and (ii) A Proxy: Some means of caching the pixel data resulting from applying a compositing operation specified by the compositing expression at every pixel inside the 627652.doc 26region description. A proxy can be a 24-bit colour bit-map, or a run-length encoded description, for example. A proxy represents pixel data in a manner which makes the pixel data efficient to retrieve and use.

The aforementioned preferred method(s) comprise a particular control flow.

There are many other variants of the preferred method(s) which use different control flows without departing the spirit or scope of the invention. Furthermore one or more of the steps of the preferred method(s) may be performed in parallel rather sequential.

The method described above, provides a very fast and lossless compression method which can be integrated into a graphical composition process in a retained-mode renderer.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive. For example, a person skilled in the relevant art would appreciate that the methods 200, 300 and 500 can be applied to any number of graphical elements.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including" and not "consisting only of'. Variations of the word comprising, such as "comprise" and "comprises" have corresponding meanings.

627652.doc

Claims (14)

1. A method of compositing a plurality of graphical objects according to a predetermined compositing operation, at least one of said graphical objects being in the form of a compressed representation, said compressed representation comprising one or more coded pixel runs each representing one or more pixel difference values, each of said coded pixel runs having an associated byte code indicating a type for the associated coded pixel run, said method comprising the steps of: decompressing at least one coded pixel run of said compressed representation by: examining the byte code associated with said coded pixel run to determine the type of said coded pixel run; and determining pixel values for said coded pixel run, depending upon the type of said coded pixel run, using the pixel difference values represented by said coded pixel run; and compositing at least a portion of the pixel values determined for said compressed representation with pixel values for at least another one of said graphical objects, in accordance with said predetermined compositing operation, to determine one or more runs of composited pixel values representing the composited graphical objects.

2. The method according to claim 1, further comprising the steps of: for each run of composited pixel values, determining pixel difference values between each adjacent pixel of said run of composited pixel values; comparing the determined pixel difference values for said run of composited pixels to predetermined difference values using a predetermined hash function; 627652 -28- determining one or more coded pixel runs for said run of composited pixel values depending on said comparison, each of said coded pixel runs representing a difference value between at least two adjacent pixels of said run of composited pixel values; and storing said one or more coded pixel runs as a compressed representation of said run of composited pixel values.

3. The method according to claim 1, wherein a first type of coded pixel run represents an index into a small difference table.

4. The method according to claim 1, wherein a second type of coded pixel run represents a run of zero pixel difference values.

The method according to claim 1, wherein a third type of coded pixel run represents a run of literal pixel difference values.

6. The method according to claim 1, wherein the portion of the pixel values determined for said compressed representation falls within a region drawn around a region of intersection of at least two of said graphical objects.

7. The method according to claim 6, wherein said region is determined by: traversing a compositing tree representing the composition of said at least two graphical objects; and selecting the region of intersection stored at a parent node of two leaf nodes representing said at least two graphical objects. 627652 -29-

8. The method according to claim 6, wherein said region of intersection is bounded by sequences of horizontal and vertical segments.

9. The method according to claim 8, wherein said horizontal and vertical segments are selected from horizontal and vertical segments of a grid.

An apparatus for compositing a plurality of graphical objects according to a predetermined compositing operation, at least one of said graphical objects being in the form of a compressed representation, said compressed representation comprising one or more coded pixel runs each representing one or more pixel difference values, each of said coded pixel runs having an associated byte code indicating a type for the associated coded pixel run, said apparatus comprising: decompressing means for decompressing at least one coded pixel run of said compressed representation by: examining the byte code associated with said coded pixel run to determine the type of said coded pixel run; and determining pixel values for said coded pixel run, depending upon the type of said coded pixel run, using the pixel difference values represented by said coded pixel run; and compositing means for compositing at least a portion of the pixel values determined for said compressed representation with pixel values for at least another one of said graphical objects, in accordance with said predetermined compositing operation, to determine one or more runs of composited pixel values representing the composited graphical objects. 627652

11. A computer program product including a computer readable medium having recorded thereon a computer program for compositing a plurality of graphical objects according to a predetermined compositing operation, at least one of said graphical objects being in the form of a compressed representation, said compressed representation comprising one or more coded pixel runs each representing one or more pixel difference values, each of said coded pixel runs having an associated byte code indicating a type for the associated coded pixel run, said program comprising: code for decompressing at least one coded pixel run of said compressed representation by: examining the byte code associated with said coded pixel run to determine the type of said coded pixel run; and determining pixel values for said coded pixel run, depending upon the type of said coded pixel run, using the pixel difference values represented by said coded pixel run; and code for compositing at least a portion of the pixel values determined for said compressed representation with pixel values for at least another one of said graphical objects, in accordance with said predetermined compositing operation, to determine one or more runs of composited pixel values representing the composited graphical objects.

12. A method of compositing a plurality of graphical objects according to a predetermined compositing operation,, said method being substantially as herein before described with reference to any one of the embodiments as that embodiment is illustrated in the accompanying drawings.

13. An apparatus for compositing a plurality of graphical objects according to a predetermined compositing operation, said apparatus being substantially as herein before 627652 -31 described with reference to any one of the embodiments as that embodiment is illustrated in the accompanying drawings.

14. A computer program product including a computer readable medium having recorded thereon a computer program for compositing a plurality of graphical objects according to a predetermined compositing operation, said program being substantially as herein before described with reference to any one of the embodiments as that embodiment is illustrated in the accompanying drawings. DATED this Seventh Day of April 2005 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant SPRUSON&FERGUSON 627652