AU2005246935A1 - Optimal Rendering-based Memory Reclamation - Google Patents

Description

I

S&FRef: 738457

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address of Applicant Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3-chome, Ohta-ku, Tokyo, 146, Japan George Politis Spruson Ferguson St Martins Tower Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Optimal Rendering-based Memory Reclamation The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5845c -1- OPTIMAL RENDERING-BASED MEMORY RECLAMATION

U

a)

C)

Field of the Invention The present invention relates generally to rendering graphic objects, and particularly to reducing the amount of memory used by a page description in a rendering system.

Background Current rendering and printing systems use several methods to reduce the amount of memory used by display lists. Such methods include dividing a page into bands, maintaining a display list for each band of the page, and rendering a band to produce an image for that band when memory usage by the band exceeds a threshold.

Various rates of compression may be used to compress such a band image, thereby trading off memory used by the display list against visual quality. In the prior art little consideration is given as to how this so-called "fallback" rendering can be used in conjunction with other methods of reducing the amount of memory used by display lists, so as to maximise the quality of pages printed from the display lists.

Summary It is an object of the present invention to provide improved mechanisms of reclaiming memory within a rendering system, while optimising the visual quality of pages produced by the rendering system.

According to a first aspect of the present invention, there is provided a method of reducing the memory usage of a display list representing an output area to be rendered, said display list comprising graphic objects and raster image data, said method comprising the steps of: compressing said raster image data to a first compression level; 738457 -2- O rendering graphic objects lying within a region of said output area to image data; compressing said image data to a second compression level, wherein said second compression level is related to said first compression level; and replacing said graphic objects lying within said region with compressed image data, N wherein at least some of said raster image data lies outside said region of said output area.

OAccording to a second aspect of the present invention, there is provided a method of estimating an amount of memory usage reduction of a display list representing an output area to be rendered, said display list comprising graphic objects and raster image data, said method comprising the steps of: estimating a first amount of reduction of memory usage that would result from compression of said raster image data to a first compression level; estimating a second amount of reduction of memory usage that would result from rendering graphic objects lying within a region of said output area to raster image data, compressing the rendered raster image data to a second compression level, and replacing said graphical objects lying within said region with the compressed rendered raster image data, wherein said second compression level is related to said first compression level; and adding said first and second amounts of reduction of memory usage together to provide said amount of memory usage reduction of said display list, wherein at least some of said raster image data lies outside said region of said output area.

According to another aspect of the present invention, there is provided an apparatus for implementing any one of the aforementioned methods.

Other aspects of the invention are also disclosed.

738457 -3- Brief Description of the Drawings One or more embodiments of the present invention will now be described with Sreference to the following drawings, in which: C1 Fig. 1 shows a schematic block diagram of the architecture of a rendering system on which embodiments of the present invention may be practised; Fig. 2 shows a schematic flowdiagram of a method of reducing memory through (N optimal fallback rendering and lossy image compression; Fig. 3 shows a schematic flowdiagram of a step of the method of Fig. 2 where memory usage of graphics objects in sets of cells is determined; Figs. 4A, 4B, 4C and 4D together show a schematic flowdiagram of another step of the method of Fig. 2 where memory reduction that can be achieved by fallback rendering is estimated; and Fig. 5 shows example graphic objects in a display list representing a page to be rendered, and the division of the page into cells.

Detailed Description Fig. 1 shows a schematic block diagram of the architecture of a rendering system 100 on which the embodiments of the present invention may be practised. The rendering system 100 is used in a digital printer. The components of the rendering system 100 may be divided into a number of main groups, which include components dealing with creating a display list from an input job 101, those components dealing with rendering the display list into pixels 120 destined for a printer engine (not illustrated), and a memory manager 119.

The job 101 may consist of one or more pages, and is sent to the rendering system 100 of the printer in a form utilizing a page description language (PDL). Typical PDLs include Adobe® PostScript®, Hewlett Packard® PCL and Adobe® PDF. The 738457 rendering system 100 includes a PDL interpreter 102 which translates the page O description of the input job 101 into high-level graphics primitives.

A display list module 104 creates and handles display lists, each of which N represents a page of output, from the high-level graphics primitives provided by the PDL interpreter 102. Before passing the high-level graphics primitives to the display list C€3 module 104, the high-level primitives may be filtered through a display list input N optimizer 103, which comprises a set of configurable filters, for idiom recognition and optimization.

The completed display list produced by the display list module 104 may be filtered through an optimizer 105 to ensure that the page represented in the display list can be rendered in real time, before being preprocessed for rendering and spooled onto a spool queue 116. The memory manager 119 invokes a display list memory optimizer 112 when the amount of memory available in the rendering system 100 is low. Optimizer 112 changes the form of all or part of the display list so as to reduce the memory usage of the display list. The display list stores graphic objects in high-level form, utilizing data stores for storing data of various types.

The rendering system 100 further includes an image store 107, property store 108, path store 110 and text cache 109 for storing image data, fill and stroke specifications, path outlines, and glyphs respectively. Any item of data stored within data stores 107 to 110 is potentially shared within and between pages, and remains in existence as long as that item of data is referenced by the display list, or any job in the spool queue 116. Data stores 107 to 110 register callback functions with the memory manager 119 for reclaiming and minimizing memory usage in accordance with urgency.

The image store 107 receives images from the PDL input job 101, compresses these images using an image compressor 111, and then stores the compressed images. The 738457 o image compressor 111 provides variable degrees, or levels, of image compression, with increased levels having corresponding quality reduction. The image compressor 111 achieves this by storing an image's data in a series of layers, each layer being responsible N for a certain level of visual quality, as is known in (for example) the JPEG2000 image compression standard. Thus, compression can be increased efficiently simply by \discarding the layers corresponding to the least visually significant information, without N' having to decompress and recompress stored image data. Images can thus be stored initially at maximum (lossless) quality and gradually degraded only to the minimum extent necessary for the rendering system 100 to have enough memory to function.

To help reduce memory usage, only one instance of an image is stored in the image store 107, with that instance being shared where necessary across multiple display lists.

The property store 108 stores fill specifications such as flat or blended colour, as well as stroking specifications.

The path store 110 stores path outlines, that include line and spline segments, in one of a number of formats. Formats include floating point or 32-bit fixed point, or smaller sized fixed-point representations, depending upon which format is the most compact for a given outline without sacrificing quality.

The text cache 109 stores glyphs (characters), represented as bitmaps and/or outlines. The text cache 109 needs to interface with a font engine 106 to obtain the glyphs. The text cache 109 has its own instance of a path store for storing glyphs in spline format. Naturally, only one instance of each glyph is stored for use any number of times on one or more pages.

The memory manager 119 is responsible for memory allocation and ensuring that the rendering system 100 has enough memory to perform its tasks. The memory 738457 -6manager 119 uses compaction and reclamation in the event of low memory to try and N ensure that other components of the rendering system 100 can continue to allocate memory. When the available memory starts to run low, the memory manager 119 invokes a series of steps, aimed at sacrificing speed and quality in exchange for reduced memory usage. The rendering system 100 is designed to degrade the quality of output pages and speed gracefully, reducing performance and output quality only as much as is needed to NI complete its task of rendering page descriptions.

Rendering may be performed in two situations namely, fallback rendering for dealing with running out of memory, and job rendering for generating pixel output 120 to the printer engine. The rendering system 100 includes two sub-systems 118 and 115-117, with the respective sub-systems 118 and 115-117 being dedicated to respective rendering situations. A render preprocessor 114 serves as a common interface for both sub-systems 118 and 115-117.

The render preprocessor 114 converts the display list into edge, level and fill format as required by both rendering sub-systems 118 and 115-117. The render preprocessor 114 also invokes edge vectorizer 113 to convert splines into line segments.

During fallback rendering, render preprocessor 114 invokes fallback renderer 118 to convert a selected region of the page currently being constructed into pixel data, which is suitably compressed using the image compressor 111. The new image thus created is stored in the aforementioned image store 107 and a reference to that new image is added to the display list.

During job rendering, render preprocessor 114 invokes render simplifier 115 to scan convert and analyse the display list. The render simplifier 115 produces a simplified spool format for the page which can be rendered in real time by a page renderer 117. In this process, new fills and images may be created. These new fills and images are stored 738457 -7o in the aforementioned data stores 107 to 110. The simplified job is then queued in spool Squeue 116, to be rendered into pixel data 120 by page renderer 117. The pixel data are ultimately used by a printer engine to produce a physical output.

(Within the rendering system 100, the role of the memory manager 119 is to satisfy memory allocation requests by other components of the rendering system 100.

\Furthermore, in a limited memory system, the memory manager's role becomes crucial in N maintaining the availability of memory regardless of the size of the display list that is being constructed. In order to achieve this goal, the memory manager 119 directs the rendering system 100 to trade off speed and quality in exchange for memory.

The strategy employed in the preferred embodiment is to seek to keep memory usage within the rendering system 100 below a high threshold. Whenever an allocation of memory would bring memory usage above the high threshold, or if an allocation of memory cannot be fulfilled immediately, the memory manager 119 attempts to reclaim memory from the various parts of the rendering system 100, requesting such parts to sacrifice some speed and/or quality in exchange for memory. In order to avoid frequent cycles of reclamation, the preferred embodiment also employs a low threshold. When the memory usage high threshold has been reached, the memory manager 119 reclaims enough memory until memory usage is below the low threshold, or until no more memory can be reclaimed. It is important that the high threshold is set so as to leave enough memory to permit the memory reclamation techniques to be carried out without the memory reclamation itself failing due to lack of memory.

The memory manager 119 begins by reclaiming memory that will have least impact on the performance of the rendering system 100 and quality of the pixel output 120, and continues as necessary, reclaiming memory in ways that will have increasingly undesirable consequences. It should be noted that a useful rendering system 100 should 738457 -8have enough memory to store enough spooled pages of typical size to keep the printer engine running smoothly without having to resort to memory reclamation. However, when a more complex set of input jobs 101 arrives, the system 100 needs to be able to Shandle such complex jobs 101 effectively, even if speed or quality is sacrificed to some degree.

The memory manager 119 seeks to reclaim memory by: N requesting that the text cache 109 flush unused characters; requesting that the image store 107 reduce image quality of stored images up to the point where there is a negligible or low impact on visual quality when those stored images are rendered; requesting that the image store 107 crop image data where the associated images are subject to clipping; waiting for one or more spooled pages in the spool queue 116 to be printed by the page renderer 117, thereby releasing the memory occupied by those pages; and invoking the display list memory optimiser 112 to perform more costly fallback rendering and visually lossy image compression.

Considering each of the strategies employed by the memory manager 119 in turn, the text cache 109 keeps a reference count of every glyph description (such as a bitmap) used in any spooled display list or in the display list currently being constructed.

When a character is no longer used, i.e. the reference count drops to zero, the text cache 109 does not normally deallocate the memory used by the glyph in case the same glyph is needed again in the near future. However, if there is a shortage of available memory, currently unused glyphs are deallocated, and are only recreated when and if required at a later stage.

738457 -9- O The image store 107, using the image compressor 111, is able to efficiently discard layers of image quality, thereby deallocating some of the memory used by the images stored in the image store 107. A first stage of image quality reduction affects image data in display lists in the spool queue 116 as well as in the display list currently being constructed, and stops short of significant visual degradation to the stored images.

While increasing compression to reduce memory usage is known in the prior art, the N rendering system .100 allows increasing compression of images referenced by spooled Sdisplay lists in addition to increasing compression of images referenced by the display list currently being constructed. The compression is intelligently handled so as to amortize quality loss over several pages rather than to egregiously penalize one page.

When the memory manager 119 requests that the image store 107 reduce the image quality of stored images, the image store 107 examines the image compression level associated with each display list still in the system 100, namely, the display list currently under construction and all the spooled display lists whose printing has not yet commenced. In order to degrade pages roughly uniformly and to not degrade one page excessively, the display list referencing images having the lowest image compression level is selected. If more than one display list reference images at the lowest image compression level, any one or more of these display lists are selected in accordance with the amount of memory reduction requested by the memory manager 119. The image compression level associated with each of the selected display lists is increased by one level and all images referenced by the selected display lists are further compressed to the new compression level. In the prefered embodiment, the compression level of an image is increased by discarding the least visually significant layer of data available for that image.

If the memory manager's request cannot be fulfilled during this stage of memory reclamation, the image store 107 performs what memory reduction it can without severe 738457 visual degradation. The memory manager 119 then moves to a different strategy to N reclaim the remaining memory required.

The image store 107 is also able to discard unused image data in cases where an N image is clipped to a boundary. Note however that if the image is shared between multiple uses on a single page or on multiple pages, cropping may not always be possible o") \aas different instances of image use may in principle have different clipping applied.

Up to this point of memory reclamation severe losses in speed or quality have not been required. If enough memory has been regained at any point so far, the reclamation process can terminate. However, if sufficient memory for successful running l0 of the rendering system 100 has not been attained, the reclamation process must continue with more costly strategies.

Since visual quality is normally more important than output speed, the preferred embodiment, as in various prior art, will wait for one or more display lists in spool queue 116 to print before performing any technique that degrades output quality. This will deallocate memory associated with the spooled display lists, hopefully releasing enough memory for processing of the current page to continue.

If, however, a particular page is so complex that all the above strategies fail to reclaim enough memory, the memory manager 119 will invoke lossy compression techniques. These involve identifying, rendering and compressing regions of the output area that will release the most memory by replacing the portions of the display list under construction that specify the graphics of those areas, with compressed pixel data.

Furthermore, image data in other regions of the page can also be lossily compressed.

While there are prior art techniques for fallback rendering replacing portions of a display list with equivalent rendered pixel data), the display list memory optimizer 112 738457 -11o achieves a maximum amount of reclaimed memory while minimizing the amount of Svisual degradation in a manner described in detail below.

When creating a display list for a page in a limited memory environment, prior NI art generally creates separate display lists for each of a number of bands. To deal with low S 5 memory situations, prior art renders and compresses a band when its memory usage e¢3 increases beyond a threshold. Various rates of compression can be used that trade off NI memory use with visual quality, according to how much memory needs to be reclaimed.

This technique is commonly known as fallback rendering.

When applying a certain level of compression to image data produced by fallback rendering, prior art does not consider also applying the same level of compression to other image data in the system. Furthermore, prior art does not consider how to balance memory obtained by fallback rendering with memory obtained by compressing images so as to maximize the overall quality. Such a balance is beneficial in the case where there are images in regions of the page that are not subject to fallback rendering. Suppose a certain level of compression of fallback rendered images was required to release a required amount of memory. If a lesser level of compression with corresponding greater degree of quality were applied, not only to fallback rendered images but to all other images, then the same amount of memory may be reclaimed, but at greater overall image quality.

The method disclosed herein for optimal memory reclamation is based on estimating how much memory usage can be reduced by increasing image compression, estimating how much memory usage can be reduced by fallback rendering at least some region of the output area, determining if total estimated reduction is insufficient, and if so, determine a new total estimate based on a further increased level of compression of the image data and of the rendered display list or part thereof.

738457 -12- Note that fallback rendering is useful when at least some portion of the display list is so complex that if rendered and compressed, the resulting compressed pixel data occupies less memory than the data structures comprising the display list portion. It must Sbe observed that as the level of compression is increased, it may be possible to reclaim more memory. This is not only as a result of increased compression of the resulting pixel e¢3 data, but more significantly, by increasing the size and number of portions of the display list where fallback rendering is effective in reducing memory usage.

Fig. 2 shows a schematic flow diagram of a method 600 of optimal memory reclamation. The method 600 starts in step 601 where the display list memory optimiser 112 analyses the spatial distribution of memory usage of the display list currently being constructed. Step 601 is described below in detail with reference to Fig. 3.

The method 600 then proceeds to step 602 where the image store 107 estimates the amount of memory reduction that can be achieved if images were compressed to a current compression level. Since all images should already be compressed at the current compression level or greater the first time step 602 is performed, the estimated memory reduction should be zero. On subsequent executions of step 602, the images that are not yet compressed to the current compression level are identified, and the memory occupied by the layers that would be discarded to compress those images to the current compression level is computed to provide the estimate of the amount of memory reduction achievable.

The method 600 then proceeds to step 603 where an estimate is made by the display list memory optimiser 112 of the memory reduction that can be achieved through fallback rendering at the current level of compression. Step 603 is described below in detail with reference to Figs. 4A, 4B, 4C and 4D.

738457 -13o In a variation, step 602 and 603 do not use the same level of compression to form their respective estimates, but use different, yet related, levels of compression. For example, the level of compression of the images could be constrained to be one degree higher than the level of compression of fallback rendered regions. Other such relationships between the two levels of compression are possible.

At step 604, the total estimate of memory reduction, which is the sum of the C1 estimates made in steps 602 and 603, is checked by the display list memory optimiser 112 to see if the total estimate satisfies the required memory reduction requested by the memory manager 119. The first time step 604 is performed the total estimate of memory reduction equals the memory reduction estimated at step 603. In the preferred embodiment, the memory manager 119 seeks to reduce memory usage down to the low memory usage threshold. Additionally, if a memory allocation fails, the memory manager 119 tries to recover enough memory to enable the allocation to be made.

If it is determined in step 604 that the total estimate is not adequate, the method 600 proceeds to step 605 where the image store 107 determines whether the level of compression can be increased. If it is determined that the maximum compression level has already been reached, then the method 600 again proceeds to step 607.

If, however, it was determined at step 605 that the level of compression has not yet reached the maximum possible level, then the method 600 proceeds to step 606 where the level of compression is increased by image compressor 111. The method 600 then returns to step 602. The second and subsequent time step 602 is performed, the image store 107 will report some degree of memory reduction based on the number and size of the images stored. If there are no images stored, then again a zero value would be returned. At step 603, a new estimate is made of the savings achievable by fallback 738457 -14rendering based on the increased level of compression. The method 600 then continues with step 604 as previously described.

If it is determined in step 604 that the total estimate is adequate, the method 600 proceeds to step 607 where images in the image store 107 are compressed to the current level of compression. In the preferred embodiment, this is achieved simply by discarding the least visually significant image data available up to the current level of compression.

The method 600 then proceeds to step 608 where fallback rendering is carried out by rendering one or more regions of the output area, storing those regions as images compressed at the current compression level, and freeing memory occupied by the corresponding objects in the display list currently being constructed. Also in step 608 the rendered images are added as new objects referenced by the display list currently being constructed. The method 600 ends after step 608.

In a further variation, step 603 does not consider the effect of fully rendering selected regions to pixels. Instead, step 603 estimates the effect of converting selected graphic objects to a compact lower level representation, not being compressed raster pixels. In this variation, step 608 performs this conversion rather than rendering the selected regions to pixels.

Having described the method 600 of optimal memory reclamation, the manner in which the display list memory optimiser 112 analyses the spatial distribution of memory usage of the display list currently being constructed as performed in step 601 is now described in more detail with reference to Fig. 3 where a schematic flow diagram of step 601 is shown. As described above, display list objects are stored as high-level graphics representations rather than as rasterized pixels. This is because high-level graphics representations are typically more compact than rasterized pixels. However, for a very complex portion of a display list, it is possible that a rasterized pixel representation is 738457 O more compact than the high-level graphics representations. The goal of the display list memory optimiser 112 when performing step 601 is to find regions within the page described by the display list under construction where it is the case that the rasterized pixel representation is more compact than the high-level graphics representations, thereby allowing memory to be reclaimed later when step 608 is performed to render those regions and to replace the corresponding objects in the display list with images that N occupy less memory.

SRasterized pixels are treated as images, which are stored in the image store 107 and are subject to compression by the image compressor 111. As stated above, the image store 107 seeks to maintain consistent image quality. Therefore, at each stage in the rendering process there is a level of compression associated with the display list under construction, with that level of compression controlling the extent to which images included in the display list are compressed. There may also be images in spooled display lists with the same level of compression. From the amount of compressed image data of these images at the same level of compression as the display list under construction, a corresponding average compression factor may be estimated, defined as a ratio of compressed image size to uncompressed image size. The estimated size of a compressed region of rasterized pixels may be determined from this compression factor.

A region on a page is classified as "memory dense" if the memory usage of the high-level display list objects lying within the region is more than the estimated memory usage of rasterized pixels for the region, compressed at a given level of compression on which the average compression factor is based. It may be that a region is initially classified as not memory dense, but later, as the given level of compression increases, that same region may be classified as memory dense.

738457 -16o Step 601 starts in sub-step 701 where the output area of the page to be rendered is divided into regions, each of which is potentially renderable. In the preferred embodiment, the output area is divided into a rectangular grid of cells, and neighbouring cells are grouped to form regions.

S 5 Prior art methods typically divide the output area into bands, which are relatively coarse regions, spanning the entire width of the output area. Since the aim of fallback Srendering is to reclaim as much memory as possible, the total memory saved by rendering a band is typically less than that saved by rendering only one or more sub-regions of the band. For example, one part of a band may be very complex and yield a memory saving if rendered, while another part of the same band may be fairly simple and increase memory usage if converted to pixels. Rendering the whole band may provide a benefit, but rendering only the complex part of the band would provide a greater benefit. Using smaller cells allows finer control and a tighter bound around objects that are worth rendering. However, smaller cells increase the. execution time of the method 600.

Accordingly, cell size may be adjusted according to the desired trade-off between speed and accuracy. In the preferred embodiment, cells of size 256 by 256 pixels are used.

This equates to about 1 centimetre by 1 centimetre of output page area at 600 dpi.

An aspect of the present disclosure is that regions are not considered individually, or even in groups of two, four or a predetermined number of neighbouring cells. Regions are considered as combinations (sets) of neighbouring cells spanned by objects occurring in the display list. This allows step 601 to consider only areas which contain actual display list data. As shall be seen, it also allows a display list object and any display list object which lies within the same region or subset thereof to be considered together. In what follows, "region" and "set (of cells)" are used interchangeably.

738457 -17- O Let memory density be defined as being the memory usage of objects contained within a set of cells divided by the number of cells in the set. Memory usage of a display list object is the memory that could be freed if the display list object were rendered. So, Cfor example, if there were two display list objects on a page, with both objects being blue, and one of them was rendered, the memory occupied by the blue fill in the property store 108 is not memory that could be freed because it is also referenced by the other object.

N A region will be selected for rendering if its memory density is greater than a Sthreshold. This threshold is the estimated memory required to store the compressed rasterized pixels for an area of the size of a cell at the current level of compression. On the other hand, if the multiple uses of an object were explicitly known, a region may be constructed that comprises all uses of that object. Thus, for example, for an image that is used twice on the same page, a region containing all the cells that cover both instances of the image's use can be constructed.

After the output area is divided into a number of cells in sub-step 701, step 601 proceeds to sub-step 702 where the display list memory optimizer 112 initialises a data structure A. Data structure A is an associative array that maps unique sets of cells to memory usage of display list objects which span the combination of cells in each set. In the preferred embodiment, an object is deemed to span a cell if there is a non-empty area of intersection between the object and the cell. In other embodiments the intersection between the bounding box of the object and the cell may be considered. Sub-step 702 also allocates memory for storing the memory density of sets, a quantity which will be calculated at a later stage.

Next, at sub-step 703, the display list memory optimizer 112 gets the first object in the display list. At sub-step 704, the display list memory optimizer 112 determines the memory usage m of the current display list object, and, at step 705, determines the 738457 -18o complete set of cells S spanned by the current object (S is termed the spanning set of cells Sfor the object). At sub-step 706 the display list memory optimizer 112 determines whether the current set S is already a key of the data structure A. If it is determined that the current r set S is not already a key of the data structure A, the step 601 proceeds to sub-step 707 S 5 where the display list memory optimizer 112 creates an entry in the data structure A with C€3 the current set S as a key. The memory usage m of the current set S is initialized to 0, and Sthe memory density of set S is initialized to unknown.

If it is determined in sub-step 706 that the current set S is already a key of the data structure A, or following sub-step 707, processing proceeds to sub-step 708 where the memory usage m of the current object is added to the running total of the memory usage of display list objects that span the same set of cells S.

At sub-step 709 the display list memory optimizer 112 then determines whether the display list contains more objects to be incorporated into the data structure A. If it is determined that there are more objects in the display list, the step 601 proceeds to substep 710 where the next display list object is obtained before step 601 returns to sub-step 704 from where that object is incorporated into the data structure A.

If at sub-step 709 it is determined that all objects of the display list have been incorporated into the data structure A, the step 601 proceeds to sub-step 711 where the display list memory optimizer 112 initializes the estimated memory savings to 0 and a list of cells to be rendered to empty. Step 601 then ends.

At the completion of step 601 there is an associative array A of uniquely occurring regions on the page with associated memory usage for each region.

Consider an example display list comprising objects 920 to 928 to be rendered into an output area divided into cells 901 to 909, as illustrated in Fig. 5. The memory usage of each display list object 920 to 928 for this example shall be taken as follows: 738457 -19- Display List Object Memory Usage 920 1 921 922 923 924 925 926 1 927 1 928 1 Table 1 Table 2 shows the data structure A produced by step 601 described with reference to Fig. 3. Each object 920 to 928 contributes to Table 2 via a set of cells S within which the object lies. For example, object 925 spans cells 902 and 905, whereas object 926 spans cells 905, 906, 908 anid 909. The order of each set of cells S is defined as the number of cells in the set S. The memory usage shown in Table 2 for any set S is the total memory usage of objects whose spanning set of cells is exactly S. Accordingly, the set {901, 902, 904, 905} having an order of 4 has a memory usage of 5 from its sole spanning object 921 (see Table However, as both objects 926 and 927 span cells 905, lO 906, 908 and 909, and because objects 926 and 927 each have a memory usage of 1, the memory usage of region {905, 906, 908, 909} is 2. Although display list object 922 lies totally within display list object 921, memory usage of the set of cells {901, 902, 904, 905} only includes the memory usage of object 921, but not that of object 922 at this stage of the method 600, since object 922 only spans cells 901, 902 and 904 but not cell 738457 905. Therefore, the memory usage of object 922 contributes to entry {901, 902, 904} rather than to entry {901, 902, 904, 905} in the data structure A.

Regions Memory Memory Order Regions Memory Usage Memory Density 1 {902} 30 unknown {904} 5 unknown 2 {902,905} 5 unknown 3 {901,902, 904} 50 unknown {905,906, 908} 1 unknown 4 {901, 902, 904, 905} 5 unknown {905,906, 908,909} 2 unknown {901, 902, 903, 904, 905, 906, 907, 908, 909} unknown Memory 0 Saving Cells To Empty Render Table 2 In an alternative embodiment of step 601, a method to determine the memory usage of sets of cells S is to consider all possible neighbouring sets of cells. The memory usage of all sets S is set to zero and data structure A is initialised to empty. Then, proceeding from sets S of the lowest order, increasing the order after all sets S of a given order have been considered, until all sets S have been considered, the following is performed. When a set S is considered, the memory usage of each display list object that 738457 -21 lies wholly within the set S is added to the memory usage associated with the set S and the N wholly contained display list object is removed from further consideration. Sets S with Snon-zero memory usage are added to data structure A.

r Referring again to the method 600 in Fig. 2, once suitable initializations have been performed in step 601, the display list memory optimiser 112, at step 603, makes an estimate of the memory reduction that can be achieved by fallback rendering at the N current level of compression. Step 603 is now described in detail with reference to Figs.

4A, 4B, 4C and 4D.

In this stage of the method 600, the memory density of regions recorded in a previous stage is used. It is the object of step 603 shown in Figs. 4A, 4B, 4C and 4D to calculate the memory density of all regions and determine which, if any, of the regions should be rendered. Step 603 first decides whether rendering a region with the lowest number of cells (lowest order) will yield a memory saving. Referring to the example data structure A in Table 2, this comprises two regions, {902} and {904}, each with one cell.

Step 603 next decides whether rendering regions with two cells will yield a memory saving (such as region {902, 905} in the example), and so on. However, this algorithm does not investigate whether rendering a region such as {907, 908} will yield a memory saving, because from step 601, no display list object was found that spans this region and hence no entry appears in data structure A for this region. It is the object of step 603 to determine if regions spanned by each set of cells S in A are worth rendering.

Some of the principles utilized by the steps described with reference to Figs. 4A, 4B, 4C and 4D shall be illustrated with reference to the example from Fig. 5. For the purpose of the illustration, assume that a region is worth rendering if its memory density is greater than a threshold t 10. The determination of the value of t shall be explained shortly.

738457 -22o Fig. 5 shows two display list objects 924 and 925, with object 924 lying wholly Swithin cell 902, and object 925 lying wholly within the region comprising cells 902 and 905. Total memory usage in the region defined by cells 902 and 905 is 35, made up from from object 924 and 5 from object 925 (Table The total memory usage of divided by 2 cells is 17.5, which is higher than the memory density threshold t 10. At e¢3 first sight, it might be worth rendering both cells 902 and 905, giving an estimated memory saving of 15 (35 20 15). However, if region {902} comprising only one cell is considered first, the region comprising cell 902 would have a memory density of since only object 924 can be freed after rendering cell 902. The memory density of 30 is greater than the threshold t=10, hence cell 902 would be rendered first.

Now object 925 spans cells 902 and 905, but since cell 902 would be rendered, the memory usage of object 925 should be used in the decision for rendering cell 905 because it could be freed if cell 905 were also rendered. However, the memory usage of is less than threshold t=10, so cell 905 would not be rendered. Accordingly, considering lower order sets S first gives a larger memory saving of 20 (30 10 20) compared with in the particular example if both cells 902 and 905 were rendered. It is shown by this example that the sequence in which regions are processed is important and also that choosing to render one region affects the choice to render another.

To prevent one region that is very dense from spuriously causing an adjacent cell to be considered as worth rendering, as demonstrated in the example above, step 603 processes regions from lowest order to highest order. Step 603 starts off by considering regions of order 1, that is, consisting of one cell, then regions of order 2, that is, consisting of two cells, and so on. At each stage, step 603 calculates the total memory usage of each region, including the memory usage of any region that is a subset of the region under consideration. Step 603 decides to render cells contained in sets whose 738457 -23memory density exceeds the current threshold t. After each stage, cells that have been identified for rendering are removed from further consideration. In the above example, cell 902 is removed from the region comprising cells 902 and 905 once cell 902 has been N identified for rendering. To facilitate considering regions in increasing order, data S 5 structure A preferably stores sets of cells sorted by order.

C€3 In the process of removing rendered regions, new regions of an order that has N already been processed may be created, or the memory density of regions considered before may change. Therefore, some regions that have been examined before require reconsideration. To facilitate this, another data structure B is preferably used to record regions which need reconsideration. For reasons similar to those presented for data structure A, data structure B also stores sets of cells S sorted by order. Regions stored in data structure B need to be considered before proceeding to the next order of regions in data structure A. Regions in data structure B which are of the lowest order are processed in much the same way as regions in data structure A, namely, by calculating their memory density and determining whether to render such regions, or not.

If cells from data structure B are chosen for rendering, these rendered cells also need to be removed from data structure B. Again, in the process of removing regions, more regions may be added to data structure B. This process is repeated until data structure B is empty, at which point the method 600 then moves onto the next order of regions in data structure A. This process of removing rendered regions and how data structure B is used will become clearer with detailed explanation of step 603 and a few examples below.

Turning to Figs. 4A, 4B, 4C and 4D, and starting at Fig. 4A the average compression factor f defined as the ratio of compressed image size to uncompressed image size at the current level of compression and mentioned above is computed in sub- 738457 -24step 801. In sub-step 802, the threshold t is calculated from the average compression Nfactorfas follows: t=fwivhp (1) where w and h are the width and height of each cell in pixels, andp is the number of bytes M€ 5 for storing one uncompressed pixel (for example 3 for RGB data). The threshold t is thus O the estimated memory required to store rasterized pixels compressed at the current t compression level for an area of the size of a cell. In sub-step 803, it is determined N whether there is any accumulated memory saving. If there is some accumulated memory saving, it means that step 603 has been executed before and is being executed again with a lower threshold t, as a result of a higher level of compression, to try to increase memory saving. In the case where a lower compression factor is provided, the estimated memory saving from the previous execution of step 603 is updated to reflect this new lower compression factor. Therefore, the estimated memory saving needs to be increased by n(t' where t'was the previous threshold, t is the current threshold and n is the number of cells marked for rendering from the previous execution of step 603.

Data structure B is next initialized to be empty in sub-step 804. As explained above, sets of cells in data structure A are considered for rendering from lowest order to highest order. A counter K is used to keep track of the current order of sets of cells being considered. Accordingly, in sub-step 805 counter K is initialized to 1 as this is the minimum number of cells that can occur in a set S in data structure A. Sub-step 832 determines whether there are any sets of cells S in data structure A that have an order greater than, or equal to, counter K. If cells S remain having an order greater than, or equal to, counter K, step 603 continues to sub-step 833 (Fig. 4B). If there are no such sets S, step 603 continues to sub-step 847 where the estimated memory saving is reported back to the caller, which is the memory manager 119.

738457 Referring now to Fig. 4B, sub-step 833 determines whether there are any sets of cells S of order K in data structure A. If there are no such sets S, then step 603 continues

U

Sto sub-step 818 (Fig. 4A). If there are some sets of cells S of order K in the data structure N A then step 603 continues to sub-step 806 from where sub-steps 806 to 816 calculate the memory density for sets of cells S having an order equal to the current value of counter K.

In particular, sub-step 806 obtains one set of cells S in data structure A of order K. Sub- N step 807 determines whether the memory density of the region comprising the set S obtained in sub-step 806 has already been calculated and is still valid. If the memory density of that set S is not valid, then the memory density of that set S must be determined, in which case step 603continues to sub-step 808 where a memory usage total M is initialized to be the memory usage recorded for set S, which is sum of the memory usage of display list objects whose spanning set is exactly S.

Step 603 next needs to determine the memory usage of all display list objects that lie within a region comprising a subset of the set S under consideration, excluding those that lie partially within that region, by summing the memory usage of display list objects that span one or more cells in that region. For a display list object to span fewer than all cells in set S, the set of cells spanned by the object must have order less than that of the set S under consideration. Therefore, in sub-step 809, the display list memory optimiser 112 checks if there is any set of cells in data structure A which has order less than counter K, and has not already been checked to determine if that set is a subset of the set S obtained in sub-step 806. If there are no more such sets to be checked, all memory usage of display list objects that lie within the region spanned by set S has been summed up, and step 603 moves to sub-step 810.

Alternatively, if there are any sets in data structure A which have order less than counter K that have not yet been checked, such sets are to be checked in turn.

738457 -26- Accordingly, in sub-step 814 a next unconsidered set T is obtained, and in sub-step 815 it is determined if that set Tis a subset of set S. If it is determined that set Tis a subset of set S, then the memory usage of this subset T is accumulated in memory usage total M in Ssub-step 816 before step 603 returns to sub-step 809. If it is determined in sub-step 815 that set T is not a subset of set S, then step 603 returns directly to sub-step 809 to check C€3 the next set until all sets have been checked.

C If it is determined in sub-step 809 that there are no sets in data structure A which have an order less than counter K, then the memory density of the region under consideration is calculated in sub-step 810, which is the total memory usage of that region divided by the number of cells comprised by that region, and the memory density is recorded in data structure A, replacing its previous invalid value. By knowing a region's memory density, it can be decided whether a region is worth rendering. This is done by checking if its memory density is above threshold t inm sub-step 811. Sub-step 811 also follows sub-step 807 if it is determined in that sub-step that the memory density of the set S under consideration is a valid value.

If the memory density of the region under consideration is above the threshold t, the cells comprised by that region are marked for rendering in sub-step 812 and the estimated memory saving obtained by rendering that region is accumulated in sub-step 813. The estimated memory saving is calculated by taking the memory usage total M and subtracting the threshold t multiplied by the number of cells in the region under consideration. Following sub-step 813, or if it is determined in sub-step 811 that the memory density is not above the threshold t, processing in step 603 returns to sub-step 833 from where further sets S in the data structure A of order K are processed, if such sets S exist.

738457 -27o Turning to the example display list of Fig. 5, at the end of step 601 of the method o 600 as detailed in Fig. 3, the data structure A should contain the data shown in Table 2, with the memory density of all sets of cells initialized to "unknown", memory saving initialized to 0 and no cells marked for rendering. In step 603, since counter K was t 5 initialized to 1 in sub-step 805, all sets of cells S in data structure A which are of order 1 are processed in turn. In sub-step 806, set {902} with memory usage of 30 is firstly N obtained. In sub-step 807, the memory density of set {902} is determined to be unknown, therefore the memory usage total M is initialized to 30 in sub-step 808. There is no set S in data structure A with order less than 1. Therefore in sub-step 809 it is decided that memory usage of all display list objects that lie within cell 902 have been considered.

Indeed, looking at Fig 5, it can be noticed that there is only one object 924 that lies completely within cell 902 and its memory usage is The memory density of the region comprising cell 902 is next calculated in substep 810, which is the memory usage total M (30) divided by number of cells which is equal to 30. The memory density of region {902} is also stored in data structure A. In sub-step 811, it is determined that the memory density of 30 is greater than the current threshold t of 10. Therefore, cell 902 is marked for rendering in sub-step 812. The memory saving of 30 10 20, is accumulated in sub-step 813.

In sub-step 833, it is determined that there is still another set S of order 1 that has not been processed, which is set {904}. Set {904} is processed in a manner similar to that described above for set {902), and its memory density is calculated to be 5, which is not high enough for cell 904 to be rendered. Therefore, at the end of processing all sets of order 1, cell 902 is marked to be rendered and the accumulated memory saving is 20. An example of how the memory usage of subsets of a set of cells is accumulated will be given below.

738457 -28- As explained above, cells that are marked for rendering are removed from further consideration. Accordingly, after it is determined in sub-step 833 that there are no more Ssets of cells S of order K in the data structure A, rendered cells are removed from Cconsideration in sub-steps 818 to 829. In particular, in sub-step 818 it is determined 5 whether any cells are marked for rendering. If there are no cells marked for rendering, Ct then there are no cells to be removed from consideration and step 603 proceeds to sub- N step 830.

If it is determined in sub-step 818 that one or more cells are marked for rendering, then in sub-step 819 (Fig. 4C), one set of cells E from the data structure A is selected, and in sub-step 820 the marked cells are removed from the selected set E. In the process of removing marked cells from a selected set E, the selected set E may remain unchanged (when all cells in the selected set E are unmarked), the selected set E may become empty (when all cells in the selected set E are marked), the selected set E may merge with another existing set S in data structure A (when the selected set E after removal of the marked cells is the same as one of the sets S already in data structure A), or a new set is made which is yet to exist in data structure A.

Data structures A and B are updated to reflect these changes. Accordingly, in sub-step 821, it is determined whether the selected set E has been modified after marked cells are removed from that set E. If it is determined that the set E that has been selected in sub-step 819 has been modified in sub-step 820, it is then determined in sub-step 822 whether the modified set E has merged with an existing set S in data structure A. If the modified set E has merged with an existing set S in data structure A, step 603 proceeds to sub-step 823 where the memory usage of the set S with which the modified set E merges is updated to include the memory usage of modified set E.

738457 -29o If at sub-step 822 it is determined that the modified set E has not merged with an o existing set S in data structure A, then step 603 proceeds to sub-step 824 where it is determined whether the modified set E is empty. If the modified set E is not empty, then C step 603 proceeds to sub-step 825 where the modified set E is added into data structure A.

At sub-step 825, the modified set E being a non-existing set in data structure A, a new entry containing the modified set E is created in data structure A, with the memory N usage and memory density value of that modified set E initialized to "unknown" since the memory density has not been calculated.

Following either sub-step 823 or sub-step 825, step 603 then proceeds to substep 826 where, if the modified set E has an order less than, or equal to, counter K, the modified set E is added to data structure B. Following sub-step 826, or if it is determined in sub-step 824 that the modified set E is empty, the original set E obtained in sub-step 819 is removed from data structure A, as well as from data structure B if set E is therein.

The memory density value of the modified set E is then invalidated in sub-step 828 since its memory usage has changed.

Following sub-step 828, or if it is determined in sub-step 821 that the set E obtained in sub-step 819 has not changed after the marked cells were removed therefrom in sub-step 820, it is then determined whether there are any more sets E in data structure A that have not been obtained at sub-step 819. If there are such sets E, then step 603 returns to sub-step 819, otherwise step 603 proceeds to step 830 (Fig. 4A).

Turning back to the example of Fig. 5, at the end of processing all sets S of order 1 through sub-steps 806 to 816 (Fig. 4B), cell 902 is marked to be rendered. In sub-step 818, because cell 902 has been marked for rendering, cell 902 is removed from further consideration in all sets S in sub-steps 819 to 829. Referring also to Table 2, each set in data structure A is processed in turn. The method 600 starts by examining set {902} in 738457 sub-step 819, and removes cell 902 from set {902} in sub-step 820. This makes set {902} Sempty. As determined in sub-step 821, an empty set is different from set {902}. In substep 824, it is decided that indeed the modified set is empty, therefore set {902} is Sremoved from data structure A in sub-step 827. Set {902} is not in data structure B, therefore set {902} is not removed from data structure B in sub-step 827.

There are still sets of cells in data structure A which do not have marked cell 902 removed from them as determined in sub-step 829. Therefore, the process of removing marked cell continues with sets {904}, {902, 905}, etc., until all sets of cells S in data structure A have been processed. Removing cell 902 from set {904} does not modify set {904} so the algorithm proceeds to sub-step 829 and continues to process set {902, 905}.

Removing cell 902 from set {902, 905} creates a new set {905} which is yet to exist in data structure A as determined in sub-step 822 and sub-step 824. Therefore the new set {905} is created as an entry in data structure A with memory usage of 5, which is the memory usage of the original set {902, 905}. Since set {905} is of order 1 and counter K is 1, set {905} is added to data structure B in sub-step 826, and the original set {902, 905} is removed from data structure A in sub-step 827. In sub-step 828 the memory density of this new set {905} is invalidated by designating the same as "unknown". Sub-steps 819 to 829 are repeated until all sets S in data structure A have been processed as determined by sub-step 829. At the end of this processing, data structures A and B will contain the data shown in Table 3.

738457 -31 Order Regions Memory Usage Memory Density 1 {904} 5 {905} 5 unknown 2 {901,904} 50 unknown 3 {905,906, 908) 1 unknown {901,904, 905} 5 unknown 4 {905, 906, 908, 909) 2 unknown {901,903,904,905,906,907,908,909) unknown B {905} Memory Saving K 1 Cells To 902 Render Table 3 Notice that although the memory density of all order 1 sets has been calculated and a determination made as to whether to render each set in sub-steps 805-816 and 833, there is now a new entry in data structure A of order 1, set {905 which has an unknown memory density. This means that it has not been determined whether or not the region comprising cell 905 is dense enough. This is a result of removing marked cells after each order has been processed. For the case of set {905 it was added to data structure A when cell 902 was removed from set {902, 905). As explained above, this algorithm processes sets of cells from the lowest order to the highest order. Therefore, before proceeding to process sets of cells with order higher than 1, those sets of cells with order 1 which have 738457 -32not been processed should be processed first. Data structure B is used to keep track of such sets of cells S. Therefore, the sets of cells S in data structure B would always have Sorder less than or equal to counter K.

C1 Referring again to Fig. 4A, if it is determined in sub-step 830 that data structure B contains at least one set, such sets are processed in sub-steps 834 to 846 (Fig. 4D).

Firstly, in sub-step 834 all sets of cells in data structure B of the lowest order are N obtained. Each of the sets obtained in sub-step 834 are processed in turn in sub-steps 835 to 846. Sub-step 835 determines whether any of those sets have not been processed as yet through sub-steps 835 to 846. If all sets have been processed, then processing returns to sub-step 818 (Fig. 4A).

If there is at least one unprocessed set then, in sub-step 836, one of those sets not yet processed is selected. Each of the sets of cells S in data structure B of the lowest order have their memory density calculated and a determination is made whether the region comprisingthe set of cells S is worth rendering by following sub-steps 837 to 846. Substeps 837 to 846 are identical to sub-steps 808 to 816 (Fig. 4B) described above, except that sub-steps 837 to 846 have an additional sub-step 840 of removing entries from data structure B once such entries have been considered. Accordingly, sub-steps 837 to 846 are not described in detail.

It is noted that only the lowest order sets of cells S in data structure B are processed in any single pass of sub-steps 834 to 846. This is because, as explained above, processing these sets may result in some cells being marked to be rendered, which therefore in turn need to be removed from further consideration. Therefore, sets of cells S in data structure B are considered from lowest to highest order, and cells that are marked for rendering are removed in between processing different orders by following sub-steps 818 to 829 described above. Furthermore, removing rendered cells from consideration 738457 -33- O could result in more sets being inserted into data structure B as seen in the example Sdescribed above. This processing of sets of lowest order in data structure B and removing d rendered cells from consideration is iterated until it is determined in sub-step 830 that Sthere are no more entries in data structure B. In that case, processing of sets of higher order may commence by proceeding to sub-step 831 where the counter K is incremented before processing returns to sub-step 832. If there is no sets S that has an order higher than, or equal to, the value of the increased counter K, as determined in sub-step 832, then all sets S in data structure A have been processed and the estimated memory saving accumulated so far can be reported in sub-step 847.

Returning to thie example based on the display list illustrated in Fig. 5, in Table 3 there is a set {905} in data structure B. Accordingly, in sub-step 830 (Fig. 4A) it is determined that data structure B is not empty. Therefore in sub-step 835 (Fig. 4D), it is determined that there is at least one set of lowest order in data structure B to be processed.

Since there is only one entry in data structure B, the lowest order is the order of set {905}, which is i.

Set {905} is thus obtained from data structure B in sub-step 836. In sub-step 837 the memory usage total M of set {905} is initialized to the memory usage recorded for set {905}, which is 5. It is then determined in sub-step 838 that there is no entry in data structure A which has an order less than 1, and therefore the memory density of set {905} is calculated in sub-step 839 to be 5, which is not dense enough to be rendered, as is determined in sub-step 841.

As there are no more sets of the lowest order in data structure B to be considered, as determined in sub-step 835, processing returns to sub-step 818 (Fig. 4A) where it is determined whether the processing of sets in data structure B has caused any cells to be marked for rendering.

738457 -34- No cells were marked for rendering while processing data structure B and data structure B is now empty, as determined in sub-step 830. At this stage, the data structures will contain the data shown in Table 4. Note that all sets of order 1 in data structure A have a valid memory density.

Order Regions Memory Usage Memory Density 1 {904} 5 {905} 5 2 {901,904} 50 unknown 3 {905,906, 908} 1 unknown {901,904, 905} 5 unknown 4 {905, 906, 908, 909} 2 unknown {901,903,904,905,906,907, 908,909} unknown B Empty Memory Saving K 1 Cells To 902 Render Table 4 Counter K is now incremented to 2 in sub-step 831. It is determined in sub-step 832 that there are sets in data structure A (Table 4) with order higher than, or equal to, 2 and that there is a set with order equal to 2 in sub-step 833. Therefore, sets in data structure A of order 2 are processed following sub-steps 806 to 816 and 833. Set {901, 904} has order 2. Since there are sets in data structure A with order less than 2, set {901, 738457 o 904} may have sub-sets. Therefore, an exhaustive search through all sets S with order less than 2 in data structure A is performed. In sub-step 809 it is determined that there are sets of order 1 to be examined. Set {904} is considered in sub-step 814. Set {904} is C determined to be a sub-set of set {901, 904) in sub-step 815, and therefore memory usage of set {904} is added to the memory usage total M in sub-step 816. The memory usage total M is now 55 (50 from set {901,904) and 5 from set {904}).

~Set {905} is not a sub-set of set {901, 904) as decided in sub-step 815, and therefore its memory usage is not added to the memory usage total M. Since the memory usage total is now 55, and the number of cells in set {901, 904) is 2, the memory density of the region comprising set {901, 904) is 27.5, as calculated in sub-step 810.

Accordingly, set {901, 904) is worth rendering as decided in sub-step 811. Therefore, cells 901 and 904 are marked for rendering in sub-step 812. The estimated memory saving from rendering cells in set {901, 904) is 55 20 35 and this value is accumulated in sub-step 813 as the memory saving through rendering the marked cells.

After processing all sets S of order 2 in data structure A, the data structures A and B will contain the data shown in Table 738457 -36- Regions Memory Memory Order Regions Memory Usage Memory Density 1 {904} 5 {905} 5 2 {901,904} 50 27.5 3 {905,906, 908} 1 unknown {901,904, 905} 5 unknown 4 {905,906, 908, 909} 2 unknown {901,903,904,905,906,907,908,909} unknown B Empty Memory Saving K 2 Cells To 901,902, 904 Render Table As there are some cells that are marked for rendering as a result of processing sets of order 2, these newly marked cells 901 and 904 are removed from consideration.

After this process, the data structure contains the data shown in Table 6. Notice that the estimated memory saving is now 55, which is 20 from rendering cell 902 and 35 from rendering cells 901 and 904. Notice further that set {905} now has a memory usage of instead of the 5 in Table 5, and has unknown memory density. As explained earlier, removing marked cells could result in some memory densities being invalidated. In this example, it is shown that processing sets of a higher order could also lead to the memory density of some sets of lower orders being invalidated. While removing marked cells 901 738457 -37o and 904 from order 1 sets in sub-step 820, set {905} remains unchanged as determined by O sub-step 821 with a memory usage of 5 and a memory density of 5. However, while removing marked cells 901 and 904 from order 3 set {901, 904, 905) in Table 5 at sub- N step 820, this order 3 set is merged with set {905} as decided in sub-step 822. Therefore, the memory usages of sets {901, 904, 905) and {905} are combined in sub-step 823 and the memory density of set {905} is invalidated in sub-step 828.

\O

N Order Regions Memory Memory Usage Density A 1 {905} 10 unknown 3 {905,906, 908) 1 unknown 4 (905, 906, 908, 909) 2 unknown 6 (903, 905,906, 907, 908,909) 1 unknown B {905) Memory Saving K 2 Cells To 901,902, 904 Render Table 6 While processing sets of order 2, set {905} is inserted into data structure B in sub-step 826 and needs to be processed before proceeding to order 3. Memory density of cell 905 is not high enough for cell 905 to be rendered as determined in sub-step 841. Set (905) is removed from data structure B in sub-step 840. Data structure B is now empty as determined in sub-step 835, which results in counter K being incremented to 3 in sub-step 831.

738457 -38- All sets in order 3 are processed, and so on, until all sets in data structure A have N been processed. The data structures A and B at the end of step 603 are shown in Table 7.

d.) SNote that step 603 has decided that cells 901, 902 and 904 should be rendered with an Sestimated memory saving of t Order Regions Memory Memory e¢3 Usage Density c A 1 {905} 10 3 {905,906,908} 1 3.6 4 {905,906, 908, 909} 2 3.25 6 {903,905, 906, 907, 908,909} 1 2.3 B Empty Memory Saving K Cells To 901,902, 904 Render Table 7 If the resultant estimated memory saving is not enough, as determined by step 604 (Fig. the display list memory optimizer 112 will increase the compression factor and perform step 603 (Figs. 4A, 4B, 4C and 4D) again at a lower threshold t as a result of a higher current compression level. Running step 603 again at a lower threshold t may l0 mark more cells to be rendered, thereby increasing the memory saving. Step 603 is run from the lowest order again, comparing each set's memory density against the new threshold t. This is where caching memory density become beneficial. The memory density of regions remain the same while the threshold t changes. Therefore, with the 738457 -39o same memory density, certain regions which were not chosen to be rendered at the Sprevious, higher threshold t may now be chosen to be rendered with a new, lower Sthreshold t.

N If the estimated memory saving determined by step 604 is enough, the display list memory optimizer 112 will issue a render instruction as in step 608, at which stage all ¢€3 marked cells are rendered, removing an object from the display list under construction N when all cells it spans are rendered. Some objects may span other cells in addition to the rendered cells. Preferably, such an object, which lies partly within rendered cells, is clipped to the cells not rendered. Such objects in the example of Fig. 5 are objects 921, 920, and 925. Depending on the painting model to be used in the final rendering of the display list, such clipping may not be required.

One method of implementing the clipping is by storing a binary map, each entry of which represents a cell, in the display list. Map entries with the bit set indicate that the corresponding cell has been fallback rendered, and other objects in the display list under construction at the time of fallback rendering may not, on rendering, encroach on the indicated cells.

An optimization to reduce the amount of image data rendered is to only render the area necessary for objects that contribute to the memory savings. Given a region R comprising the set of cells to be rendered, for each object i in the display list that lies within a region Bi (for example, a bounding box), the area that needs to be rendered is for all Bi in R, union(Bi). If objects that do not contribute to memory savings are omitted from this determination, then the result is a region that may be smaller than the region R established by step 603. This will increase the savings by producing a rendered result that is smaller than if rendering the entire region R.

738457 40 O Associative array A can be stored using an array of hash tables, one for each set order. The hash table is keyed by sets comprising one or more cells, and the data associated with the key provides the memory usage and memory density of the set.

Methods for storing the potentially large number of sets that can be generated by the method 600 include bit arrays, where a set bit represents the presence of an element in the set and a clear bit represents the absence of an element, and explicit representations, N where, for example, a count of the number of elements is followed by that many values Srepresenting the elements.

Since bit sets are preferable for large order sets while explicit representation are preferable for small order sets, a hybrid scheme is preferred. For example, a set of more than three elements can be stored in a bit array, while a set of up to three elements can be stored as a sparse set in a 32-bit word comprising three 10-bit elements and a 2-bit count.

Alternatively, if one observes that many sets in practice are rectangular, a 16-bit halfword may contain a top, left element in 10 bits, a 3-bit width and a 3-bit height. Various embodiments may be practiced without departing from the spirit and scope of the invention.

Data structure B can similarly be stored using a hash table keyed by sets, but without a usage or density value associated with each set.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise" and "comprises" have correspondingly varied meanings.

738457

Claims (7)

1. 2. A method according to claim 1 wherein said first compression level is chosen so as to reduce memory usage of said display list by a desired amount.
2. 3. A method according to claim 2 wherein said method comprises the further step of: selecting said region of said output area, wherein selection of said portion depends on said second compression level.
3. 4. A method according to any one of claims 1 to 3 wherein said raster image data is stored in layers, each said layer corresponding to a level of visual significance, and said compressing step comprises discarding one or more of said layers corresponding to levels of least visual significance. 738457 42 O 5. A method according to claim 1 wherein said second compression level is equal to said first compression level.
4. 6. A method of estimating an amount of memory usage reduction of a display list representing an output area to be rendered, said display list comprising graphic objects and raster image data, said method comprising the steps of: (NI estimating a first amount of reduction of memory usage that would result Sfrom compression of said raster image data to a first compression level; estimating a second amount of reduction of memory usage that would result from rendering graphic objects lying within a region of said output area to raster image data, compressing the rendeired raster image data to a second compression level, and replacing said graphical objects lying within said region with the compressed rendered raster image data, wherein said second compression level is related to said first compression level; and adding said first and second amounts of reduction of memory usage together to provide said amount of memory usage reduction of said display list, wherein at least some of said raster image data lies outside said region of said output area.
5. 7. A method according to claim 6 comprising the further step, if said amount of memory usage reduction is less than a desired amount, of repeating steps to with a an increased compression level; or otherwise, performing the steps of: compressing said raster image data to said first compression level; rendering graphic objects lying within said region to raster image data; compressing the rendered raster image data to said second compression level; and 738457 -43- replacing said graphical objects lying within said region with the compressed N rendered raster image data. S8. A method according to claim 6 or 7, wherein neither the first compression level nor the second compression level may be increased beyond a predetermined limit. S9. A renderer for reducing the memory usage of a display list representing an output area to be rendered, said display list comprising graphic objects and raster image data, said renderer comprising: means for compressing said raster image data to a first compression level; means for rendering graphical objects lying within a region of said output area to image data; means for compressing said image data to a second compression level, wherein said second compression level is related to said first compression level; and means for replacing said graphic objects lying within said region with compressed image data, wherein at least some of said raster image data lies outside said region of said output area. A system for estimating an amount of memory usage reduction of a display list describing an output area to be rendered, said display list comprising graphic objects and raster image data, said system comprising: means for estimating a first amount of reduction of memory usage that would result from compression of said raster image data to a first compression level; means for estimating a second amount of reduction of memory usage that would result from rendering graphic objects lying within a region of said output area to raster 738457 -44- image data, compressing the rendered raster image data to a second compression level, N and replacing said graphic object lying within said region with the compressed rendered raster image data wherein said second compression level is related to said first compression level; and means for adding said first and second amounts of reduction of memory usage together to provide said amount of memory usage reduction of said display list, CI wherein at least some of said raster image data lies outside said region of said output area.
6. 11. A method substantially as described herein with reference to the accompanying drawings.
7. 12. A renderer substantially as described herein with reference to the accompanying drawings. DATED this 16th Day of DECEMBER 2005 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant SPRUSON&FERGUSON 738457