AU2005239671A1 - Improving the Consistency of Image Quality - Google Patents

Description

S&FRef: 733044

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address of Applicant: Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3-chome, Ohta-ku, Tokyo, 146, Japan Dixon De Sheng Deng Spruson Ferguson St Martins Tower Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Improving the Consistency of Image Quality The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5845c -1- IMPROVING THE CONSISTENCY OF IMAGE QUALITY (Ni Field of the Invention

O

Z The present invention relates generally to the printing of digital images and, in Cc particular, to altering the compression levels of digital images in a printer.

Background A printer has a limited amount of memory. A variety of methods are used to Inreduce the memory used in a printer when more memory is needed to process further objects which are to be printed.

One method used is to rasterize a portion of a page to produce an image, compress the image, and replace the objects in the portion of the page with the compressed image. This has the effect of reducing quality of not only embedded natural images but also of other types of graphics where loss of quality is more noticeable.

In a variation, when rasterizing a portion of a page, the method takes into consideration the most prominent type of graphics in the selected portion in order to determine the best compression method. All graphical objects in the selected portion are then subject to the one chosen method of compression.

Another method used is to pause the construction of the display list of graphical objects being constructed to represent a page, while some completed display lists which are spooled for printing, are printed, and the memory they use is released. When sufficient memory becomes available, construction of the display list resumes. This has the effect of possibly introducing delays in the printing of some pages.

Summary It is an object of the present invention to provide an improved management of memory associated with display lists in a printer.

733044 -2- According to an aspect of the present invention, there is provided a method of

O

N reducing memory usage of two or more display lists within a printer, each display list z comprising one or more rendering commands and at least one image, and each image has an associated image compression level, said method comprising the steps of: identifying one or more images with a lowest compression level; increasing the image compression level associated with the identified images; and O compressing said identified images to said increased compression level.

According to another aspect of the present invention, there is provided a printer for implementing the aforementioned method.

Other aspects of the invention are also disclosed.

Brief Description of the Drawings One or more embodiments of the present invention will now be described with reference to the following drawings, in which: Fig. 1 is a block diagram of a rendering system in which the embodiments described herein may be practiced; Fig. 2 is a flow diagram of a method of adding images to an image store of the rendering system of Fig. 1; Fig. 3A is a flow diagram of a method of reducing the memory usage of the image store according to an embodiment of the present invention; Fig. 3B is a flow diagram of a method of reducing the memory usage of the image store according to an alternate embodiment; Fig. 4 is a diagram showing an example of some areas of images that may be cropped; and Fig. 5 is a diagram showing an example of a tiled image that is to be cropped.

733044 -3- 0 Detailed Description including Best Mode o The disclosed embodiments of the present invention permit the management of Smemory used in storing images referenced in display lists for printing by a printer through e¢3 the management of the quality of the stored images. The embodiments of the invention Smay be practiced on a rendering system 100, the architecture of which is shown in Fig. 1.

NO

\The components of the rendering system 100, which may be used in a digital printer, may t be divided into a number of main groups, which include those components dealing with Screating a display list from an input job 101, those components dealing with rendering the display list into pixels 120 destined for a printer engine (not illustrated), and a memory manager 119.

The job 101 may consist of one or more pages, and is sent to the rendering system 100 in a form utilizing a page description language (PDL). Typical PDLs include Adobe® PostScript®, Hewlett-Packard® PCL and Adobe® PDF. The rendering system 100 includes a PDL interpreter 102 which translates the page description of the input job 101 into high-level graphics primitives.

A display list module 104 constructs display lists from the high-level graphics primitives provided by the PDL interpreter 102. The display lists include rendering commands to place objects on a page. Before passing the high-level graphics primitives to the display list module, the high-level primitives may first be filtered through a display list input optimizer 103, which comprises a set of configurable filters, for idiom recognition and optimization. The completed display list produced by the display list module 104 may be filtered through a display list output optimizer 105 to ensure that the page, represented in the display list, can be rendered in real time, before the display list is preprocessed for rendering and spooled onto a spool queue 116 of the rendering system 100. The memory manager 119 invokes a display list memory optimizer 112 when the 733044 -4amount of available memory in the rendering system 100 is low. The display list memory optimizer 112 changes the form of all or part of the display list so as to reduce the 0 z memory usage of the display list. The display list stores objects in high-level form, Cutlizing data stores for storing data of various types.

The memory includes an image store 107, a property store 108, a path store 110 \and a text cache 109 for storing image data, fill and stroke specifications, path outlines, Cc and glyphs respectively. Any item of data stored within the memory is potentially shared ¢In within and between pages, and remains in existence as long as that item of data is referenced by the display list, or any job in the spool queue 116. The data stores 107 to 110 register callback functions with the memory manager 119 for reclaiming and minimizing memory usage in accordance with urgency.

The image store 107 receives images from the PDL input job 101, compresses these images using an image compressor 111, and then stores the compressed images. The image compressor 111 provides variable degrees of image compression with corresponding graceful quality reduction. To help reduce memory usage, only one instance of an image is stored, where possible, across multiple pages.

The property store 108 stores fill specifications, such as flat or blended colour, as well as stroking specifications.

The path store 110 stores path outlines that include line and spline segments in one of a number of formats. Formats include floating point or 32-bit fixed point, or smaller sized fixed-point representations, depending upon which format is the most compact for a given outline without sacrificing quality.

The text cache 109 stores glyph (character) bitmaps and/or outlines. The text cache 109 needs to interface with a font engine 106 to obtain the glyphs. The text cache 109 has its own instance of a path store for storing glyphs in spline format. Naturally only one instance of each glyph is stored for use any number of times on one or more pages.

733044 The memory manager 119 is responsible for memory allocation and ensuring N that the rendering system 100 has enough memory to perform its tasks. The memory 0 z manager 119 uses compaction and reclamation in the event of low memory to try and ensure that the caller can continue to allocate memory. When the available memory starts to get limited, the memory manager 119 invokes a series of steps, aimed at sacrificing t-speed and quality in exchange for reduced memory usage. The rendering system 100 is designed to degrade the speed and quality gracefully, reducing perfomance and output quality only as much as is needed to complete its task of rendering page descriptions.

Rendering may be performed in two situations namely, fallback rendering for dealing with running out of memory, and job rendering for generating pixel output 120 to the printer engine. The rendering System 100 is accordingly divided into two parts (or render subsystems) 118 and 115-117, with the respective subsystems 118 and 115-117 being dedicated to respective situations. A render preprocessor 114 serves as a common interface for both subsystems.

The render preprocessor 114 converts the display list into edge, level and fill format as required by both rendering subsystems 118 and 115-117. The render preprocessor 114 sorts the display list into the order (Y order) required by the rendering subsystems 118 and 115-117, and vectorizes spline-based paths and text to provide data to a fallback renderer 118 or a render simplifier 115.

The render preprocessor 114 invokes edge vectorizer 113 to convert splines into line segments.

During fallback rendering, render preprocessor 114' then invokes fallback renderer 118 to convert a selected region of the page currently being constructed, into pixel data, which is suitably compressed using the image compressor 111. The new image thus created is stored in the aforementioned image store 107 and a reference to that new image is added to the display list.

733044 -6- During job rendering, render preprocessor 114 invokes render simplifier 115 to scan convert convert to a scanline representation) and analyse the display list. The

O

0 render simplifier 115 produces a simplified spool format for the page which can be rendered in real time by a page renderer 117. In this process, new fills and images may be created. These new fills and images are stored in the aforementioned data stores. The simplified job is then queued in spool queue 116, to be rendered into pixel data 120 by page renderer 117. The pixel data are ultimately used by a print engine to produce a Sphysical output.

Images that are to be included in the rendered output for a page, whether referenced from a completed display list in the spool queue 116, or referenced from the display list under construction, are held in the image store 107. As mentioned above, the image store 107 uses the image compressor 111 to compress the images to reduce the amount of memory they occupy. The level to which each image is compressed is associated with each image in the image store 107. The images are removed from the image store 107 when such images are no longer referenced by the display list under construction or by the spooled display lists. This removal may occur when a spooled display list is printed or when a region of the display list under construction which wholly contains the image is fallback rendered.

Where possible, the image store 107 shares images within and between pages.

The image store 107 does so by calculating a signature for each image received. If the signature matches that of an existing stored image, the image store 107 discards one of the two images. Since it is possible that the existing copy of the image might have been cropped, the new, complete copy is retained. One method for calculating a signature is computing a Cyclic Redundancy Checksum (CRC) -32 on the pixel values of the image.

More reliable methods include the message digest algorithms MD4 and 733044 -7- If a new image has a signature that is different from all images stored in the Simage store 107, the new image is also stored in the image store 107 and is assigned a 0 z unique image reference which is used to refer to the image by other parts of the rendering Csystem 100. Alternatively, if the new image has a signature that is the same as one of the images already in the image store 107, that is the new image is the same as an image r already stored in the image store 107,then the old copy of the image is discarded, the new Cc copy is stored in the image store 107, and the image reference of the matching existing (Ni image is used to refer to the image.

The image compressor 111 is able to compress images at a range of compression levels, each level increasing the degree of compression with consequent reduction in memory usage, but also with consequent degradation in visual quality. In practice, a certain threshold level of compression can be applied with negligible visual quality loss.

However, at levels higher than that threshold level of compression, artifacts typically begin to appear. In the preferred embodiment, a compression method that preserves the edges and colour of regions of uniform colour is utilized so that even at relatively high compression levels, the image store 107 can be used to store images created as a result of fallback rendering.

A further aspect of the image compressor 111 is that it stores an image's data in a series of layers, each layer being responsible for a certain level of visual quality, as is known in (for example) the JPEG2000 image compression standard. Thus, compression can be increased efficiently, simply by discarding the layers corresponding to the least visually significant information, without having to decompress and recompress stored image data. Images can thus be stored initially at maximum (lossless) quality and gradually degraded only to the minimum extent necessary for the rendering system to have enough memory to function.

733044 -8- Furthermore, the image compressor 111 stores images in tiles. This enables Ncropping of portions of an image that are not needed, due for example, to clipping.

O

z When memory manager 119 determines that memory usage is above a predetermined threshold, it makes a request to the image store 107 to reduce its memory usage. The image store 107 coordinates the compression and cropping, including the discarding of tiles and layers by the image compressor 111 in order to fulfill as far as possible, the memory manager's request.

OIn order to coordinate compression of stored images, the image store 107 in one embodiment associates an image compression level with each display list. When a display list is created this compression level is set to the lowest possible value representing minimum compression. When the memory manager 119 requests the image store 107 to reclaim memory, the image store 107 reduces image quality of stored images. To achieve this the image compression level of the images referenced by the display list under construction, and/or the images referenced by spooled display lists may be raised as detailed below.

New images added to the image store 107 are compressed to the current compression level of the display list referencing those images. Thus, for example, a first image referenced in a display list may be stored at minimum compression. After a while, the memory manager 119 may request that the image store 107 reduces its memory usage, in which case the image store 107 may increase the compression of the display list. The first image is then compressed to the increased compression level of the display list.

When a second image is to be stored in the image store 107 and referenced by that display list, the second image will be compressed and stored at the current, increased compression level of the display list.

Fig. 2 shows a schematic flow diagram of a method 200 for adding a new image to the image store 107. The method 200 starts at step 201 where the image store 107 733044 -9receives image data from an image creator process, e.g. the PDL interpreter 102 or the N render preprocessor 114, and compresses the received image data to the current z compression level of the display list referencing that image using the image compressor 111. In the process of reading the pixel data (that is, before compression), the image store 107 computes a signature for the image.

At step 202, the image store 107 compares the signature with that of all other images already stored in the image store 107. This may be implemented efficiently, for (Ni Sexample, by means of a hash table. At step 203 the image store 107 then determines whether the signature of the new image matches that of an existing image. If it is determined that a match does not exist, the method 200 proceeds to step 204 where the compressed image data is added to image store 107. At step 205, an identifier is created for the image and returned to the image creator process for the purpose of storing that identifier in the referencing display list.

If it is determined in step 203 that a match does exist, the method 200 proceeds to step 206 where the new image data replaces the image data of the image sharing the same signature. The reason for preferring the new image data to that of the an existing image is that the new image data is guaranteed to be complete, whereas earlier data for the same image may have been subjected to cropping and thus may be incomplete.

Cropping is further described below. At step 207, the image store 207 then returns the existing identifier to the image creator process for the purpose of storing that identifier in the referencing display list. The method 200 for storing an image to the image store 107 terminates following step 205 or 207.

When the memory manager 119 requests that the image store 107 reduce the image quality of stored images, the image store 107 examines the image compression level for each display list still in the rendering system 100, namely, the display list currently under construction and all the spooled display lists in the spool queue 116 733044 whose printing has not yet commenced. In order to degrade pages roughly uniformly and Nto not degrade one page excessively, the display list which has the lowest image 0 z compression level (least compression) is selected. If more than one display list has the lowest image compression level, any one, or more, or all, of the display lists are selected dependent upon the amount of memory reduction requested by the memory manager. The image compression level of all the images referenced by the selected display list is then 0 increased to thereby release momory in the image store 107.

ODuring the stage of memory reclamation now being described in more detail, the image compression level will not be increased to a level where there is a significant impact on the visual quality. If the memory manager's request cannot be fulfilled during this stage, the image store 107 only performs what memory reduction it can without causing severe visual degradation of compressed images. Once the level of compression has been reached for all stored images where further compression will result in severe visual degradation of those images the memory manager 119 moves to a different strategy to reclaim the remaining memory required.

As noted above, when the image compression level for a selected display list is increased, the images that are referenced by that display list are adjusted to the new compression level. In the preferred embodiment, the compression level of an image is increased by discarding the least visually significant layer of information available for that image. In alternative embodiments, it may be necessary to achieve the same effect more slowly by decompressing, and recompressing each image referenced by the display list at the higher compression level. This is typically required where the compression standard used does not store different layers of data, with each layer being associated with a particular visual quality level.

As images can be shared between display lists, it is possible that one or more images referenced by the selected display list is already compressed more highly than 733044 -11other images referenced by that same display list. When the image compression level of the selected display list is increased, images already compressed at that level or above are 0 z not further compressed.

Fig. 3A shows a schematic flow diagram of a method 300 of reducing the memory usage of the image store 107 in response to a request from the memory manager 119. The method 300 starts at step 301 where the image store 107 determines whether all display lists in rendering system 100, (being display lists in the spool queue 116 or the Sdisplay list under construction), are at or beyond a maximum acceptable compression level. This level is not the maximum achievable compression level, but the preselected maximum compression level for which there is no, or only slight, visual degradation.

If it is determined that there is at least one display list not at or beyond that preselected maximum compression level, the method 300 proceeds to step 302 where a display list with a lowest compression level is chosen. At step 303, the compression level for the chosen display list is then increased by one level. At step 304, all images referenced by the chosen display list that are compressed to a level below the increased compression level of the display list are then compressed to that display list's new compression level.

The image store 107 next determines at step 305 whether or not the amount of memory requested by the memory manager 119 has been recovered. If it is determined that sufficient memory has been recovered, the method 300 terminates and control is returned to the memory manager 119. Alternatively, if it is determined in step 305 that an insufficient amount of memory has been recovered, control continues back to step 301 from where the compression level of one of the display lists is increased if the condition of step 301 is not met.

If at step 301 it is determined that all display lists have been compressed to the preselected maximum compression level, then the method 300 terminates and control is 733044 12returned to the memory manager 119. In this case, the memory manager 119 may take other steps to reclaim memory within the rendering system 100.

An alternate embodiment of reducing the memory usage of the image store 107 in response to a request from the memory manager 119 is now described. In this alternate embodiment, the image store 107 maintains a global compression level, and each image in the image store 107 has an associated image compression level. When a new display list is created, the global compression level is set (or reset) to the lowest possible value, representing minimum compression. When the memory manager 119 tries to reclaim memory from the image store 107, the image store 107 reduces image quality of stored images by raising the global compression level of the image store 107, as is detailed below.

New images added to the image store are compressed to the image store's current global compression level. When a new display list is being constructed the global compression level is reset to a minimum level.

When the memory manager 119 requests that the image store 107 reduce its memory usage, the image store 107 may increase the global compression level. At this stage of memory reclamation, the global compression level is not increased to a level where there is a significant impact on the visual quality of images. If compression to such a level is required to fulfil the memory manager's request, the memory manager 119 moves to a different strategy to reclaim the memory required.

Following an increase in the global compression level, the image store 107 examines the image compression level for all images on the display list under construction and all images on all spooled display lists whose printing has not commenced. All examined images with a compression level lower than the new current global compression level are (further) compressed to the new global compression level.

733044 -13- Also, all images added to the display list currently under construction are added to the N image store 107 at the current global compression level.

O

z Fig. 3B shows a schematic flow diagram of a method 350 of reducing the Cmemory usage of the image store 107 in response to a request from memory manager 119. The method 350 starts at step 310 where the image store 107 determines whether the t current global compression level is at a maximum acceptable compression level. As in the Cc case of the method 300 described with reference to Fig. 3A, this is not the maximum (Ni achievable compression level, but rather the maximum compression level for which there is no, or only slight, visual degradation of images. If it is determined in step 310 that the current global compression level is lower than the maximum acceptable compression level, the method 350 proceeds to step 311 where the image store's current global compression level is increased by one level. The method 350 then proceeds to step 312 where all images with compression level lower than the current global compression level are further compressed to the current global compression level. The compression level of each image is also updated.

The method 350 then continues to step 313 where the image store 107 determines whether or not the amount of memory requested by the memory manager 119 has been recovered. If so, the method 350 terminates and control is returned to the memory manager 119. However, if it is determined in step 313 that an insufficient amount of memory has been recovered by the image store 107, control returns back to step 310 from where the global compression level is further increased if the condition of step 310 is not met.

If at step 310 it is determined that the current global compression level is already at the maximum acceptable compression level, then the method 350 terminates and again control is returned to the memory manager 119.

733044 14- As noted above, each time a new display list is constructed the global N compression level is reset to a minimum level. This typically results in the first image at 0 least of that new display list being stored at the minimum level. As more memory is requested to be reclaimed by the memory manager 119, the image store 107 increases the global compression level. As a result the image store 107 (further) compresses the images of the currently formed display list and other images in the image store 107 still compressed at a level below the current global compression level. Accordingly, there may Sbe images stored at various compression levels in the image store 107.

As noted above, only one copy of an image shared by more than one display list is stored in the image store 107. It may occur that images in a display list have been compressed to different compression levels. In the embodiment described with reference to Fig. 3A this typically occurs when an image is shared between display lists which are compressed to different levels, in which case the shared image is typically compressed at the level of the display list having the lower compression level. In the embodiment described with reference to Fig. 3B this may occur when an image is shared between display lists, resulting in a less compressed version of that image replacing a previously stored, more compressed version in the image store 107 while the remainder of the images of the older display list are compressed at a higher global compression level that was current at a time before the more compressed version of the image was replaced. In these cases, it is possible to maintain a consistent level of image quality across all images on a page by degrading images where necessary to a level consistent with that of the most highly compressed image when rendering the page in page renderer 117. The copies of the degraded images remaining in the image store 107 are not compressed in this process as those images are still required by one or more remaining display lists.

In both embodiments described with reference to Figs 3A and 3B, the mefnory manager 119 may also request that the image store 107 should attempt to recover memory 733044 by cropping images. The cropping is designed to discard portions of images that are not contributing to any page upon which such images appear. Consider the examples in Fig. 4 0 z where a page 41 is illustrated. If an image 42 is partially off the area of the page 41, only the portion 42a that is on the page 41 contributes to the appearance of the page 41.The remaining portion 42b of the image 42 can be safely discarded without affecting the appearance of the page 41.

Cc Similarly, an image 43 may be clipped to a path 43a so that less than the entire ¢In image 43 contributes to the appearance of page 41. Again, the non-visible portion 43b of image 43 can be discarded without affecting the appearance of the page 41.

Additionally, it may be possible to determine that part of an image 44 is hidden by another object 45 that appears on the page 41, for example where the two images 44 and 45 overlap. In this case the hidden portion 44b of image 44 is not required and can be deleted without affecting the appearance of the page 41. The latter example is particularly significant when fallback rendering is used to replace a portion of output area 41. by an image which is the result of rendering objects wholly or partially contained in that portion. In such cases, the resulting image is placed over all the objects, thus making it possible to remove such objects from the display list, or where possible, to clip or crop such objects to outside the rendered portion.

In the case of an image that is used more than once, either on the same page (and hence the same display list) or on different pages (or different display lists), the portion that is visible in each case must be considered. Only the portion that is hidden in every use can be discarded. So for example, if an image is used on one page where only the left half is shown and on a second page where only the right half is shown, in total the entire image is used and nothing can be discarded.

In the preferred embodiment, the entire hidden portion of an image is not necessarily discarded. Instead, only the image tiles that are wholly contained in the hidden 733044 -16portion are discarded. This is more efficient than discarding the entire hidden portion of N' the image because the hidden tiles can simply be discarded. Image 51 in Fig. 5 is an 0 z example of an image that is able to be cropped. The image 51 comprises tiles 5a through formed by the image compression. In this example shaded portion 53 is not visible. The tiles 5g, 5h and 5i are contained entirely within the hidden portion 53. Accordingly, ththese tiles 5g, 5h and 5i can be discarded without affecting the appearance of the page Mc) containing image 51. Tiles 5d, 5e and 5f are only partly contained with the hidden portion

(N

53. Because they are partially visible, these tiles 5d, 5e and 5f cannot be discarded.

The foregoing describes only some embodiments of the present invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

In the context of this specification, the word "comprising" means "including principally but not necessarily solely" or "having" or "including", and not "consisting only of'. Variations of the word "comprising", such as "comprise" and "comprises" have correspondingly varied meanings.

733044

Claims (4)

1. 2. The method according to claim 1 wherein each display list has an associated compression level; and wherein said identifying step comprises: identifying a display list with a lowest compression level, and identifying the images associated with the identified display list.
2. 3. The method according to claim 1 wherein said one or more images are identified through identifying images having associated compression. levels below a global compression level.
3. 4. The method according to any one of claims 1 to 3, wherein an upper bound is applied to the amount to which said compression level is increased. The method according to any one of claims 1 to 4, further comprising the step of determining whether said compressing step has reduced memory usage by a sufficient amount, and if it is determined that said amount is insufficient, repeating steps to 733044
4. 18- 6. The method according to any one of claims 1 to 5, wherein at least one of said N display lists is in a spool queue awaiting rendering. 0 O7. The method according to any one of claims 1 to 6 wherein said images are organized into layers of increasing quality, and wherein said compressing step compresses said identified images to said increased compression level by discarding one Nor more of said quality layers of each identified image. 8. The method according to claim 2 wherein said images are stored in an image store, and each image being added to said image store is added at the compression level associated with the display list referencing said image. 9. The method according to claim 8 wherein each new display list added to said rendering system is associated with a minimum compression level. The method according to claim 3, further comprising the steps of increasing said global compression level and repeating steps to 11. The method according to claim 10 wherein said images are stored in an image store, and each image being added to said image store is added at the global compression level. 12. The method according to claim 11 further comprising the step of resetting said global compression level to a minimum value before the creation of a new display list. 13. A rendering system configured for reducing memory usage of two or more display lists, each display list comprising one or more rendering commands and at least one image, and each image has an associated compression level, said rendering system comprising: 733044 -19- Smeans for identifying one or more images with a lowest compression level; means for increasing the compression level associated with the identified 0 z images; and means for compressing said identified images to said increased compression level to thereby reduce memory usage of said one or more display lists within said rendering system. NO S14. A rendering system according to claim 13, wherein at least one of said display lists is in a spool queue awaiting rendering. A method of reducing memory usage of two or more display lists within a rendering system, said method being substantially as described herein with reference to the accompanying drawings. 16. A rendering system configured for reducing memory usage of two or more display lists within said rendering system, said method being substantially as described herein with reference to the accompanying drawings. DATED this 30th Day of November 2005 CANON KABUSHIKI KAISHA Patent Attorneys for the Applicant SPRUSON&FERGUSON 733044