NO329337B1 - Mixed raster content files - Google Patents

Abstract

A system and process for generating and using mixed raster content files is described. MRC files may be created directly from an application, rather than being processed from a composite image, to separate a single image into background, foreground, and selection masks for each page. Also, an improved MRC structure is described in which only relevant portions are contained in background, foreground, and selection layers for each page. <IMAGE>

Description

[01 ] Aspects of the present invention are directed to generating and using files. More specifically, the present invention relates to generating and using files with mixed raster content.

[02] The use of non-modifiable files has increased with the rise of the internet. One standard file format for publishing documents is the Adobe Acrobat file format (commonly associated with the .PDF file extension). Other file formats include JPEG and bitmap files. An advantage of these files is that one can use these file formats to exchange information over the internet with minimal concern about modification of the documents.

[03] Another file format that is commonly used is MRC files (mixed-raster-content), i.e. files with mixed raster content. This file format divides an image into three equal-sized images and stores them at different resolutions and, in some cases, with different compression ratios or technologies. Figure 2 shows a typical process for generating MRC files. An image generator 301 produces a source image 302. The source image 302 has an image resolution of 300 dpi. The source image 302 is divided by a separation process 303 into three separate images 304-306. The three separate images comprise a black and white (B/W) mask image 304, an image foreground 305 and an image background 306. The three images 304-306 have the same dimensions (x,y) as the source image 302. Figure 2 is the resolution of the three images 304-306 respectively shown as 300 dpi, 300 dpi and 100 dpi. The three images can be selectively compressed in a compression process 307 in different ratios using different types of compression technologies.

[04] Despite the advantages of being able to decompose a source image 302 into various other files 304-306, the decomposition process (separation process 303) for MRC files is time consuming, as is well known to those skilled in the art. The separation process 303 attempts to analyze all the components of an image and separate out aspects for inclusion in each of the files 304-306. There exists a need to generate MRC files without the long delay.

[05] Furthermore, the composite files 304-306 can be large, even if they have little content. Figure 3 shows an example of a conventional MRC file. The file comprises three layers, including a background layer 401, a selector layer or black and white layer or mask layer 402 and a foreground layer 403. For illustration purposes, the composite image 411 is shown as the resulting image when the three layers (401-403) are combined. As known to those skilled in the art, the background layer has a color resolution per pixel of 24 bits per pixel (24 bpp), the selector layer a color resolution per pixel of 1 bpp and the foreground layer a color resolution of 24 bpp.

[06] Each layer includes one image. Each image has the same dimensions (x,y)

(as shown in figures 2 and 3). For example, the background layer 401 includes one image with content 404 in the center. The selector layer 402 comprises one image with three-part content 405-407. The content 405 is "Black text", which is meant to represent black text. The content 406 is "Color text A", which is text that has a color A in the composite image 411, but which is nevertheless represented as black text in the selector layer 402. This is because the selector layer 402 has a color resolution of only 1 bpp. Correspondingly, the content 407 is "Color text B", which is text that has a color B in the composite image 411, but is still black text in the selector layer 402. The foreground layer 403 comprises three sets of color palettes comprising black 408, color A 409 and color B 410. One problem with the MRC file in Figure 3 is the significant amount of unused space. The unused space increases the underlying image size in each layer. This unused space increases the time required for the encoding, decoding and compression steps. Furthermore, the combination of unused space and content sometimes results in poor compression of the layers. Accordingly, there is a need to minimize the amount of unused space in the composite files 304-306.

Bottou U et al: "Efficient conversion of digital documents to multilayer raster formats" Document Analysis and Recognition, 2001. Proceedings, Sixth International Conference on Seattle, WA, USA, 10-13 Sept. 2001, pages 444-448, discusses how the description of a digital document can be transformed into something efficient for multi-layer raster formats.

The present invention is defined in the independent claims 1, 2, 4, 5, 7, 8, 10, 11 and 12. Preferred embodiments are indicated in the associated non-independent claims.

[07] Aspects are directed to generating and using MRC files to fulfill one or more of the needs described above. In some aspects, MRC files are generated by printer drivers without needing to be processed in a separation process. In other embodiments, each layer in MRC files may comprise one or more images (or subimages) to enable compression that is more efficient than having a single image covering each layer. These different aspects can be used separately or in combination.

[08] These and other special features of the invention will be obvious after a review of the following detailed description of various embodiments.

[09] The foregoing summary of the invention, as well as the following detailed description of preferred embodiments, will be better understood when read in conjunction with the accompanying figures, which are included by way of example and not as limitations with respect to the claimed invention.

[10] Figure 1 is a schematic diagram of a general digital computer environment that can be used to implement various aspects of the invention.

[11] Figure 2 shows a conventional process for creating a file with mixed raster content.

[12] Figure 3 shows a file with mixed raster content of the conventional type.

[13] Figure 4 shows a process for creating a file with mixed raster content according to aspects of the invention.

[14] Figure 5 shows an example of a file with mixed raster content according to aspects of the present invention.

[15] Figure 6 shows another example of a file with mixed raster content according to aspects of the present invention.

[16] Figures 7A and 7B show various processes for creating mixed raster content files according to aspects of the present invention.

[17] Figures 8A and 8B show alternative process steps for the process of Figure 7A.

[18] Figures 9A and 9B show various processes for rendering a composite image from a file with mixed raster content in accordance with aspects of the present invention.

[19] Figure 10 shows another exemplary process for creating a file with mixed raster content according to aspects of the present invention.

[20] Figure 11 shows an exemplary process for merging images according to aspects of the present invention.

[21] Figure 12 shows various results of merging images according to aspects of the present invention.

[22] Figure 13 shows an exemplary process for setting a mutual proximity value between images according to aspects of the present invention.

[23] Figure 14 shows two images and different representations of the space occupied by combining them according to aspects of the present invention.

[24] Figures 15A, 15B and 15C show different possibilities for calculating space consumption for combined images according to aspects of the present invention.

[25] The following description is aimed at files with mixed raster content. It is understood that the described concepts and methods can be applied to other types of files. Different processes are also described. These processes are provided as examples that may be used, and are not intended to be the only possible representations of the various processes that may be used to generate or use the enhanced mixed raster content files. Furthermore, the color of the pixels in the selector layer is generally designated as black and white. It is understood that the selector layer can only have a color resolution of 1 bit per pixel and therefore can only represent the pixels as off or on (or color state 1 and color state 0). Black and white are used to simplify the terminology. However, any colors can be used instead of these (e.g. fuchsia and cyan), or "off and on" and the like.

[26] Figure 1 illustrates a schematic diagram of a conventional digital computer environment that may be used to implement various aspects of the present invention. In figure 1, a computer 100 comprises a processing unit 110, a system memory 120 and a system bus 130 which connects various system components including the system memory to the processing unit 110. The system bus 130 can be any of many types of system buses, including a memory bus or memory controller, a peripheral bus and a local bus that uses any of a number of available bus architectures. The system memory 120 comprises a read-only memory (ROM) 140 and a random-access memory (RAM) 150.

[27] A BIOS (basic input/output system) 160, which contains the basic routines that transfer information between elements of the computer 100, for example during startup, is stored in ROM 140. The computer 100 also includes a hard disk drive 170 for reading from and writing to a hard disk (not shown), a magnetic disk drive 180 for reading from or writing to a removable magnetic disk 190 and an optical disk drive 191 for reading from or writing to a removable optical disk 192, such as a CD-ROM or another optical medium. Hard disk drive 170, magnetic disk drive 180, and optical disk drive 191 are respectively connected to system bus 130 via a hard disk drive interface 192, a magnetic disk drive interface 193, and an optical disk drive interface 194. The drives and their associated computer-readable media provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for the personal computer 100. Those skilled in the art will appreciate that other types of computer-readable media capable of storing data that can be accessed by a computer, such as magnetic cassettes, flash memory cards, digital image discs , Bernoulli cassettes, RAM, ROM and the like, can also be used in the exemplified operating environment.

[28] A number of program modules may be stored in the hard disk drive 170, magnetic disk 190, optical disk 192, ROM 140, or RAM 150, including an operating system 195, one or more application programs 196, other software modules 197, and program data 198. A user may input commands and information to the computer 100 via input devices such as a keyboard 101 and a pointing device 102. Other input devices (not shown) may include a microphone, joystick, game controller, satellite dish, scanner or the like. These and other input devices are often connected to the processing unit 110 via a serial port interface 106 which is connected to the system bus, but may also be connected via other interfaces, such as a parallel port, a game output or a universal serial bus (USB). Furthermore, these devices can also be connected directly to the system bus 130 via a suitable interface (not shown). A monitor 107 or another type of display device is also connected to the system bus 130 via an interface, such as a video card 108. In addition to the monitor, personal computers typically include other peripheral output devices (not shown), such as speakers and printers. In some embodiments, a font digitizer 165 and an associated pen or position pointer 166 are provided to digitize freehand written information. Although a connection between the character digitizer 165 and the serial port interface 106 is shown, in practice the character digitizer 165 may be connected directly to the processing unit 110, via a parallel port or other interface and the system bus 130, which is well known to those skilled in the art. Furthermore, although the font digitizer 165 is shown separate from the monitor 107, the writable input area of the font digitizer 165 may coincide with the display area of the monitor 107. Still further, the font digitizer 165 may be integrated into the monitor 107, or may exist as a separate device located above or on in another way, the monitor 107 is attached or built on.

[29] The computer 100 may operate in a network environment via logical connections to one or more remote computers, such as a remote computer 109. The remote computer 109 may be a server, a router, a network PC, a peer device, or another common network node, and typically includes many or all of the elements described above in connection with the computer 100, although only a memory storage device 111 is illustrated in Figure 1. The logical connections shown in Figure 1 include a local area network (LAN) 112 and a wide area network (WAN) 113. Such network environments are common for offices, company-wide computer networks, intranets and the Internet.

[30] When used in a LAN, the computer 100 is connected to the local network 112 via a network interface or network card 114. When used in a WAN, the personal computer 100 typically includes a modem 115 or other devices to establish a communication link over the remote network 113, for example internet. The modem 115, which may be internal or external, is coupled to the system bus 130 via the serial port interface 106. In a network environment, software modules shown in connection with the personal computer 100, or portions thereof, may be stored in the memory storage device located elsewhere .

[31] It will be understood that the network connections shown are only examples, and that other techniques can be used to establish a communication link between the computers. The existence of the various well-known protocols, such as TCP/IP, Ethernet, FTP, HTTP and the like, is poisoned, and the system can be operated in a client-server environment so that a user can download web pages from a web server. Any of various conventional browsers can be used to display and manipulate information on web pages.

[32] Figure 4 shows a process for creating a file with mixed raster content according to aspects of the invention. An image generator 500 (for example, a drawing program, a word processing program, an online publishing system, or the like) produces the contents of the MRC file (the three layers: the selector layer 502, the foreground layer 503, and the background layer 504) without creating a composite image that needs to be converted (i.e. separated) using the process 303 of Figure 2. Producing this file may include function calls to a graphics rendering engine 501 (eg, the GDI API available from Microsoft Corporation of Redmond, Washington). Other graphics engines 501 can also be used. Alternatively, the entire rendering process can take place in the image generator 500. Finally, the MRC file can optionally be compressed by the selective compression engine 505. This compression is not necessary, which is indicated by the box having a dashed frame.

[33] Figure 5 shows an example of a file with mixed raster content according to aspects of the present invention. The MRC file comprises three layers: a background layer 601, a selector layer 602 and a foreground layer 603. A composite image 604 is shown for explanatory purposes. The background layer comprises content 605. The content 605 may be represented by an image of the same size as the composite image 604. Alternatively, the background content may be represented by an image comprising a bounding rectangle that is smaller than the size of the composite image 604. The selector layer 602 comprises content 606 represented by "color text A" which, while represented by text having font color A, the selector layer 606 represents as black text. The foreground layer 603 comprises black text 607, a colored rectangle 608 with color A and content 609 in the form of "colored text B" with font color B. The colored rectangle 608 can be omitted in a number of cases. For example, if the text consists of one color and can be represented by the combination of the selector and background layers. The content 607-609 can be represented by a single image with the same size as the composite image 604 (corresponding to the description with identical size above). Alternatively, the content 607-609 can be represented by two or more images (or partial images). The images may be bounded by bounding areas (rectangles or other shapes, as known to those skilled in the art) which are represented by the bounding frames that surround each of the contents 607-609.

[34] As shown in Figure 5, the selector layer 602 is used when an image foreground overlaps an image background. In this case, moving content to the foreground layer from the selector layer will make rendering faster, as there is no need to check and select or perform alpha blending when there is no overlap between the foreground and background content. Another aspect is that solid text and certain line art can be saved as binary images in the foreground layer with an appropriate color palette, instead of being saved as a binary image in the selector layer plus a color map in the foreground layer. It is believed that moving content out of the selector layer and into the foreground layer will improve compression and rendering performance.

[35] Another combination is shown in Figure 5, where selector and background layers are provided, but not foreground layers. In this case, when there is a non-white pixel in the selection layer but no corresponding pixel in the foreground layer, the color of the pixel in the composite image is that of the selection layer. This combination is common and can be more efficient in terms of rendering performance and compression than the combination of background, selector, and foreground layers.

[36] The foreground layer can have a high color resolution (eg 24 bbp). In one embodiment, all the images in the foreground layer have the same color resolution. Alternatively, the images in the foreground can have one or more color resolutions. For example, the content 607 may have a color resolution of 1 bpp, while the content

608 and 609 have a different color resolution of 24 bpp. There are situations where the actual color to be associated with an area can be determined on the basis of less than a high color resolution. In yet another embodiment

thus, the content 608 may have a color resolution of 8 bpp while the content 609 may have a color resolution different from 1 bpp and 8 bpp (including, but not limited to, 2 bpp, 4 bpp, 16 bpp, 24 bpp, 32 bpp or equivalent ). Using the color resolution necessary to represent a range of colors (rather than a pre-selected resolution for an entire layer) enables better and/or more efficient compression by eliminating redundant color-specifying bits per pixel and/or faster compression and rendering performance.

[37] The ability to define the color resolution for each image or collection of images in a layer also applies to the background layer.

[38] Figure 6 shows another representation of an MRC file. Here, the image background 610 is combined with "color text C" 611 and "black text" 612 to form a composite image 613. Each image 610, 611 and 612 has a size smaller than the size of the composite image. In this alternative example, the combined sum of images 610, 611 and 612 is less than the size of the composite image.

[39] Figure 7A shows an exemplary process for creating a file with mixed raster content according to aspects of the present invention. The process starts in step 701 from the image generating program. First, text and line drawings are processed in step 702. If there are text and/or line drawings, they are added to the selector layer in step 703. The color of the lines or text is used in step 704 to create a color range for the foreground. The color area can be rectangular or any other shape necessary to delimit the area in the foreground layer to sufficiently specify the color of the text and/or lines from step 702. Next, the images in the image file are processed in step 705, and added in the background layer in step 706.

[40] The example in Figure 7A shows the possibility of processing images directly from a program, which is sent to a device driver, or from a queued metafile. In contrast, the conventional method of generating the MRC file is to take an intact composite image and divide it into different components in a separation process 303. The separation process 303 is error prone and consumes significant resources to perform the image decomposition process. However, the example in Figure 7A allows for generating MRC files without the separation process, as the example in Figure 7A handles the different components of an image while the different components can still be accessed separately.

[41] Figure 7B shows an alternative process for generating an MRC file. In this case, when a graphics engine starts the MRC generation process in step 701, the graphics function calls in step 708 are listed with the appropriate image regions. The list may include one or more of the following for each image area: Boundary area;

The type of graphics to be produced; and

The color used.

[42] Then, in step 709, if the image area is text or line drawings, the image area is added to the selector layer in step 703.1 otherwise, in step 710, the image area is added to the foreground/background layer. Any type of image is added to the foreground/background layer so that the foreground/background layer is a good representation of the document.

[43] Figures 8A and 8B show different possibilities for step 704 of Figure 7A. In Figure 8A, the font or brush color for text and images, respectively, is determined in step 801 and sent to step 803. Further, in step 802, the bounding area(s) (which may be rectangles or other shapes that define the bounding of the text and /or the dashes), and is sent to step 803. Steps 801 and 802 can be performed in any order, or can also be performed simultaneously. It is further understood that the fields determined in step 802 need not be completely filled. For example, a line drawing of a circle can have a bounding area in the form of a torus. In step 803, the area or areas identified in step 802 are painted as filled areas on the foreground layer.

[44] Figure 8B shows another alternative for step 704. First, each pixel in the selector layer 804 is scanned. If the pixel is black, which is determined in step 805, the color of the background pixel is used to fill in the color of a corresponding foreground pixel in step 806. The scanning step 804 can include every pixel in the entire selector layer.

[45] Alternatively, as the selector layer may be divided into images containing content, step 804 may scan only the images in the selector layer. Since the area of the selector layer not occupied by images will not contain content, these areas need not be scanned in step 804.

[46] In another aspect of the invention, the format of the MRC file is modified.

Figure 9A shows an exemplary process for rendering a composite image from a file with mixed raster content from Figure 6.1 step 901 the system begins to construct a composite image from the various MRC layers. For all the pixels in the composite image, it is determined in step 902 whether the pixel is in a selector layer. If so, the system then determines in step 905 whether the pixel in the selector layer is white. If the pixel is white, the system checks in step 906 whether there is a corresponding pixel in the background layer. If there is no such pixel in the background layer, the default background color is used in step 909, otherwise the color of the corresponding pixel in the background layer is used in step 910.

[47] If the pixel in the selector layer is not white in step 905, the system checks in step 907 whether there is a corresponding pixel in the foreground layer. If there is no such pixel in the foreground layer, the color of the pixel in the selector layer is used in step 908. If in step 907 there is a pixel in the foreground layer, the color of the corresponding pixel in the foreground layer is used in step 904.

[48] If there is no pixel in the selector layer in step 902, the system determines whether there is such a pixel in the foreground layer in step 903. If there is a pixel in the foreground layer, the system applies in step 904 the color of the corresponding pixel in the foreground layer. Otherwise, the system determines in step 906 whether a pixel exists in the background layer, and continues as described above.

[49] If there was no pixel in the selector layer as determined in step 902, the system determines whether there is a corresponding pixel in the foreground layer in step 903. If not, the color of the corresponding pixel in the background layer is used (step 906). If there is no corresponding pixel in the image background, a standard pixel color can be used (for example white or grey, or it can possibly be marked as transparent). If the result of step 903 is yes, the color from the corresponding pixel in the foreground layer is used (step 904).

[50] Figure 9B shows a process for combining the multiple layers into a composite image. Initially in step 901, the system obtains the bounding rectangles, or other shapes, for the contents of the layers. The system then creates four lists. The first list (NOL FGBG) represents areas where the foreground does not overlap with the selection layer's areas and areas where the background does not overlap with the foreground's areas. The second list (SEL) represents areas where the selector layer does not overlap with the foreground layer. The third list (SELFG) represents areas where only the selector layer and the foreground layer overlap. The fourth list (BGFGSEL) represents areas where the background, selector, and foreground layers all overlap.

[51] Then, in step 913, the composite image is filled with a standard background color. Each list is processed in steps 914-917.1 step 914 fills each area in the first list (NOL_FGBG), when there is a foreground layer, with the image foreground. Otherwise, the area is filled with the image background.

[52] In step 915, each area in the second list (SEL) is processed. If the pixel in the selector layer is white, no modifications are made to the composite image. Otherwise, the color of the pixel in the selection layer is used.

[53] In step 916, for each area in the third list (SELFG), if the pixel in the selector layer is white, no modifications are made to the composite image. Otherwise, the color of the pixel in the foreground layer is used.

[54] In step 917, for each area in the fourth list (BGFGSEL), if the pixel in the selector layer is white, the background color is used. Otherwise, the color of the pixel in the foreground layer is used.

[55] Figure 10 shows another exemplary process for creating a file with mixed raster content. The process illustrated in Figure 10 produces the MRC file in Figure 6. The process begins at step 1001 where it is initialized. In one example, step 1000 may be receiving a command to print the MRC file. Since step 1000 may be part of an application, part of a graphics toolbox called by an application, or may be any number of other mechanisms for initiating creation of the MRC file, step 1000 is shown in a box with a dashed border.

[56] For the sake of simplicity, the following is described in connection with creating MRC files in a printer driver. The following includes reference to the GDI API, available in the WINDOWS® operating system from Microsoft Corporation, which includes various function calls for image manipulation. One must understand that other operating systems, such as LINUX and APPLE-based systems, will include variations of the different function calls. Accordingly, as all types of operating systems and printer drivers are intended to be covered by the scope of this specification, the references to GDI function calls are not intended to be limiting, and are provided for explanatory purposes only.

[57] To construct a printer driver using GDI, a number of function calls are available. The following describes a number of function calls that help separate content into different layers. The function calls can be generally described as DrvXXX GDI API functions. Specifically, they include:

• DrvCopyBits

• DrvBitBIt

• DrvStretchBIT

• DrvStrokePath

• DrvFillPath

• DrvStrokeAndFillPath

• DrvLineTo

• DrvTextOut

• DrvAlphaBlend

• DrvGradientFill

• DrvPlgBIT

• DrvStretchBltROP

• DrvTransparentBIT

[58] These function calls are divided into foreground generating functions and background generating functions as follows:

Foreground generating functions

• DrvStrokePath

DrvLineTo

DrvTextOut

DrvBitBit

Background generating functions

DrvCopyBits

DrvBitBit

DrvStretchBIT

DrvAlphaBlend

DrvGradientFill

DrvPlgBIT

DrvStretchBltROP

DrvTransparentBIT

DrvFillPath

DrvStrokeAndFillPath

[59] One understands that graphics renderers change over time. Accordingly, the above list is provided as an example only. It is understood that other function calls can be used or will be known to the person skilled in the art, and these are considered to be within the scope of graphics functions as shown above.

[60] For explanatory purposes, the printing process can be divided into two steps. These two steps are represented by the left and right halves of Figure 10. During the first step, two bitmap or bitmap files are created in step 1001. The two files are the selector layer and a combined background/foreground layer. The combination layer can be referred to as BFGG_SURFACE and the selector layer as SELSURFACE. Both bitmap files cover the entire page. In step 1002, the system determines whether one of the received function calls was a foreground generating DrvXXX call. Alternatively, the FillPath and StrokeAndFillPath function calls, unlike the BitBlt function calls, can be used when a brush is a solid color and applies different raster operations. If yes, in step 1004 it renders both BFGG_SURFACE and SEL_SURFACE by applying a corresponding EngXXX function call (which is part of the GDI API) to BFGG_SURFACE and SEL SURFACE separately. If the function call was a background generating DrvXXX function call (represented as "no" from determination step 1002), it renders BFGG_SURFACE using a corresponding EngXXX function call from the GDI toolbox in step 1006.

[61] Furthermore, for each DrvXXX function call, a data node is created that contains at least part of the following information: • Bounding rectangle or area that can be affected by the DrvXXX function call. • The color value of the text for the DrvTextOut function call and for line drawings.

• An ID to identify the type of DrvXXX function call.

[62] In addition, each node is added to a list of background elements (BGEJJST) or a list of foreground elements (FGEJJST) for processing in the second step. Process steps 1003 and 1005 show that the node can be brought into each list. However, since these steps can be performed before or after steps 1004 and 1006, steps 1003 and 1005 are shown with dotted frames.

[63] BFGG_SURFACE can have the same resolution as the highest resolution to be used for the background and foreground images. Alternatively, the resolution of BGFG_SURFACE can be set to the same resolution as the page resolution in the final image.

[64] The second step begins after all DrvXXX function calls are processed. At this stage, images have been generated for the three layers. In step 1007, the process determines whether the images in the background/foreground list overlap. If yes, the images are processed and added to the images to be combined by the MRC process(es). If the images do not overlap, the process attempts to add the images to the selector list in step 1009. To determine where a selector image is needed, an intersection test is performed for each image in BGEJJST with each image in FGEJJST. All areas (rectangular or not) where foreground elements and background elements intersect are handled by a selector image.

[65] In one embodiment, the process attempts to move as much of the images in the selector layer as possible into the foreground layer. In another embodiment, the process attempts to minimize the sum of the areas of all the images to reduce the compressed size and increase the performance of the compression, decompression and rendering steps. In a further embodiment, the process attempts to have the lowest possible total number of images in all layers. These embodiments can be combined or used separately. They are described together for explanatory purposes.

[66] In step 1010, the process attempts to reduce the number of elements or images per layer. For example, two non-overlapping text images of the same color can be combined.

[67] In step 1010, the process attempts to reduce the number of elements in the layers. In this step, elements from the same layer are combined to make larger elements large. It should be noted that this step will be able to introduce new items into the lists. The following lists different methods of combining the items. One or more of these can be used to determine how elements in the layers should be grouped: • The combined element should avoid creating new overlaps with elements in other layers, if this does not provide advantages in terms of other aspects such as

are listed below;

• The area of the combined element should not be much greater than the sum of the individual elements. A threshold value can be set (for example 30% - 40%) against which a comparison is made to determine when a combined

eement becomes too large; and

• Rendering-specific function calls may have qualifiers (for example, foreground elements created using DrvTextOut should not be combined with elements created using other DrvXXX function calls because they may have different BPP values. Furthermore, in foreground elements created using DrvTextOut, only elements of the same color can be combined).

[68] After it has been decided how the page should be decomposed, there are three lists: • BGEJJST;

• SLEJJST; and

• FGEJJST.

[69] Each list contains a list of nodes that describe the area (eg a rectangle or other shape) to be cut from the page. In a further embodiment, the lists can include a scale factor that describes how the image in the list should be scaled in the final image. The scale factor makes it possible to store each image element at a different resolution, even within the same layer.

[70] Finally, the various layers are rendered in step 1011. For each element in the list of background elements that do not cut selector images, an image can be created by cutting out an area from BFGG_SURFACE and placing it in the background layer. For each element in the list of foreground elements that don't clip selector images, an image can be created by clipping an area from SEL_SURFACE and adding it to the foreground layer. For elements in the background and foreground list that intersect selector images, images are created for each layer using both SEL_SURFACE and BFGG_ SURFACE.

[71] For example, also referring to the clipping/pixel-by-pixel comparison capability of Figure 9B, with a background element intersecting the selector image, the image background can be constructed by clipping an area from the BGFG_ SURFACE. Then, for each pixel in the image background, the process checks the corresponding pixel in SEL_ SURFACE. If the pixel in SEL_ SURFACE is black, the pixel value in the image background is replaced with a nearby background pixel whose corresponding pixel in SEL_ SURFACE is white. For a foreground element that intersects the selector image, the image foreground can be constructed by clipping an area from BFGG_ SURFACE. Then, for each pixel in the image foreground, the process checks the corresponding pixel in SEL_ SURFACE. If the pixel in SEL_ SURFACE is white, the pixel value in the image background is replaced with a nearby foreground pixel whose corresponding pixel in SEL_ SURFACE is black. One advantage of moving images to the selector layer and/or reducing the number of images per layer is that the resulting layers can be compressed with better efficiency.

[72] As mentioned above, different compression techniques can be used to compress the different images. For example, one can compress each image that is 1 bpp using G4 compression. Each image that is 24 bpp can be compressed using JPEG compression, which is well known to those skilled in the art.

[73] Figure 11 shows a process for merging images. The images can be processed together, or can be divided into groups (hereinafter referred to as "image sets"). The images in the image sets can be grouped by image type. Alternatively, the image sets can be grouped in the order in which they were created. The image sets can have a predetermined maximum number of images, but this is not necessary.

[74] In figure 11, image set comprises 1 N images. The images are arranged in order of how close they are to each other. Proximity is described below in connection with Figure 13. Proximity can be considered the physical distance between the images, or can describe how close the images are to each other for merging purposes. If two images overlap, their mutual proximity value is equal to "0", which means that they have the lowest proximity value and thus should be merged. Image pairs that should not be merged can be given a proximity value of 1 or higher. It is understood that any scale can be used to describe closeness.

[75] All the images in image set 1 are compared with each other, and each pair of images is ranked according to their mutual proximity value. If N = 5 in Figure 11, the image pairs will be as follows:

Picture 1, Picture 2

Picture 1, Picture 3

Picture 1, Picture 4

Picture 1, Picture 5

Picture 2, Picture 3

Picture 2, Picture 4

Picture 2, Picture 5

Picture 3, Picture 4

Image 3, Image 5

Image 4, Image 5

[76] The proximity value for each pair is determined, and the pairs are arranged as shown in figure 11 for image set 1 on the left side of the figure.

[77] In step 1101, for the first image pair, the process determines whether the proximity value is lower than a predetermined threshold value. If so, the images are merged in step 1102 and the proximity value is recomputed for the remaining images. If the proximity value in step 1101 is higher than the threshold value, merging can be stopped in step 1104. For example, the threshold value can be set to any value between the lowest proximity value (which means that the images should be merged) and the highest proximity value (which means that the images should not be braided). The range of the threshold value can be 0.4 - 0.6. It is noted that the threshold value may be higher or lower than this range, depending on the type of images, the relationship between the types of images, and other factors.

[78] As described above, the process attempts to merge images that have a low mutual proximity value. Another advantage can be obtained by minimizing the number of resulting images that the MRC operation must process. In this case, although the images do not necessarily have a mutual proximity value lower than the threshold value used in 1101, a desired number of images per image set may be specified to minimize the number of images that must be processed. Step 1103 represents a determination of whether the number of images in the image set is lower than the desired number of images in the image set. For example, the desired number of images may be 4. Thus, although the mutual proximity value of any pair of images may be high, the images may still be merged (step 1102) to reduce the number of resulting images, and the proximity value of the resulting image pairs is calculated again in step 1105.

[79] Figure 12 shows an example of the merge function running through pairs of images as well as the resulting images. The individual images are represented by image areas 1 - N, respectively numbered 1201 - 1205. The merging process 1206 runs over images 1201 - 1205. Merging any pair of images can be a simple merging process (process 1207) where the resulting image (image area A, which is a single image and is treated as an image background) does not result in MRC images. Alternatively, the resulting interlacing may be a complex interlacing with MRC as shown in process 1208. There are at least two types of interlaced results from the MRC interlacing 1208. The first 1211 is an image area C where background and selector layers have been produced. The third 1212 is an image area D where background, selector and foreground layers have been produced. This order of a single (background) image, MRC with background and selector layers, and MRC with background, selector and foreground layers is also the preferred one when determining the type of merging to be performed. That is, one would rather produce an MRC file with a single background layer than an MRC file with background, selector and foreground layers.

[80] Figure 13 shows a process for determining the proximity value for two arbitrary image areas. Two image areas A, 1301, and B, 1302, are checked for overlap in step 1303. If the images overlap, the proximity value is set to 0 in step 1307. If not, the process determines in step 1304 whether both images are MRC images. If both images are MRC images, the proximity value for the images (step 1308) is set to a value between 0 and a high value (the high value here being 1). This intermediate value can be a fixed value for all cases (eg 0.5). Alternatively, the value can be set as a ratio. In one example, the ratio may be the efficiency ratio of combining the image areas. In other words, the ratio can be the unused space in the merged image divided by the total area of the combined images. This relationship is shown in the form of an equation in step 1308.

[81] Figure 14 shows an example of determining the unused space. In Figure 14, the text image A, 1401, is considered for combination with the text image B, 1402. In this example, the bounding areas 1401 and 1402 are shown as rectangles. The combined image area from the two areas 1401 and 1402 results in an unused space 1403 which is shown delimited by a dashed line. In an alternative embodiment, the unused space can be calculated based on a rectangular grid, resulting in the total area comprising 1401, 1402, the area 1403 enclosed by the dashed lines, as well as the outlying areas 1404 and 1405. The first example includes the unused the square area 1403.1 the second example includes the unused square area 1403, 1404 and 1405.

[82] Referring back to Figure 13, if both images at the relevant time are not MRC images, the process determines in step 1305 whether both images are text/line images. If this is the case, and if the images have the same color (step 1306), the proximity value is set in step 1308 as described above.

[83] If the text/line mappings do not have the same color, it is necessary to apply MRC, so that the proximity value is set high (e.g. 1) as shown in step 1309. Alternatively, the proximity value can be set high in the manner shown in step 1310, where the proximity value is set to a coefficient multiplied by the new MRC area created by merging the two images divided by the total area of the combined images.

[84] If the outcome of step 1305 is no, the proximity value is set in step 1310. Alternatively, the proximity value can be set as shown in step 1309.

[85] Once the merged files are created, an indication of how much space is occupied by the original images can be stored in the combined images. There are different methods for calculating the space for interlaced images. Figure 15A shows three images (1-3) each having a size of three units. It is understood that any unit of area measurement may be used, including square centimeters, square millimeters, square inches, pixels and the like. Image 1 overlaps with image 3, and image 2 overlaps with image 3.1 Figure 15B shows 1501 the combination of images 1 and 2. After merging the combined image 1+3 with image 2, the resulting combination is represented by 1502.1 1502 is the number actually used units equal to 7. However, this number can be approximated by taking the combination of images 1 and 3 (1502) and adding the size of image 2.1504, giving an approximate size of 8.

[86] Figure 15C shows yet another representation. Here, images 1 and 2 are first combined, resulting in image 1505. Adding image 3 results in image 1506, which also has a size of 7 units as the center unit in image 2 and the lower unit in image 3 are not counted separately, but together. Alternatively, this value for the size of the occupied space in the combined image may include 1507 for images 1 and 2 and image 3, 1508, giving a total of 9 units. In yet another embodiment, the size of the interlaced image may be replaced by the total size of the combined image.

[87] Although exemplary systems and methods comprising the present invention are shown as examples, it is of course understood that the invention is not limited to these embodiments. The person skilled in the art can make modifications, especially in light of the foregoing ideas. For example, each of the elements of the above-described embodiments can be used alone or in combination with elements from the other embodiments. Although the invention is defined by application of the subsequent patent claims, these claims are exemplary in that the invention is intended to include the elements and steps described here in any combination or partial combination. Accordingly, there are any number of alternative combinations for defining the invention, incorporating one or more elements from the specification, including the description, claims and figures, in different combinations or sub-combinations. It will be obvious to those skilled in the art, in light of the present specification, that alternative combinations of aspects of the invention, either alone or in combination with one or more elements or steps defined herein, may be used as modifications or rearrangements of the invention or as a part of the invention. It is intended that the written description of the invention provided herein shall cover all such modifications and rearrangements.

Claims (16)

1. Process for generating a mixed raster content file, comprising the steps of: processing text or line art; adding said text or line art to a selector layer; processing a list of images to form a set of foreground, selector, and background images; applying a color to a font of said text or a color to said line and adding an area of said color to said font of said text or said color to said line in a foreground layer; and add images to a background layer. 2. Process for rendering a file of mixed raster content into a composite image, comprising the steps of: determining whether a pixel exists in a selector layer; determining whether said pixel is white; determining whether said pixel exists in a foreground layer; and applying a color in said pixel from said foreground layer to color a corresponding pixel in said composite image when said decisions determined that said pixel is found in said selector layer, that said pixel is not white in said selector layer and that said pixel is found in said foreground layer. 3. Process according to claim 2, further comprising the steps of: if it is determined that a pixel is not found in said selector layer, then determine whether a corresponding pixel exists in said foreground layer; and applying a color in said corresponding pixel from said foreground layer to color a corresponding pixel in said composite image when said decisions determine that said pixel is found in said foreground layer, otherwise apply a color in a corresponding pixel in a background layer. 4. Computer-readable medium, wherein said medium comprises a data structure with mixed raster content that can be rendered into a composite image, wherein said data structure with mixed raster content comprises: a background layer; a foreground layer; and a selector layer, said selector layer comprising an image which is smaller than said composite image. 5. Computer-readable medium, wherein said medium comprises a data structure with mixed raster content comprising: a background layer; a foreground layer; and a selector layer, said foreground layer comprising one or more images with color palettes. 6. Computer-readable medium according to claim 5, wherein said one or more images comprise two or more images that have different color resolutions. 7. Computer-readable medium in which a computer program is stored, wherein said computer program generates a file with mixed raster content, said computer program performing the steps of: processing text or line drawings; adding said text or line art to a selector layer; applying a color to a font of said text or a color to said stroke and adding an area of said color to said font of said text or said color to said stroke in a foreground layer, and adding images to a background layer. 8. Computer-readable medium in which a computer program is stored, wherein said computer program renders a file with mixed raster content into a composite image, said program performing the steps of: determining whether a pixel exists in a selector layer; determining whether said pixel is white; and applying a color in said pixel from said foreground layer to color a corresponding pixel in said composite image when said decision step determines that said pixel is found in said selector layer, that said pixel is not white in said selector layer and that said pixel is found in said foreground layer. 9. Computer-readable medium according to claim 8, where said program further performs the steps of: if a pixel is not found in said selector layer, then determine whether a corresponding pixel exists in said foreground layer; and applying a color in said corresponding pixel from said foreground layer to color a corresponding pixel in said composite image when said decision step determines that said pixel exists in said foreground layer, otherwise apply a color in a corresponding pixel in a background layer. 10. Computer-readable medium in which a data structure is stored, said data structure having a mixed raster content comprising: a selector layer having a color resolution of one bit per pixel, a color being associated with the selector layer and the selector layer representing color text . 11. Process for rendering images, comprising the steps of: determining a bounding area for each image; generating at least two lists, where said first list represents the areas where there is no overlap between images and said second list represents the areas where there is overlap; render the non-overlapping areas area by area; and render the overlapping areas pixel by pixel. 12. Process for rendering composite images, comprising the steps of: determining a bounding area for each image; generate four lists, where said first list represents the areas where there is no overlap between images, where said second list represents those areas where the selector layer's areas do not overlap with the foreground layer's areas, where said third list represents the areas where only the selector layer's and foreground layer's areas overlap and where the aforementioned fourth list represents the areas where the areas of the background layer, selector layer and foreground layer overlap. 13. Process according to claim 12, further comprising the step of filling said composite image with said areas in the foreground layer from said first list when said areas in the foreground layer are found in said first list, otherwise filling said composite image with areas in the background layer. 14. Process according to claim 12, further comprising the step of filling said composite image with said area in the foreground layer from said second list when said second list contains a non-white area in the selector layer. 15. Process according to claim 12, further comprising the step of filling said composite image with said area in the selector layer from said third list when said third list contains a non-white area in the selector layer, otherwise filling said composite image with said area in the foreground layer. 16. Process according to claim 12, further comprising the step of filling said composite image with an area in the foreground layer when said area in the selector layer is not white, otherwise filling said composite image with said area in the background layer.