AU2004242418A1 - Generating compressed output representation - Google Patents

Description

S&F Ref: 690186

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT Name and Address of Applicant: Actual Inventor(s): Address for Service: Invention Title: Canon Kabushiki Kaisha, of 30-2, Shimomaruko 3-chome, Ohta-ku, Tokyo, 146, Japan Michael Gould James Austin Besley Michael Jan Lawther Spruson Ferguson St Martins Tower Level 31 Market Street Sydney NSW 2000 (CCN 3710000177) Generating compressed output representation The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5845c -1- SGENERATING COMPRESSED OUTPUT REPRESENTATION

(N

0 COPYRIGHT NOTICE This patent specification contains material-that is subject to copyright protection. The copyright owner has no objection to the reproduction of this patent specification or related materials from associated patent office files for the purposes of review, but otherwise reserves all copyright whatsoever.

TECHNICAL FIELD The present invention relates generally to the field of digital image processing and in particular to inpainting and compressing digital images.

BACKGROUND

Inpainting is a technique of removing objects from images by painting over the objects with an estimated background colour or texture. Such techniques have applicability to a wide variety of fields. A photographer may use inpainting techniques for retouching photographs. A motion picture studio may use inpainting techniques for removing safety harnesses from images of stunt performers.

The proliferation of scanning technology combined with ever increasing computational processing power has lead to many advances in the area of document analysis systems.

These systems may be used to extract semantic information from a scanned document, often by means of OCR technology. This technology is used in a growing number of applications such as automated form reading and can also be used to improve compression of a document by selectively using an appropriate compression method depending on the content of each part of the page. Improved document compression lends itself to applications such as archiving and electronic distribution.

In document compression systems that segment the image into background and foreground planes, the high detail elements like text are typically stored in a foreground 690186_final S-2- _plane. These foreground elements are removed from the background plane by inpainting to improve the compressibility of the background and reduce unwanted artefacts that might otherwise appear when a lossy compression method is used.

00 5 Previously, various techniques have been used. One technique is to paint out N foreground elements with a uniform colour found by averaging the colours of several of the surrounding pixels. A similar process can also be done in raster order using either a predetermined constant colour or the last known correct background colour.

Inpainting in this way leaves a sharp discontinuity in places around the border of the inpainted area causing increased image size and visible artefacts. These artefacts can be further emphasised when the image is compressed with a lossy compression scheme like JPEG. When additional areas around foreground elements are painted, an unwanted 'halo' artefact can also be produced.

Other techniques attempt to generate a smooth function over the area covered by the foreground components, which matches the colours of pixels around the edge. One class of such techniques recursively erodes the edges of the foreground elements painting with the colour of the nearest non-foreground pixel. Other classes fill the region with a basis function, the parameters of which are selected to give a good compromise between the smoothness of the function and the fit to the edge pixel colours. These techniques produce fewer artefacts than the previously mentioned techniques, but are complex to implement and computationally expensive.

Many systems use a tile-based method, such as JPEG, to compress the background image plane after inpainting, and the inpainting method may be tuned to improve the compression. Some of these techniques increase compression further by setting all the pixels in a tile to a constant colour if the level of background pixel variation is low.

690186_final c-3-

SUMMARY

N In accordance with a first aspect of the invention, there is provided a method of segmenting a digital image comprising a plurality ofpixels. The method comprises the 00 steps of: generating a plurality of blocks of pixels from the digital image; and producing at least one connected component for each block using the blocks of pixels in a one-pass manner. In turn, the producing step comprises: segmenting a block of 8 pixels into at least one connected component, each connected component comprising a c, group of pixels that are spatially connected and semantically related; merging the at least one connected component of the block with at least one connected component segmented from at least one other block that has been previously processed; and storing in a compact form a location in the image of the connected components of the block.

The semantically related pixels may comprise pixels that are similarly coloured.

The generating step may comprise the sub-steps of: arranging the digital image into a plurality of bands, each band comprising a predetermined number of consecutive lines ofpixels; and buffering and processing the bands one-by-one. In turn, the processing step comprises the sub-steps performed on each currently buffered band: arranging the current band into a plurality of blocks of pixels; and buffering and processing the blocks of the current band one-by-one for the producing step.

The storing sub-step may comprise storing M-1 binary bitmaps, where M-connected components are in a block, M being an integer.

The storing sub-step may comprise storing an index map.

The segmenting sub-step may comprise: estimating a number of representative colours for each block; quantizing each block to the representative colours; and forming connected components from each quantized block. The segmenting sub-step may further comprise merging a subset of the connected components that are formed. The 690186_final -4merging sub-step may comprise gathering statistics of the connected components. The N,1 statistics may comprise any one or more of bounding boxes, pixel count, border length, and average colour. The method may further comprise the step of removing the formed 00 connected components that are deemed to be noise. The noise may comprise connected components having a pixel count that is below a predefined threshold and a borderlength-to-pixel-count ratio above another predefined threshold. The merging step may Scomprise: merging connected components of a block with connected components of a block on the left and above; and updating the statistics of the merged connected components. The statistics may comprise any one of more of bounding boxes, pixel count, fill ratio, and average colour.

The estimating sub-step may comprise: forming a histogram related to a plurality of colour bins based on YUV data of pixels in each block; classifying each block based on histogram statistics; and merging bin colours to form the representative colours based on the block classification. The method may further comprising the step of forming an indexed map for each pixel in one pass. The quantizing step may comprise: quantizing non-empty bins to representative colours; creating a bin mapping to the representative colours; and remapping the indexed map to the representative colours using the bin mapping. Thee forming sub-step may comprise: deciding a luminance band based on Y value; deciding a colour column based on U and V value; accumulating the pixel colour to the mapped bin; and incrementing the pixel count of the mapped bin. The step of deciding a luminance band may further comprise luminance-band anti-aliasing.

The step of deciding a colour column may further compris colour column anti-aliasing/ The merging step may comprise the following sub-steps performed for each connected component in a current block touching the left and above border: finding a list of connected components that touch the current connected component along the common border; and deciding the best candidate to merge.

690186_final In accordance with another aspect of the invention, there is provided an apparatus comprising a processor and memory for segmenting a digital image comprising a 00 plurality of pixels in accordance with anyone of the aspects of the foregoing method.

C' 5 In accordance with still another aspect of the invention, there is provided a computer program product comprising a computer readable medium having recorded therein a O computer program for segmenting a digital image comprising a plurality of pixels in accordance with anyone of the aspects of the foregoing method.

In accordance with yet another aspect of the invention, there is provided a method of automatically producing a compact representation of a colour document. The method comprises the steps of: segmenting a digital image of a colour-document page into connected components in one-pass, block raster order; partitioning the digital image of the page into foreground and background images using layout analysis based on compact, connected-component statistics of the whole page; inpainting in one-pass block raster order at least one portion of the background image where at least one portion of the foreground image obscures the background image; and combining the foreground and background images to form a compact document.

The method may further comprise the step of downsampling the background image.

Still further, the method may further comprise the step of compressing the background image. The compressing step may involve lossy compression. Further, the method may comprise a different compressing of the lossy compressed background image.

In accordance with a further aspect of the invention, there is provided an apparatus comprising a processor and memory for automatically producing a compact representation of a colour document in accordance with anyone of the aspects of the foregoing method.

690186_final In accordance with still a further aspect of the invention, there is provided a computer program product comprising a computer readable medium having recorded therein a computer program for automatically producing a compact representation of a colour O0 document in accordance with anyone of the aspects of the foregoing method.

CK1 In accordance with another aspect of the invention, there is provided a method of analysing a digital image comprising a plurality of pixels. The method comprises the N steps of: segmenting the digital image into objects, where the segmentation is represented by more than two labels; providing a set of properties for each object; for a subset of the objects, using a measure of containment to determine if a parent-child relationship exists between adjacent objects sharing a boundary; forming groups of objects that share a common parent based on object properties; and classifying objects according to their properties and grouping.

The containment may be determined using bounding boxes around each object and information describing touching relationships between objects. An object contains another object if the two objects touch at a boundary and the bounding box of the object completely contains the bounding box of the other object.

The forming groups step may comprise: considering pairs of child objects from a list of children of a common parent; and determining using the object properties whether each pair should be grouped together. Only neighbouring objects from a list of child objects with the same parent may be considered for grouping. Objects may grouped based on bounding box and colour information.

A group of objects may be classified as text according to a test of text-like qualities of the objects within the group. The test for text-like qualities may comprise: identifying a single value for each object representing that a location of the object; forming a histogram of the values; and identifying text by a property of the histogram.

690186_final The method may further comprise the step of adding further objects to a text-classified group of objects according to their properties, but regardless of their parent-child properties.

00 In accordance with still another aspect of the invention, there is provided a method of analysing a digital image comprising a plurality of pixels of a document page. The method comprises the steps of: segmenting the digital image to form objects based on the image; forming groups of the objects; and determining if the groups of objects each represent text. The determining step comprises: identifying a single value for each object dependent on a location of the object on the page; forming a histogram of the values; and identifying text by a property of the histogram.

The property of the histogram may be the total number of objects in bins in the histogram that have more than a specified number of objects. Alternatively, the property may be the sum of the squares of counts in the histogram.

The single value for each object representing the location of the object may be an edge of a bounding box of the object.

In accordance with another aspect of the invention, there is provided an apparatus comprising a processor and a memory for analysing a digital image comprising a plurality of pixels in accordance with the method according to any one of the foregoing aspects.

In accordance with another aspect of the invention, there is provided a computer program product comprising a computer readable medium having recorded therein a computer program for analysing a digital image comprising a plurality of pixels in accordance with the method according to any one of the foregoing aspects.

690186_final -8- In accordance with yet another aspect of the invention, there is provided a method of inpainting a digital image comprising a plurality of pixels. The method comprises the steps of: generating a plurality of blocks of pixels from the digital image; and changing 00 in raster order pixel values of at least one run of pixels for at least one block. The S 5 changing step comprises the following sub-steps performed on each block: determining start and end pixels for a run of pixels in the block relating to an object, the run comprising adjacent pixels grouped together; modifying at least one pixel value of the object in said run dependent upon pixel values of pixels outside the run; and determining an activity measure for pixels that do not correspond to an object in the block; and changing all pixel values in each block having at least one run of pixels to a set value if the activity measure for the block is less than a predetermined threshold.

The method may further comprise the step of modifying at least one pixel value of a dilated object pixel outside the object dependent upon pixel values of pixels outside the dilated object.

The generating step may comprise the sub-steps of: arranging the digital image into a plurality of bands, each band comprising a predetermined number of consecutive lines of pixels; and buffering and processing the bands one-by-one. The processing step may comprise the following sub-steps performed on each currently buffered band: arranging the current band into a plurality of blocks of pixels; and buffering and processing the blocks of the current band one-by-one for the changing steps.

The run comprises adjacent pixels in a raster line of pixels of the block.

The method may further comprise the step of compressing each block using a blockbased compression method. The block-based compression method may be JPEG. The method may further comprise the step of further compressing the block-based compressed blocks using another compression technique.

690186_final ,1 -9- The at least one pixel value of the object may be modified dependent upon pixel values N, of pixels outside the object uses a value interpolated from pixel values to the left and C right of the run, or a value from a pixel to the left of the run.

oO C 5 All pixel values in each block having at least one run of pixels may be changed to an N average value of a previously processed block or an average value of visible pixels in the block.

The method may further comprise the step of, if an end of the run of dilated object pixels is not found, setting colour values of pixels to a colour value of a pixel that does not correspond to an object to the left of the run of dilated object pixels.

The pixel values may be colour values.

In accordance with a further aspect of the invention, there is provided an apparatus comprising a processor and memory for inpainting a digital image comprising a plurality of pixels in accordance with the method of any one of the foregoing aspects.

In accordance with a further aspect of the invention, there is provided a computer program product comprising a computer readable medium having recorded therein a computer program for inpainting a digital image comprising a plurality of pixels in accordance with the method of any one of the foregoing aspects.

In accordance with another aspect of the invention, there is provided a method of changing pixels values of a digital image comprising a plurality of pixels, at least a portion of the pixels corresponding to an object in the image, the method comprising the steps of: arranging the digital image into a plurality of bands, each band comprising a predetermined number of consecutive lines of pixels; and buffering and processing the bands one by one in turn. The processing step comprises the following sub-steps for each currently buffered band: arranging the current band into a plurality of blocks 690186 final c ofpixels; and processing the blocks one by one in turn. The block processing step comprises the following sub-steps for each block: determining an activity measure for pixels in the block that do not correspond to objects in the image; if the activity oO measure is less than a predetermined threshold, changing pixel values of all pixels in C 5 the block to a pixel value; compressing the block with JPEG; and compressing the N, JPEG compressed block using another compression method.

N The bands each may comprise 16 lines of pixels of the digital image, the blocks comprise 16x16 pixels, and the compressing steps are performed in a pipeline manner.

The step of changing colour values of all pixels in the block may comprise setting the colour values of the pixels to colour values obtained by linearly interpolating between pixels that do not correspond to objects immediately to the left and right of a run of dilated object pixels, the dilated object pixels being pixels outside and adjacent to the run.

The method may further comprise the step of, setting colour values of pixels in the block to an average colour value of pixels that do not correspond to an object in the block.

The method may further comprise the step of, setting colour values of pixels in the block to an average colour of a preceding block.

Pixels of a dilated object may be determined by dilating a mask defining a location of the object.

The other compression method may comprise ZLIB.

The pixel values may be colour values.

690186_final 'q -11-

U

SIn accordance with still another aspect of the invention, there is provided an apparatus comprising a processor and memory for changing pixels values of a digital image oO comprising a plurality of pixels, at least a portion of the pixels corresponding to an object in the image, in accordance with the method of any one of the foregoing aspects.

O In accordance with a further aspect of the invention, there is provided a computer c, program product comprising a computer readable medium having recorded therein a computer program for changing pixels values of a digital image comprising a plurality of pixels, at least a portion of the pixels corresponding to an object in the image, in accordance with the method of any one of the foregoing aspects.

BRIEF DESCRIPTION OF THE DRAWINGS A number of embodiments are described hereinafter with reference to the drawings in which: Fig. 1 is a high-level flow diagram providing an overview of segmenting, analysing, and compressing a digital image in accordance with an embodiment of the invention; Fig. 2 is a flow diagram of the colour segmentation step of Fig. 1; Fig. 3 is a detailed flow diagram of the step of obtaining a next pixel of Fig. 2; Fig. 4 is a flow diagram of the layout analysis step of Fig. 1 Fig. 5 is a flow diagram of the step of generating a compressed output image of Fig. 1; Fig. 6 is a block diagram of an image comprising the character and associated connected components; 690186_final -12- Fig. 7 is a block diagram of a general-purpose computer system with which embodiments of the invention may be practice; Fig. 8 is a block diagram of a system for producing a compressed output for a scanned input page in accordance with an embodiment of the invention; Fig. 9 is a detailed flow diagram of the colour connected component analysis and blob statistics step of Fig. Fig. 10 is a flow diagram of the colour segmentation step of Fig. 2; Fig. 11 is a detailed flow diagram of the colour quantisation step of Fig. Fig. 12 is a detailed flow diagram of the step of forming a new blob or growing an existing blob of Fig. 9; Fig. 13 is a flow diagram of the inter-tile merging step of Fig. Fig. 14 is a flow diagram of the step of processing a best blob and CC pair of Fig. 13; Fig. 15 is a block diagram illustrating an example of the blob merging process; Fig. 16 is a block diagram illustrating merging between a current tile and the states of two adjacent tiles; Fig. 17 is a table showing inter-tile merging conditions; Fig. 18 is an example of the outcome of a CC mapping process of Fig. 14; Fig. 19 is a flow diagram of the intra-tile merging step of Fig. 690186_final -13- Fig. 20 is a flow diagram of grouping CCs step of Fig. 4; Fig. 21 is a flow diagram of the checking groups of CCs step of Fig. 4; Fig. 22 is a flow diagram of the step of finding child CCs of a group of Fig. Fig. 23 is a flow diagram of the initial grouping step of Fig. Fig. 24 is a flow diagram of the checking alignment step of Fig. 21; Fig. 25(a) is an image of a simple example illustrating segmentation based on more than 2 quantisation levels; Figs. 25(b) and are images illustrating respective results of binary segmentation of the image of Fig. Fig. 26 is a graphical depiction of an area of a document containing two objects, the letters and Fig. 27(a) shows a histogram of the values of the bottoms of the bounding boxes for an image of Fig. 27(b); Fig. 27(b) shows a selection of irregularly arranged bounding boxes for parts segmented from an image.

Fig. 27(c) shows a corresponding histogram for the page of Fig. 27(d); Fig. 27(d) shows an arrangement of bounding boxes on a page for a text group; 690186_final -14- Fig. 28 is a block diagram of a colour blobbing system; Figs. 29(a) to 29(c) are images of the colour histogram output, including the original tile, an indexed map, and the palette where the upper and lower grey portions of the palette represent empty bins; Fig. 30 is a flow diagram of the step of inpainting a tile of Fig. Fig. 31 is a flow diagram of the step of forming a tile foreground bitmask of Fig. Fig. 32 is flow diagram of the step ofinpainting pixels and measuring tile activity of Fig. Fig. 33 is a flow diagram of the step of tile flattening of Fig. Fig. 34 is a block diagram of an example pixel in a background image tile and a corresponding pixels examined in a full-resolution foreground mask; Fig. 35 comprises plots of one-dimensional inpainting examples; Fig. 36 is a block diagram illustrating colour blob processing using an example of a 6x6 tile; Fig. 37 comprises images showing an example of blob merging; Fig. 38 comprises images showing an original tile with the merged blob output for comparison purposes; Fig. 39 is a block diagram illustrating merging blobs over tiles in raster order; 690186_final Fig. 40 is a flow diagram illustrating the step of forming a 2D histogram and a first palette of Fig. 11; Fig. 41 is a flow diagram illustrating the step of forming a second palette of Fig. 11; Fig. 42 is a flow diagram illustrating the step of generating a two-coloured palette of Fig. 41; Fig. 43 is a flow diagram illustrating the step of generating a multi-coloured palette of Fig. 41; Fig. 44 is a flow diagram illustrating the step of associating pixels with the second palette of Fig. 11; Fig. 45 is a detailed flow diagram illustrating the step of mapping a bi-level tile of Fig.

44; Fig. 46 is a detailed flow diagram illustrating the step of mapping a multi-coloured tile of Fig. 44; Fig. 47 is a flow diagram of the step of analysing a histogram and classifying a tile of Fig. 11; Fig. 48 is a table showing CC and blob candidate counts before and after updating for all operating conditions; Fig. 49 is a block diagram illustrating examples of merging blob and CC candidates; Fig. 50 is a flow diagram illustrating the step of post-merging processing of Fig. 690186_final C -16- Fig. 51 is a block diagram of a system in accordance with another embodiment of the c invention; 00 Fig. 52 is a block diagram of a colour segmentation module of Fig. 51; and Fig. 53 is a diagram illustrating a Delaunay triangulation and a Voronoi diagram of set O points.

DETAILED DESCRIPTION Methods, apparatuses, and computer program products are disclosed for processing and compressing a digital image. In the following description, numerous specific details, including particular lossless compression techniques, colour spaces, spatial resolutions, tile sizes, and the like are set forth. However, from this disclosure, it will be apparent to those skilled in the art that modifications and/or substitutions may be made without departing from the scope and spirit of the invention. In other circumstances, specific details may be omitted so as not to obscure the invention.

Where reference is made in any one or more of the accompanying drawings to steps and/or features, which have the same reference numerals, those steps and/or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

In the context of this specification, the word "comprising" has an open-ended, nonexclusive meaning: "including principally, but not necessarily solely", but neither "consisting essentially of' nor "consisting only of'. Variations of the word "comprising", such as "comprise" and "comprises", have corresponding meanings.

The contents of the detailed description are organised into sections as follows: 1. Overview 2. Colour Segmentation 690186_final c, -17-

U

2.1. Obtain Next Tile of Input Image 2.2. Form Colour Segmentation of Tile 2.2.1. Colour Quantisation 00 2.2.1.1. Form 2D Histogram and First Palette 1 5 2.2.1.2. Analyse Histogram and Classify Image 2.2.1.3. Form Second Palette S2.2.1.3.1. Generate 2-Coloured Palette 2.2.1.3.2. Generate Multi-Coloured Palette 2.2.1.4. Associate Pixels with Second Palette 2.2.1.4.1. Map Bi-Level Tile 2.2.1.4.2. Map Multi-Level Tile 2.2.2. Colour CC Analysis and Blob Statistics 2.2.2.1. Form Blobs 2.2.2.2. Example of Blob Merging 2.2.3. Intra-tile Merging 2.2.4. Inter-Tile Merging 2.2.4.1. Inter-Tile Merging Conditions 2.2.4.2. Inter-Tile Merging Examples 2.2.4.3. Process Best Blob and CC Pair 2.2.4.4 CC Mapping Outcome 2.2.5. Post Merging Processing 2.3. Segmentation Example 3. Layout Analysis 3.1. Grouping CCs 3.1.1. Finding Children for Parent CC 3.1.2. Initial Grouping 3.1.2.1. Grouping Test for Two CCs 3.2. Checking Groups 690186 final -18- 3.2.1. Check Alignment N 3.2.2. Alignment Example 4. Generating Compressed Output Image 00oO 4.1 Inpaint Tile 4.1.1. Form Tile Foreground Bitmask 4.1.2 Inpaint Pixels and Measure Tile Activity O 4.1.3. Inpainting Examples 4.1.4. Tile Flattening Hardware Embodiment 5.1 Colour Segmentation Module 6. Computer Implementation 7. Industrial Applicability The foregoing sections are described in detail hereinafter in the foregoing order.

1. Overview The first embodiment of the invention is a process running on a general-purpose computer. Fig. 1 provides high-level overview of the process 100 of segmenting, analysing, and compressing a digital image. The input to the process 100 is preferably an RGB image at a resolution of 300 dpi. However, images in other colour spaces may be input with appropriate modifications to the process 100. Likewise, images at different resolutions may be input. In step 110, colour segmentation of the image into Connected Components (CCs) is performed using a single pass over the input image.

Using one-pass processing, digital images can be processed rapidly, so that a high volume of high resolution images can be processed. Connected components are groups of similarly coloured pixels that are all connected (touching). For example, the pixels forming the body or stem of the letter printed in black ink on a white page form a connected component, the dot on the forms another CC, and any white pixels surrounding the letter as background form yet another CC. Fig. 6 depicts an image 600 comprising the character where black dots represent black pixels and white 690186_final -19- _dots represent white pixels. The stem of the CC 610 and the dot of the CC 612 C, are depicted. Also shown is another resulting CC 614 of blank pixels.

00 As part of the segmentation, compact information and statistics describing the CCs are N 5 calculated. The digital is downsampled and provided directly to step 130. In step 120, N a layout analysis is performed on the CCs using this compact CC information and statistics. The layout analysis determines the layout of the features on the page, which 1 includes for example text characters, paragraphs, tables, and images. In step 130, this layout information is used to create one or more foreground images. A foreground image is typically made up of text characters identified in step 120, and is preferably a binary image. The foreground images may be stored at the input resolution 300 dpi), while a background image may be stored at a lower resolution 150 dpi). The foreground elements are removed from the background image.

Both the foreground and background are then compressed using different techniques and stored in a compound image format. The compound format may be a PDF document, for example. Colour segmentation, layout analysis, and compressing a digital image are each described hereinafter in separate sections.

One application for the embodiments of the invention is to analyse raster pixel images from a scanner and extract as much high-level information as possible. From this information, a high-level description of the page can be produced. To this end, a system may be designed to run faster by doing pixel analysis in hardware. However, it will be apparent from the following description that the system may be implemented entirely in software, as well. Also, while a particular output format PDF is described hereinafter, variations may be made to the system to utilise other page description formats as the output format.

Fig. 8 is a high-level block diagram of a system 800 in accordance with an embodiment of the invention for producing a compressed or compact representation 850 of a scanned input document 810. The system 800 comprises a front end module 820, a 690186_final I- middle module 830, and a back end module 830. The front end module 820 is a tilebased front end that is preferably implemented in hardware, but may be a combination of an ASIC and software executed by an embedded processor. Tile raster order is a

O

tile-based processing method in which tiles are made available for processing from top- 5 to-bottom and left-to-right one at a time. This module performs colour segmentation N on the input image and provides the background image of a tile to the back end module 840 (indicated by an arrow). The front end module 820 performs a colour connectedcomponent analysis at the full page resolution (eg. 300 dpi). The front end module 820 produces connected components (CCs) also from the digital image 810, and the CCs are provided to the middle module, which performs layout analysis. This module may be implemented entirely in software. The front-end module 820 provides a downsampled image to module 840. The output of the middle module 830 is provided to the back end module 840, which performs tile-based inpainting and produces the compact representation of the digital image 810. Like the front end module 820, the back end module 840 may be implemented at least partially in hardware with software executed on an embedded processor and an ASIC. This compacted output may comprise a digital image at a lower spatial resolution and a foreground bitmap at a higher resolution.

The front end module 820 performs all the analysis work that involves examining each pixel and forms colour CCs of regions of semantically related pixels. The output from the front end module 820 is information about all the colour CCs on the page. The information for each CC includes the bounding box, average colour, touching list, and the number ofpixels. When implementing algorithms in hardware, for best performance, the algorithm should be bandwidth efficient. For image processing tasks, the algorithm does not have random access to an entire scanned page. Instead, the algorithm works on small tiles at a time.

2. Colour Sementation 690186_final -21- _Fig. 2 illustrates step 110 of Fig. 1 in greater detail. The process shown in Fig. 2 work N on tiles of the input image, which for example may be of size 32x32 pixels. The tiles may be processed in raster order that is, from left to right, and top to bottom. The oO first tile to be processed is the top-left tile of the input image, and the last tile to be processed is the bottom-right tile of the input image. This tiling is done for efficiency purposes.

In step 210, the next tile of the input image to be processed is obtained. Pointer information may be used to efficiently access each tile. Step 210 is described in more detail with reference to Fig. 3. The processing in steps 220, 230, 240 and 250 is confined to the pixels of the current tile. In step 220, dehalftoning is optionally performed on the current tile. For example, scanned input documents processed by this method may contain halftones from a printing process. Halftones may make the subsequent analysis difficult and may not compress well. Therefore, this step 220 may be used to detect and remove halftones. By way of example only, the following dehalftoning process may be practiced. The dehalftoning process may work on 16x16 tiles. Each input tile of 32x32 RGB pixels may be divided into four 16x16 tiles, and each is processed separately. Halftone detection may work on one colour channel (ie, R, G and B) at a time. If halftones are detected in any channel, the halftone removal may be performed on all channels. For detecting halftones, each pixel in the tile is quantised to four levels. The input colour channel value ranges from 0-255. The four levels are the ranges 0-63, 64-127, 128-191 and 192-255. The number of level changes between pixels that are next to each other can then be measured. This measurement may be done horizontally and vertically.

Because halftones are typically small dots, the detection requires that the number of changes be large and the number of horizontal changes to be similar to the number of vertical changes. This prevents changes in level caused by edges of text characters being detected as halftones. A threshold value for detection may be specified. If halftones are detected in a 16x16 tile, a spatial blur for example may be used to remove 690186_final -22the halftones. The halftone detector can also use information from previously analysed N tiles. For example, if the tiles touching the current one have halftones detected in the tiles, the current tile likely also contains halftones. When this information is used, the 00 threshold value may be adjusted to relax or tighten the halftone detection requirements.

C N In step 230, if the colour space of the current tile is not already in the YUV colour O space, then a conversion is performed to convert the pixels of the tile to YUV colour N,1 space. Therefore, this step is optional dependent upon the colour space of the input image. While the YUV colour space is used in this embodiment, other colour spaces could be practiced without departing from the scope and spirit of the invention. The conversion formula used to convert from RGB to YUV may be the same as that used in the Independent JPEG Group (UG) JPEG library, for example.

In step 240, a colour segmentation to form Connected Components is performed on the current tile, and compact information and statistics about the CCs in the tile are calculated. A colour CC comprises one or more semantically related blobs, spanning across one or more tiles. For example, semantically related blobs may be similarly coloured. A blob is a connected group ofpixels, within a single tile, having similar colour characteristics. Blobbing is a process of colour segmentation and forming connected component representations. Each CC has the following statistics: size in pixels, mean colour, binary mask, blob boundary length, and bounding box. Fig. illustrates this step in greater detail.

In step 250, the current tile is downsampled to form a corresponding part of a background image. For example, a box filter may be used to downsample by 2:1 in both dimensions, but other methodologies may be practiced without departing from the scope and spirit of the invention. In decision step 260, a check is made to determine if there are any more tiles remaining to be processed. If the result of step 260 is No (that is all the tiles in the image have been processed), processing terminates. However, if the result of step 260 is Yes, processing continues at step 210.

690186_final c- -23- N4 2.1 Obtain Next Tile of Input Image Fig. 3 illustrates in detail step 210 of Fig. 2. The process 210 works on bands of the oO input digital image. A band of the image is a number of consecutive image lines. The 5 height of each band may be the same as the height of a tile. Thus, for example, the first 4 32 lines of the image form a band, and in this case the first band of the image; the next 32 image lines form a next band, and so on. The width of each band may be the width of the input image. In decision step 310, a check is made to determine if another band needs to be read in. Another band needs to be read in when all the tiles of the current band have been processed. The other time another band needs to be read in is the first time that step 310 is performed, because at that point no bands have been read. If the result of step 310 is Yes, processing continues at step 320. Otherwise, processing continues at step 340.

In step 320, the next band of data is read from the input image, for example, from disk into an in-memory buffer. The memory buffer may be arranged to contain each line of pixels of the band in contiguous memory locations. Further, a record of the memory location of the start of each line is kept: that is, a pointer to each band row. In step 330, a variable tx that determines the tile to be accessed, the current tile, is initialised set to In step 340, row pointer information is updated to point at the current tile. Other processes that call the process 210 of Fig. 3 obtain a new tile by referencing the row pointer information updated in step 340. The row pointer information may comprise a pointer per tile row. Thus, there may be 32 row pointers per tile. A given row pointer points to the memory location of the first pixel of the tile in the given row.

The row pointer information may be updated by stepping each row pointer along (tile width tx) memory locations beyond the pointer to the start of each corresponding band line. In step 350, the variable tx is incremented by 1, so that the next time the process 210 of Fig. 3 is called another tile is input to the process.

2.2 Form Colour Segmentation of Tile 690186_final c-I -24- Colour blobbing is an image segmentation algorithm that works in tile raster order. A blob is a connected group of pixels of the same quantised labels within a single tile.

Each blob has the following statistics: size in pixels, mean colour, binary mask, blob 00 boundary length, and bounding box. Its goal is to segment a document image into a set Ci 5 of non-overlapping connected components, where each connected component contains Na connected set of semantically related pixels, e.g. the set of pixels in a particular text Sletter forms one connected component, the pixels in a section of image around the text C would form another connected component, etc.

Fig. 28 is a block diagram of a colour blobbing system 2800 that has four modules 2810, 2820, 2830, and 2840. The colour quantisation module 2810 receives an input colour tile 24-bit pixel values for the RGB colour space), determines the number of dominant colours in the tile, and quantises the tile according to the dominant colours.

The dominant colours and quantised tile are provided as input to a connected component and blob statistics module 2820 that performs an 8-way connected component analysis on the quantised tile in a single raster pass. Blob statistics such as the number of pixels, mean colour, blob boundary length and bounding box information are collected in the same raster pass. The blobs and statistics are provided as input to an intra-tile merging module 2830, which reduces the number of spurious and small blobs within a tile by merging blobs based on colour, size, and boundary statistics. The resulting blobs and statistics from this module 2830 are provided as input to an inter-tile merging module 2840 that groups blobs in the current tile with touching blobs in the adjacent tiles (left and above) according to blob statistics to form connected components as the output of the system 2800. This is described further with reference to the processing of Fig. Fig. 10 is a detailed flow diagram of step 240 for segmenting an image into colour CCs.

As part of the segmentation process, compact information and statistics describing the CCs are calculated. The segmentation process begins by receiving an input tile of pixels from step 230. In decision block 1005, a check is made to determine if the tile is 690186_final flat. If the tile is flat, processing continues at step 1040 to the inter-tile merging stage.

C,1 Otherwise, processing continues at step 1010. In step 1010, colour quantisation is performed on the tile. Colour quantisation finds the dominant colours within a tile oO without considering pixel geometry, using three principal steps: 1) colour reduction, 2) C 5 tile classification, and 3) finding dominant colours and quantisation. The dominant colours in the input tile are determined by a colour histogram method. A quantised tile Sof colour labels is created by quantising the input tile pixels according to the dominant N, colours. A dominant colour is one that a human viewer would perceive as significant visually within the tile. The algorithm suits a hardware (HW) implementation and is also reasonably fast with a software (SW) implementation.

Step 1020 performs an 8-way connected component analysis on the quantised tile in a single raster pass to form blobs. In step 1030, an intra-tile merging process is performed to reduce the number of spurious and small blobs within the tile by merging blobs based on colour, size and boundary information. In step 1040, an inter-tile merging is performed. Blobs identified in the quantised tile are compared with blobs identified in two previously processed tiles that are to the left of the current tile and above the current tile for merging into colour CCs. Thus, a colour CC comprises one or more similarly coloured blobs spanning across one or more tiles. As such, colour CCs have the same types of statistics as mentioned above for the blobs except for boundary information.

In step 1050, blobs in the current tile and the colour CCs that the blobs form are stored in a compact tile-state data structure. This tile state does not contain pixel data. The tile state only contains information required for merging newly created blobs to existing colour CCs. The inter-tile merging process 1040 can be performed with high memory efficiency because at any stage of the segmentation process only two or less tile states are required for merging with the current tile. Furthermore, step 1050 updates a touching list for each colour CC. A touching list describes which connected components are next to each other. The touching list is generated as part of the colour 690186_final -26- CC analysis in the front end. Step 240 in Fig. 2 generates the touching list. Processing N then terminates.

00 2.2.1 Colour Quantisation CI 5 The purpose of colour quantisation is to reduce the full colour input to a reduced set of colours for the preparation of connected component generation. To find dominant Scolours, each input pixel is examined once and a histogram is generated. The C, embodiments of the invention employ histograms that use luminance as the first dimension and combine the two chrominance components as the second dimension.

This is unlike a conventional colour histogram that divides the bins in three dimensions according to the axes of three colour components. The embodiments of the invention produce a compact histogram that helps finding good dominant colours easier. From the characteristics of the histogram, a tile is classified into three types flat, bi-level and multi-coloured. A palette is generated for the tile depending on the tile classification. After the palette is generated, each pixel is assigned with a quantised label according to the palette colour that the pixel maps to. The method is designed for high-speed processing and low-memory requirements. A flat tile has only one quantised label. A bi-level tile has two quantised labels. A multi-coloured tile has up to four quantised labels.

The colour quantisation step 1010 of Fig. 10 is further expanded upon in Fig. 11. A brief description of each step in Fig. 11 is given here and the detailed description for each step follows. In step 1110, a 2D histogram and an indexed map with the first palette are formed at the same time. Fig. 40 provides further details of step 1110. In step 1120, a tile classification is carried out from the statistics of the 2D histogram.

Fig. 47 provides further details of step 1120. In step 1130, the second palette is formed based on the tile classification. This involves condensing the first palette. Fig. 41 provides further details of step 1130. In step 1140, pixels are associated with the second palette. The indexed map is remapped to one of the second palette colours to generate a quantised tile with quantised labels. Processing then terminates.

690186_final (N -27-

U

N, 2.2.1.1 Form 2D Histogram and First Palette Fig. 40 illustrates the processing of step 1110 of Fig. 11 in greater detail. The input full 00 coloured tile is quantised to an indexed map in pixel raster order in one pass by a CN 5 predetermined mapping method. Fig. 29(a) shows an example of an input original tile and the resulting indexed map that is produced. The mapping may be configured for 32 O colour bins organised in 8 luminance bands and 4 colour columns. Each colour bin 1 may have a colour accumulator, a pixel counter and a registration ID which is set by the YUV value of the first pixel put into the bin. The predetermined mapping method may be changed depending on the stats of the 2D histogram. As a result, there is no predetermined colour for each bin and the order of pixel colour may affect the composition of the first palette. The average colour of each non-empty bin at the end makes up the first palette. Fig. 29(c) shows a palette that is produced, where the upper and lower grey portions represent empty bins.

In step 4010, a pixel with colour value (YUV) is obtained from the tile. The predetermined mapping is performed in step 4015 to map the pixel to a luminance band and colour bin binmapped). The predetermined mapping may be as follows: band Y 5, and column (IU REF_U| IV REF_VI) NORMALISING_FACTOR[band].

The chrominance value of grey may be used for REF_U and REF_V: (that is, REF_U 128 and REF_V 128 for 8 bit RBG input data). The NORMALISING_FACTOR for each band is pre-calculated using the chosen REF_U and REF_V for normalising each band into 4 bins from the RGB colour space. The NORMALISING_FACTOR can be generated using the pseudocode of Table 1.

690186_final tc -28- TABLE 1 Set maxdist to 0 00 for each r in 0 to 255 Sfor each g in 0 to 255 0 for each b in 0 to 255 c RGB2YUV(r, g, b); band c.y dist c.u- 1281 Ic.v- 1281; if (dist maxdist [band]) max_dist [band] dist; for each band in 0 to 7 NORMALISING_FACTOR [band] 1 max_dist [band] 4 Steps 4020 to 4025 perform an optional "band anti-alias" for bi-level tile outline enhancement. In step 4020, if "band anti-alias" is enabled and the difference in luminance between a mapped band and the band above or below does not exceed a specified threshold 16), a "band anti-alias" is performed in step 4025. Otherwise, processing continues at step 4035.

690186 final -29- In step 4025, a band anti-alias is performed. An attempt to find a close non-empty bin in the band above or below is carried out. The candidate bin is the one mapped by band 1 or band 1. In either of the two conditions below, the candidate bin replaces oO the mapped bin (binmapped): N N, 1. The candidate bin is not empty and its registration ID(Y) is less than 16 Saway from Y and bin mapped is empty.

2. Both the candidate bin and binmapped are not empty and Y is closer to the candidate bin's registration ID(Y) than the registration ID(Y) of binmapped.

Steps 4035 to 4055 carry out a "bin anti-alias" process. Step 4035 checks if the mapped bin (binmapped) is empty. If the mapped bin is not empty, step 4040 checks the mapping error as follows: max(IU registration ID(U)I, IV registration ID(V)I) MAX_BIN_ERROR[band], where MAXBIN_ERROR[band] is one eighth of the maxdist in each band as defined in the above pseudo code for generating the normalising factor.

If step 4035 returns false processing continues at step 4040. Otherwise, processing continues at step 4045. In decision step 4040, a check is made to determine if there is a mapping error exceeding a specified threshold which is maximum bin error for this band.

If the mapping error is within the threshold, the processing continues at step 4060.

Otherwise, step 4055 is carried out to find a closer bin. In step 4055, a search starts from column 0 and moves forward to column 3 in the mapped band. The search terminates when any of the following conditions is met: 1. An empty bin is found, and 690186_final c 2. A bin with a mapping error within the allowed threshold is found.

If the search of step 4055 terminates on condition 1, the (YUV) value is registered into 00 the empty bin,and the empty bin replaces bin_mapped. If both conditions fail, a bin N. 5 with the smallest mapping error replaces binmapped. Processing then continues at N, step 4060.

N Following the test in step 4035, if the mapped bin is empty, processing continues at step 4045. In decision step 4045, a check is made to determine a close non empty bin in the same band has been found. Step 4045 searches from column 0 to 3 trying to find a non-empty bin satisfying the mapping error threshold defined previously. If such a bin is found, the empty bin replaces bin_mapped in step 4052 and processing then continues at step 4052. Otherwise, if step 4045 returns false binmapped is registered with colour (YUV) value in step 4050 and processing continues at step 4060.

In step 4060, pixel colour (YUV) is accumulated in the mapped bin (binmapped) and the pixel count in binmapped is incremented. In step 4065, the location of binmapped is recorded for the current pixel. In step 4070, a check is made to determine if there are more pixels left in the tile. If the result is YES, processing continues at step 4010. Otherwise, it continues at step 4075 where the accumulated colour in each non-empty bin is divided by its pixel count. The average colour of each non-empty bin forms the first palette. Processing then terminates.

2.2.1.2. Analyse Histogram and Classify Image Tile classification is a method of finding the dominant colours within a tile. Based on the distribution and the colour variance in a palette, tiles are classified into three groups: flat, bi-level, and multi-coloured. Flat tiles have visually constant colour to human eyes and normally form a cluster in the 2D histogram. The flat palette has up to three colours and the colour variance is small. Bi-level tiles have two distinctive colours and normally line up vertically in the 2D histogram. The bi-level palette has 690186_final (c -31-

U

colours that span across a few luminance bands, but the colour variance in each C luminance band is small. Multi-coloured tiles usually spread over a large number of bins in the 2D histogram. The multi-coloured palette comprises tiles that fail the first 00 two tests.

C- Step 1120 of Fig. 11 analyses the bin distribution and colour characteristics in the 2D O histogram and classifies the tile accordingly. Tiles are classified into the three groups: CI flat, bi-level and multi-coloured. Fig. 47 illustrates the processing of step 1120 in greater detail. In step 4710, a flat tile test is carried out. If the result is Yes, the tile is classified as flat in step 4712. Otherwise, a second test in step 4720 is carried out to determine if the tile is bi-level. If the result for the bi-level test is Yes, the tile is classified as bi-level in step 4722. Otherwise, the tile is classified as multi-coloured in step 4724. Steps 4710 and 4720 are described in greater detail hereinafter.

Regarding step 4710, first LumRange is defined as the range between the highest and the lowest luminance bands at which bins are not all empty. For a tile to pass flat test, the tile has to satisfy all three conditions below: 1. Number of non empty bins 3, 2. LumRange 2, and 3. FlatColourVariance FLAT_COLOUR_VARIANCE, where FlatColourVariance is defined as the sum of the pixel count weighted Manhattan distances between the largest bin and the rest. The threshold parameter may be FLATCOLOUR Regarding step 4720 for a tile to pass bi-level test, the tile has to satisfy all three conditions below: 1. Number of non empty bins BILEVEL_MAX_BIN_CNT, 2. LumRange 2, and 3. MaxColourVariance BILEVEL COLOUR VARIANCE 690186_final S-32- N, MaxColourVariance is defined as max(ColourVariance[band]), where ColourVariance[band] is the sum of the pixel count weighted Manhattan distances 00 between the largest bin and the rest in a band. The parameter values may be C 5 BILEVEL MAX BIN CNT 16 and BILEVELCOLOUR_VARIANCE 2.2.1.3. Form Second Palette Fig. 41 illustrates the processing of step 1130 of Fig. 11 in greater detail. Step 4110 tests ifa tile is classified as flat. If the tile is, step 4120 forms a flat colour. This may be done by calculating the weighted average of the non-empty bins. If the test result in step 4110 is No, processing continues at step 4130 to test if the tile is classified as bi-.

level. If the test result is Yes, processing moves to step 4140. In step 4140, a twocoloured palette is generated. Otherwise, if step 4130 returns false, processing continues at step 4150. In step 4150, a multi-coloured palette is generated.

2.2.1.3.1. Generate 2-Coloured Palette Fig. 42 provides further details of generating a 2-coloured palette for a bi-level tile in step 4140. The aim is to find two contrasting colours to represent the image. Due to halftoning and registration errors in printing, the colours representing the original two contrast colours are normally polluted. As a result, the average colours of the foreground and background areas are not a good representation for the original image.

Excluding the colours in the transition area makes the image look sharper.

In step 4210, the darkest and the brightest colours are chosen to form the initial palette for dominant colours. In step 4220, the six most populated bins are used to generate to a list of bins. The top six bins are found from the palette according to pixel count. Step 4230 to step 4270 process the colours in the list sequentially. In step 4230, the next bin colour, C, from the list is obtained. Decision step 4240 tests if the colour C has been included in the initial palette or if the colour is too far from the two extremes. If the result is Yes, the colour is ignored and processing returns to step 4230 to get the next 690186_final C1- -33-

U

bin colour for processing. If the result in step 4240 is No, processing continues at step

C

1 4250. Decision step 4250 tests if the colour is suitable for merging to the initial palette.

A suitable colour for merging is a colour located close to any colour of the initial 00oO palette. If the test result from step 4250 is Yes, the colour is merged to one of the C 5 initial palette colours based on the closer Manhattan colour distance with weighted pixel count. The pixel count of C is added to the pixel count of the palette colour to O merge to. Processing continues at step 4270. If the result of step 4250 is No, the colour is ignored and the process moves to step 4270 to check if there are any unprocessed colours. If the test in step 4270 returns Yes, processing returns to step 4230. Otherwise, processing terminates.

2.2.1.3.2. Generate Multi-Coloured Palette Fig. 43 expands on step 4150 of Fig. 41, which generates a multi-coloured palette for a multi-coloured tile. In step 4310, the darkest and the brightest colours are chosen to form the initial palette as the initial dominant colours. In step 4320, a third colour is added to the palette if both conditions below are true: 1. LumRange THIRD_COLOUR_MIN_LD, and 2. LargestVar THIRD_COLOUR_MIN_VAR, LargestVar is defined as the largest Manhattan colour distance from the mean colour of the brightest and the darkest colours among the bins between the darkest and the brightest luminance bands. If the above test is true, the colour that generates LargestVar is added as the third initial palette colour. The threshold may be THIRDCOLOURMINLD 4 and THIRDCOLOUR_MIN_VAR In step 4330, the top most populated) 6 bins are added to a list of bins. Steps 4340 to 4395 process colours of the list sequentially. In step 4340, the next bin colour, C, from the list is obtained. Step 4350 tests if the colour has been included in the initial palette. If the result is Yes, the colour is ignored and processing returns to step 4340.

Otherwise, step 4360 tries to merge the colour to one of the palette colours. The colour 690186_final c- -34-

U

is merged to the colour in the palette with the closest Manhattan colour distance if the C distance is within BIN MERGETHRESHOLD1 where the threshold may be BINMERGETHRESHOLD1 10. If the attempt in step 4360 succeeds, processing 00 continues at step 4395 to check if there are more colours to process. Otherwise, C 5 processing moves to step 4370.

O Step 4370 tests if an extra colour can be added to the palette. If step 4370 returns true c-I (YES), processing continues at step 4380. An extra colour is added in step 4380 if the test in the pseudo code below is true.

Number_palette_colours MAX_NUM_PALETTE_COLOURS minDist BIN MERGE THRESHOLD2

|I

minDist BIN MERGETHRESHOLD3 (minDist pCnt) BIN_NEW_MIN pixel_countclosest_palette_colour

BIN_DONT_TOUCH_CNT

minDist is the closest Manhattan colour distance of C to the palette colours. pCnt is the pixel count of C. pixel_count_closest_palettecolour is the pixel count of the palette colour which generates minDist. The threshold values may be MAX NUM PALETTECOLOURS 4, BIN_MERGE_THRESHOLD2 BIN MERGE THRESHOLD3 40, BINNEWMIN 4000 and BIN DONTTOUCHCNT 150 690186_final From step 4380, processing continues at step 4395.

00 If the test in step 4370 is false, processing continues at step 4390. In step 4390, the bin 5 colour C is merged with the palette colour with the closest Manhattan colour distance.

NColours are merged with weighted pixel count and the pixel count of C is added to the Spixel count of the palette colour to merge to. Processing continues at step 4395. If there are no more colours to process at step 4395, processing terminates.

2.2.1.4 Associate Pixels with Second Palette Once the dominant colours are found, pixels within a tile are quantised to one of the dominant colours. A quantised map is produced along with the dominant colour list for connected component analysis. The quantisation process for each group is as follows: 1) Flat no quantisation; 2) Bi-level remap the palette to one of two dominant colours or find a threshold value to binarise the original pixels; and 3) Multi-coloured remap the palette to one of the dominant colours.

Binarisation produces sharper outlines, but takes longer since binarisation requires finding a proper threshold value. The steps of finding the threshold are: 1) perform the first derivative on the luminance channel, 2) identify edge pixels, and 3) use the average luminance value from the edge pixels as the threshold. Edge pixels are pixels where their surrounding 3x3 first derivative output is all above a predefined threshold.

Fig. 44 provides further details of step 1140 of Fig. 11. Step 4410 tests if the tile is classified as bi-level. If the test result is Yes, processing continues at step 4420. In step 4430, the multicoloured tile is mapped. Otherwise, processing continues at step 4430. The bi-level tile is mapped in step 4420.

2.2.1.4.1 Map Bi-Level Tile 690186_final -36- Fig. 45 provides further details of step 4420. The whole process 4420 maps all non-

C

empty bins to the two colours in the second palette with quantisation error checking.

Step 4510 to step 4570 perform the quantisation and error checking for each non-empty 00 bin. If a big quantisation error is not found, the process remaps all pixels to the second N 5 palette after the bin quantisation. If a big quantisation error is found, the tile is C reclassified as multi-coloured and subjected to multi-coloured tile mapping. The O details of mapping bi-level tiles are explained below.

Step 4510 decides if outline enhancement is required and chooses the preferred extreme colour for quantisation for the bin that is subject to outline enhancement. Outline enhancement is required if one pixel count of the two palette colours outweighs the other by a factor of 5. Let the two colours in the second palette have the first colour C1 and pixel count P1 and the second colour C2 and pixel count P2. If (P1/P2) or (P2/P1) is greater than 5, outline enhancement is required and OUTLINE_ENHANCE is set to true. The preferred extreme colour may be the colour with smaller pixel count.

Step 4515 gets the next non-empty bin with pixel count, pCnt. Step 4520 calculates the quantisation error of the two colours. The Manhattan distances (D1 and D2) to the two palette colours are calculated and the smaller one is defined as minDist. minDist is the quantisation error. The process then continues at decision step 4525 to check if the quantisation error is too big. The pseudo code below defines the condition when quantisation error is too big: (minDist pCnt) BIN_NEW_BILEVEL_THRESHOLD

II

minDist BIN NEW BILEVEL COLOUR DIFF pCnt BIN_MERGE_BILEVEL_CNT_MIN 690186_final -37- ,1 where the threshold values may be BIN_NEW_BILEVEL_THRESHOLD 6000, BIN NEW BILEVELCOLOUR DIFF 50 and 00 SBIN MERGE BILEVEL CNT MIN 100.

c- If the test of step 4525 is true, processing continues at step 4540. In step 4540, the current bin is added to the extra dominant colour list. Processing then continues at step 1 4570. If the test in step 4525 is false, processing continues at decision step 4530 to check if the extra dominant colour list is empty. If the list is not empty the process switches to step 4570 to see if there are more non-empty bins to be processed.

Otherwise, if the result of step 4530 is Yes, processing continues at step 4545 to determine if outline enhancement is required and if the two distances are close. The test condition is given as the pseudo code below:

OUTLINEENHANCE

abs(Dl D2) BILEVEL_THRESHOLD_MARGIN where the threshold may be BILEVEL_THRESHOLD_MARGIN 16.

If the test in step 4545 is true, processing continues at step 4550. In step 4550, the bin is quantised to the preferred colour. Processing continues at step 4570. Otherwise, processing continues at step 4555 to quantise the bin to the closer colour based on D1 and D2. The process then continues at step 4570 to check if there are more non-empty bins to be processed. If there are more non-empty bins, processing returns to step 4515. If there are no more non-empty bins, the process continues at decision step 4560 to check if the extra dominant colour list is empty. This determines if there is any big quantisation error during the bin quantisation process. If the list is empty, the process continues at step 4575 and remaps all pixels to one of the two palette colours according to the bin mapping in either of steps 4550 and 4555, as appropriate. Processing then 690186_final S-38-

U

terminates. If the list is not empty in step 4560, the process continues at step 4565 to Sadd one extra colour to the palette. The bin with the highest pixel count in the extra dominant colour list is chosen as the third palette colour. In step 4430, the tile is

O

remapped as a multi-coloured tile. Processing then terminates.

CK1 C, 2.2.1.4.2 Map Multi-Level Tile 0 Step 4430 of Fig. 44 is expanded upon Fig. 46. Step 4610 gets the next non-empty bin.

SStep 4620 quantises the bin to one of the palette colours. This may be done based on the closest Manhattan colour distance. Step 4630 checks if there are more non-empty bins to be processed. If the test in step 4630 is true, processing continues at step 4610.

If the test in step 4630 returns No, processing continues at step 4640. In step 4640, all pixels are remapped to one of the palette colours according to the bin mapping in step 4620. Processing then terminates.

2.2.2 Colour CC Analysis and Blob Statistics This process 1020 of Fig. 10 takes the quantised tile from the previous step and forms blobs. Each blob has the following statistics: bounding box, size, mean colour, bitmask, and blob boundary length. The blobs are formed in a single raster pass in a fast and efficient manner. In raster order, adjacent pixels in the quantised tile belonging to the same colour class are grouped to form "runs". At the end of each run (segment), the run is compared with touching segments (in terms of 8-way connectivity) on the row above for growing or merging. Growing occurs if a run that has yet to be given a blob label touches a blob of the same class. In contrast, merging occurs when two blobs of the same class come into contact. A new blob is formed if no growing is possible. Blob statistics are updated at end of each run.

Fig. 36 is an illustration of the colour blobbing process 3600 using an example 6x6 input tile 3610. Colour quantisation is applied to the input tile 3610, which has a number of colours to produce quantised tile 3620. The quantised tile 3620 contains class labels corresponding to the dominant colour classes. In this case, there are two 690186_final ,1 -39dominant colours in the tile, thus giving class labels 0 and 1. Connected component analysis begins, in a row, by grouping adjacent labels of the same class to form runs. In Fig. 36, the first four 'O's in the first row form a run and the next two 'l's form another 00 run, for example. Blobs are formed by joining runs of the same class from successive C 5 rows in the quantised tile 3620. Tile 3630 shows the runs in the tile. When comparing N1 the current segment with the touching segments from the row above, there are three O possible actions: 1 Grow an existing blob by adding the current segment to the blob; 2. Merge two blobs by consolidating statistics for the two blobs into one; and 3. Form a new blob by initialising the blob using the current segment.

Tile 3640 shows the resulting blobs, blob0 and blob 1, where blob 0 has an outer bounding box and blobl 1 has an inner bounding box. Blob statistics are accumulated at the same time as the runs and blobs are formed. So by the end of the processing stage, each blob has all the statistics as shown in Fig. 36 for blob0 and blobl. The bitmasks 3650 and 3660 are representative. The illustrative bitmaps 3650 and 3660 are not actually formed at this stage. The quantised colour is used as the mean colour of a blob. Alternatively, the mean colour of a blob may be determined by accumulating the actual pixel values in that blob, not the quantised colour. This gives a more accurate mean colour. Again, the blob statistics may comprise size, mean colour, boundary length, bounding box, and bitmask.

Fig. 9 illustrates in more detail step 1020 of Fig. 10, which employs a loop structure to form blobs from the quantised tile, processing a tile row at a time, from top to bottom.

In step 910, a current tile row is obtained for processing. In step 920, a contiguous segment of pixels of the same quantised label is formed. This is done by recording its start and end positions. This segment is the current segment Sc. The start position is the pixel to the right of the end position of the previous segment. In the case of a new tile row, the start position is the first pixel in that row. The end position is the last pixel 690186_final C before a change in quantised label occurs during a pixel-by-pixel examination of the quantised label across the current tile row from left to right. In the case where the current tile row ends before a change is detected, the end position is the last pixel of the 00 row. For the purpose of a later re-estimation of the colour of a blob, YUV values for each pixel in the segment in the original full colour tile are accumulated during the N, examination from its start to end positions. Alternatively, the quantised colour of the O segment maybe used without colour re-estimation.

In step 930, a new blob is formed or an existing blob is grown using the segment.

Fig. 12 provides further details of this step. In decision step 940, a check is made to determine if there are unprocessed pixels remaining in the row. If decision step 940 returns true (Yes), processing returns to step 920. This occurs until all pixels in the current tile row are processed. If step 940 returns false processing continues at step 950. In decision step 950, a check is made to determine if unprocessed rows remain. If step 950 returns true (Yes), processing continues at step 910. This occurs until there are no more unprocessed tile rows. If step 950 returns false processing terminates; every pixel in the quantised tile has been assigned to a blob.

2.2.2.1 Form Blobs Fig. 12 illustrates step 930 in greater detail. The process of Fig. 12 takes as input the current segment Sc, as identified in step 920 of Fig. 9. At step 1205, a variable k is initialised to 1. This value refers to the kth segment Sk of the tile row above the current tile row that is connected to the current segment Sc. Preferably, the connection is 8way connected. In step 1210, a comparison is performed between Sk and Sc. In decision step 1215, a check is made to determine if Sc and Sk are of the same class. If the two segments, Sk and Sc, have the same quantised label, processing moves from step 1215 to step 1220. Otherwise, processing continues at decision block 1235. In decision step 1220, a check is made to determine if Sk is the first connected segment of the same class. Thus, if Sk is the first connected segment with the same quantised label as Sc, processing continues at step 1225. Otherwise, processing continues at step 1230.

690186_final -41- _In step 1225, the current blob is grown. Thus, the current segment is grown upon the blob that the kth segment belongs to. Growing a blob involves updating the blob's size, boundary information, bounding box, and accumulated YUV values. Processing then 00 continues at step 1235. Conversely, if decision step 1220 returns false, this indicates N 5 that the current segment has already been assigned a blob label, blob[i], and come C, into contact with another blob, blob[j], and processing continues at step 1230.

These two blobs are merged together in step 1230 if the blobs have different blob N, labels, i j. The blob merging process in step 1230 is depicted in Fig. Processing then continues at step 1235.

From steps 1215, 1225 and 1230, processing continues to decision block 1235. In decision block 1235, a check is made to determine if the last connected segment has been processed. If the kth segment is not the last segment connected to the current segment, processing continues at step 1240 and k is incremented by 1. Processing then returns to step 1210 to process the next connected segment. Otherwise, if decision step 1235 returns true, processing moves to decision block 1245. In decision block 1245, a check is made to determine if none of the connected segments is of the same class (i.e.

of the same quantised label as the current segment). If step 1245 returns true (Yes), processing continues at step 1250. A new blob is formed using the current segment in step 1250. Forming a new blob involves assigning a new blob label to the current segment, incrementing the number of blobs by 1, and initialising blob statistics using the current segment information. Processing terminates following step 1250.

Likewise, if decision block 1245 returns false processing terminates.

2.2.2.2 Example of Blob Merging Fig. 15 illustrates an example of blob merging 1500. Segments 1510 and 1520 both belong to blob[i], which is connected to segment 1530 belonging to blob[j]. The current start and current end for the segment 1520 are depicted, as are the above start and above end for the segment 1530. Also the overlap of segments 1520 and 1530 are shown in Fig. 15. The blob merging process 1500 involves combining the statistics of 690186_final -42- I the two blobs, mapping one blob label to another, and reducing the number of blobs by

C

1. For example in Fig. 15, the label j is mapped to label i. The following pseudo code instructions are used to combine the blob statistics: 00 blob[i].boundingBox combine(blob[i]. boundingBox, blobl[]. boundingBox)

C

B blob[i].size blob[j].size 0 blob[i].tileBorderPixelCount blob[j].tileBorderPixelCount C N blob[i].horizontalEdges blob[j]. horizontalEdges 2 overlap B blob[i].verticalEdges blob[j].verticalEdges blob[i].YUV blob[j].YUV 2.2.3. Intra-Tile Merging Once the blobs for a tile have been formed by the connected component analysis, the next stage is to use colour, size, and blob boundary length statistics to merge semantically related blobs together. From this stage onwards, blobs can only be merged, not split. Therefore, the tile moves from being over segmented to much closer to the correct level of segmentation. An example of intra-tile merging is shown in Figs.

37 and 38, where blob statistics are used to assess whether to merge a given pair of blobs. In the example, there are four quantisation colours and the connected component analysis returns ten blobs. Fig. 37 shows the blobs prior to merging 3710 on the left. A number of these blobs are due to residual halftone patterns and colour "bleeding" effect between two distinct colour regions. After applying intra-tile merging, small blobs with relatively long boundary length are merged into larger blobs with similar colour features. The blobs after merging 3720 are shown in the example 3700. Fig. 38 contains a comparison 3800 of the original tile 3810 with the merged blobs 3820.

The quantisation and blob forming processes often create many small unwanted or erroneous blobs. These are in the form of small blobs caused by noise speckles, remaining halftones in the input image, or thin, high-aspect-ratio blobs caused by 690186_final C'N -43- _bleeding effects at the edge of larger connected components. The erroneous blobs may be removed by merging those blobs with a blob the erroneous blobs are touching that is closest in colour.

00 Speed and memory usage are improved by limiting the number of blobs in a tile N produced by the segmentation and connected component processes. If there are too many blobs in a tile, the number may be reduced by merging some blobs of similar N, colour even if those blobs are not touching. This produces blobs that have separate disconnected parts, but are treated as a single element. Quality is unaffected as this only occurs in tiles with a lot of small noise elements, which are discarded in a later step.

Fig. 19 illustrates in detail step 1030 of Fig. 10. In step 1905, a blob in the current tile is obtained. In step 1910, the blob perimeter to size ratio is checked to determine if it is high. If this parameter is found to be greater than a threshold level, processing continues at step 1915. Otherwise, if step 1910 returns false, processing continues at step 1930. In step 1915, the ratio of pixels in the blob touching the tile edge is checked.

If the ratio is above a threshold value, processing continues at step 1920. In step 1920, the blob is marked "force merge inter-tile". The force merge flag for the blob is set, which makes the blob much more likely to be merged with connected components in adjacent tiles in step 1420 of Fig. 14. After step 1920, processing continues at step 1930. If in step 1915 the blob's tile edge ratio is below the threshold, processing continues at step 1925. In step 1925, the blob is merged with the neighbour blob with the closest colour. All the blobs touching the current one and their distance in colour from the current blob are found. The current blob is then merged with the neighbour closest in colour. Processing then continues at step 1930. In decision step 1930, a check is made to determine if there are more blobs in the tile to be processed. If step 1930 returns true (Yes), processing returns to step 1905. Otherwise, processing continues with step 1935.

690186_final -44- In decision step 1935, the current number of blobs in the tile is checked to determine if there are too many. If the number of blobs is found to be higher than a predefined 00 limit, processing continues at step 1940. Otherwise, processing ends. In step 1940, the 00 blobs of the same quantised colour class that are not touching the tile edge are merged S 5 together. This is done for each of the quantised colour classes. The blobs touching the N tile edge are not merged, because these blobs may form part of much larger CCs and merging them could have a detrimental effect on quality. In step 1945, the current Nnumber of blobs in the tile is checked again to determine if there are still too many blobs in the tile. If the number is now found to be below the predefined limit (No), processing terminates. Otherwise, processing continues at step 1950. In step 1950, blobs of each colour touching the tile edge are merged. Step 1950 performs a similar process to step 1940 but considers blobs that do touch the tile edge to reduce the number of blobs below the limit. Processing then terminates.

2.2.4. Inter-Tile Merging Fig. 13 illustrates in detail the inter-tile merging process of step 1040. This process is repeated for each of the left and top borders of the current tile in no particular order.

The following description is applicable for merging along either the left or top border of the current tile. For example, a 32x32 tile has a 32-pixel border with its adjacent tile state and each pixel has a blob label. The adjacent tile state also has a 32-pixel border, and each pixel has a CC label. So for each pixel step along the border, there is a blob label in the current tile and a corresponding CC label in the adjacent tile state.

Processing begins, in step 1310, along the common border by obtaining a blob label in the current tile for the next tile border pixel and a CC label in its adjacent tile state.

A test is performed at decision block 1320 to detect a change in the CC label, the blob label, or the last pixel as processing moves along the border. Decision block 1320 returns Yes if the current pixel is the last border pixel. If step 1320 returns false (No), processing continues at step 1380. Otherwise, if step 1320 returns true (Yes), processing continues at step 1330. Step 1330 checks the number of blobs and the 690186_final number of CCs that are available as candidates for merging. In decision step 1340, a check is made to determine if the candidate count condition is satisfied. Decision block 1340 returns Yes if the merging candidate counts of blobs and CCs satisfy a predefined 00 condition 1700 as shown in Fig 17 and described hereinafter.

N~ c-i N If decision step 1340 of Fig. 13 returns false processing continues at step 1370.

Otherwise, processing continues at step 1350. Step 1350 identifies the best blob and CC pair among the merging candidates, for merging based on a colour distance metric.

Let (Y UcVj) and (Yb,,b U,,bob, Vbob) be the YUV colour values for a CC and blob candidate pair, the colour square distance, sd, is given by: sd W (Y bob )Y Ublob) 2 Wv(VI, -Vbob 2 where Wy, Wy and Wy are weights for the Y, U and V channels, respectively. The weights Wy, W, and Wy may be set to 0.6, 0.2 and 0.2 respectively. The best blob and CC pair is the one that has the minimum square distance value.

This best blob and CC pair is processed by step 1360, which performs various merging operations and is described in greater detail with reference to Fig 14. Following step 1360 or a No from decision block 1340, processing continues at step 1370, where blob and CC candidate counts are updated. Fig. 48 shows the CC and blob candidate counts before and after updating for all operating conditions. If both the CC and blob labels are found to have changed, the CC and blob candidate counts are set to 1 for all possible count combinations prior to merging for prior to update indicates "Don't Care"). If only one label is found to have changed and there are two candidates on either side of the tile border, the counts may be set to either 0 or 1 depending on which one of the two candidates is merged. If the second blob is selected for merging (see 4920 in Fig. 49), then both candidate counts are set to 0 after the merging has taken place. However if the first blob or no blob is selected for merging, then both counts are set to 1. In the remaining cases where candidate count update is required, the count for 690186_final c-I-46the label that has changed is incremented by 1, whereas the count for the unchanged C label is set to 1.

00 Following either of steps 1320 or 1370, processing continues at decision block 1380.

C-1 5 In decision block 1380, a check is made to determine if the current pixel is the last N border pixel. If step 1380 returns false processing moves to the next pixel location in step 1310. Otherwise, processing terminates.

2.2.4.1 Inter-Tile Merging Conditions According to Fig. 17, merging usually occurs when there are two candidates on one side and one on the other, except when both CC and blob labels change at the same time in which case one candidate on each side is sufficient for merging. This is so as to avoid two adjacent candidates on one side merging to the same candidate on the other.

Fig. 17 sets out the conditions under which an inter-tile merging operation may be performed. If the current CC and blob counts are then both labels must change at the same time for merging to occur. However if the current CC and blob counts are or then a change in either label is sufficient for merging to occur. The case never occurs.

2.2.4.2 Inter-Tile Merging Examples Fig. 49 provides an illustration of merging between blob and CC candidates. There are three typical cases: i) in 4910 each side has only one candidate; ii) 4920 has one CC and two blobs, and iii) 4930 has two CCs and one blob. In case 4920, the CC candidate is connected to two blob candidates. Suppose both blobs are close in colour to the CC but only one may merge with the CC. If merging is performed in sequence from top to bottom one candidate on each side at a time, then the top blob merges with the CC, leaving the bottom blob unmerged. This might not produce the most desirable result, as the bottom blob could be a better candidate for merging. Thus, two candidates are required on one side for cases similar to those in 4920 and 4930. Case 4910 only 690186_final c, -47- _requires one candidate on each side because there are no alternative merging C combinations for 4-connectedness.

00oO Colour-connected components that span across more than one tile are formed by inter- 5 tile merging blobs in tile raster order. As shown in the example 3900 of Fig. 39, this is performed in raster order, where blobs within a tile 3910 are merged with blobs in either of the two adjacent tiles 3930 and 3920 located to the left of and above the tile 3910, respectively.

Each CC is stored in a data structure that holds information about its bounding box, mean colour, size in pixels, and touching CCs. During the inter-tile merging process, every blob in the current tile is assigned a CC label, and the corresponding CC data structure is updated using the blob statistics. Merging is performed between the current tile 1630 and the states 1612, 1622 two adjacent tile 1610, 1620 along the left and top borders of the current tile 1630 as shown in Fig. 16. The current tile 1630 is a block of pixel data that has been processed to form blobs and has blob labels 1634 for the pixels along the common borders. In contrast, the previous tile state 1612, 1614 does not contain pixel data, only blob statistics and connected component information linked to those blobs. A tile state 1612, 1622 is a compact data structure that contains information about the blobs along the borders in that tile 1610, 1620 and pointers to the CCs 1614, 1624 that those blobs belong to. In particular, there are two different tile states: the left tile state 1612 and the top tile state 1622; each has CC label information 1614, 1624 for each pixel along their common borders with the current tile 1630 for merging with tiles to their right and below, respectively. The blobs in the previous tile are now part of connected components.

2.2.4.3 Process Best Blob and CC Pair Fig. 14 illustrates in greater detail step 1360 of Fig. 13, which takes as input the best blob and CC pair identified in step 1350. Processing commences in step 1410. In decision step 1410, a check is made to determine if the blob has already merged with 690186_final -48another CC. If, at decision block 1410, the identified blob has already been assigned a CC label, processing continues at decision block 1440. Otherwise, processing moves to decision block 1420. At decision block 1420, the colour distance, calculated in step 00 1330, between the identified blob and CC pair is compared with a colour-merging 5 threshold. The value for the threshold may be 450. If the force merge flag has been set, then the threshold may be 900. If the colour distance is less than the threshold, processing continues at step 1430. Otherwise, processing continues at step 1460. In C, step 1430, the blob and identified CC are merged together. This is done by updating the CC's statistics using the blob's statistics and assigning the CC's label to the blob.

Processing then terminates. In step 1460, a new CC is formed for the current blob.

Processing then terminates.

In decision block 1440, the colour distance between the identified CC and the CC to which the identified blob belongs is compared with a colour threshold for merging. If the colour distance between the two CCs is below the threshold, processing continues at step 1450. In step 1450, the CCs are mapped together. This is done by combining their statistics and setting a "map-to" pointer that links the CCs together. Processing then terminates. Likewise, if step 1440 returns false processing terminates.

2.2.4.4 CC Mapping Outcome Fig. 18 is an illustration of the outcome of a CC mapping process in accordance with step 1450 of Fig. 14. The drawing shows that a linked list 1850 of CCs may be formed as a result of merging CCs 1805, 1810, 1820, 1830. In this illustration, the CC (k) 1830 has its map-to pointer 1832 pointing to NULL 1840, which indicates it has never been merged into another CC, thus it is called a root CC. Furthermore, a root CC's statistics 1834 are determined by accumulating individual CC statistics over a number of merges. For example, the final statistics of CC 1830 are the combined statistics of CC 1805, CC 1810, CC 1820 and CC 1830 before those CCs were merged together. The order in which the statistics are combined is not important. In Fig. 18, 690186_final -49- CC 1810 points to CC j) 1820, and both CC 1805 and CC 1820 point to CC 1830.

00 2.2.5 Post Merging Processing S 5 Fig. 50 is a flow diagram of the post merging processing step 1050 of Fig. 10. In step 5010, a new CC is formed for each unmerged blob in the current tile in a process that is 8 the same as step 1460 of Fig. 14. In step 5020, binary images for storing the shape and c, appearance of the blobs are output using the blob labels. For a tile with n blobs, only n-i binary images are needed to be output, because the nth binary image is stored implicitly as the remaining region after the n-I regions have been taken out. Therefore, a flat tile that contains a single blob does not require storing any binary image. In an alternative embodiment, the binary images may be stored in a single, compact data structure as an index map, where each pixel location has a blob index and is represented using log2(n) bits. For example, if the maximum number of blobs per tile is 16, the blob index at each pixel location is encoded using a 4-bit number. In another alternative embodiment, binary images for bi-level tiles may be stored using 1-bit bitmaps. In step 5030, the touching list of each CC inside the current tile is updated.

This is done by identifying all the adjoining CCs in that tile. In step 5040, the tile states are output. The colour segmentation process stores blob and CC information in a compact tile state data structure for merging with the next input tile. As mentioned above, there are two tile states where the left tile state has CC label information along the right tile border, and the top tile state has CC label information along the bottom tile border.

2.3. Segmentation Example Fig. 25(a) shows a simple example that illustrates the advantage of a segmentation based on more than 2 quantisation levels. The background 2510 is black and onto this is placed a white triangle 2520 and the letters of the word 'text' in grey 2530. A binary segmentation of this image would typically result in the merging of the text 2530 with either the background 2510 or the triangle 2520, as seen in Fig. 25(b) and Neither 690186_final of these segmentations could be used for document layout analysis to select text regions N the text features have been lost.

00 At the same time, there are certain features of a binary segmentation that permit a 1 5 simplification of layout analysis of connected components. Consider the case of a page N, with a connected outer boundary and CCs formed by 4-way connectedness. In this case, except on the edges of the page, each CC that touches another CC at a boundary is C, either contained by that CC or contains that CC, and a CC can only be contained by a single other CC. An unambiguous containment hierarchy can be generated and represented in the form of a tree structure. Each successive layer of the tree contains CCs of opposite polarity to the previous, and each branch consists of a set of CCs that share a unique parent. Such a hierarchy is useful in grouping CCs as the hierarchy can be used to select subsets of CCs (those that share a unique parent), which are candidates to be grouped together. This is beneficial in terms of processing speed as fewer CCs need to be considered at a time and in terms of accuracy. CCs from different regions of the page may be on different branches of the tree and are not to be grouped together. Furthermore, processing of CCs can start at the top of the tree and can be terminated for branches of the tree below CCs of a certain classification (e.g.

text) to further improve the processing time.

If the segmentation is not binary, an unambiguous hierarchy generally cannot be usefully generated. Consider the letter in Fig 25(a). The outer boundary of this letter touches both the black background and the white triangle, so that either the background or triangle could be considered as a parent for this CC. The case of the triangle is even more complicated as its outer boundary touches the background and all of the letters. The benefits of using a hierarchy in grouping objects can be achieved despite this ambiguity. This is done by defining the characteristics of a parent-child relationship between two CCs that touch at a boundary, without requiring a unique parent for each child. For example, a parent-child relationship can be defined between two CCs if those CCs touch at a boundary and the bounding box of one of the CCs (the 690186_final cK -51- _parent) completely encloses the bounding box of the second (the child). Using this C, definition for the example shown in Fig. 25(a), the triangle and all of the letters of the word 'text' would be defined as children of the background CC.

00 3. Layout Analysis Layout analysis is the part of the system where foreground content of the page is O identified. The middle (layout analysis) module takes as input from the front-end Smodule a list of connected components and a "touching list". The output of the layout analysis is essentially a decision on which connected components represent foreground content text, tables, bullet points) in the scanned image. The layout analysis is based on a colour segmentation, instead of a binary image. This has numerous benefits in terms of the sorts of foreground objects which it is possible to find, but there is no clear containment hierarchy like there is for binary images. For efficiency, the layout analysis only uses the bounding boxes and a few other general statistics for the connected components to base its grouping on. The layout analysis does not have access to the original pixel data or even the bit-level segmentation.

The main steps of the layout analysis are: forming a containment hierarchy based on the touching list, grouping CCs based on their bounding boxes and colours, and testing these groups to determine whether the CCs are well aligned like rows of text. The touching lists are used to provide a hierarchy for the CCs that is a multi-colour equivalent of a bi-level containment hierarchy. A given CC can be considered as the parent of a subset of its touching list elements. Specifically, the given CC may be a parent of those CCs which the given CC touches and whose bounding boxes are completely contained within the bounding box of the parent CC.

Fig. 4 illustrates in detail step 120 of Fig. 1, which takes as input the compact CC information, statistics, and 'touching list' information generated by step 110. The touching list describes which CCs are next to each other that is, which CCs share a 690186_final -52boundary. The process of Fig. 4 does not have access to the input image. Part of this N information is a list of all the CCs from the input image.

oO 00 In step 410, the CCs are classified based on their statistics, so that a colour containment N, 5 hierarchy is formed from the list of CCs. The colour containment hierarchy is a N structure, where each node is a CC. A parent node has as its children the CCs that the O parent node touches, and whose bounding boxes are completely contained within the bounding box of the parent CC. A child node may have more than one parent node.

The analysis may be based only on the bounding box size and shape. CCs with a width and height both less than 1/100 of an inch 3 pixels at a resolution of 300 dpi) are considered noise and removed. Connected components with a width or height above one inch, or with both width and height above 8/15 of an inch are classified as images.

Everything else is classified as potential text. Alternative embodiments may include classifications related to other document layout features such as tables, the number of pixels in a connected component, and may use other values.

In step 420 potential text CCs are grouped together, representing areas of text. CCs are typically grouped with nearby CCs, and an efficient grouping algorithm takes advantage of this fact by finding the neighbouring CCs before determining grouping.

The high resolution colour segmentation method used in the front end can find thousands of siblings that are considered for grouping on a typical scan document. In these cases, finding neighbouring CCs using a simple pair-wise comparison, an O(N2) method, can become slow, and a more sophisticated method of determining neighbours must be used. A triangulation may be performed on the nodes of the colour containment hierarchy. If the centres of the bounding boxes of the CCs on the page define nodes in a plane, an efficient triangulation method can be used for this purpose, such as Delaunay triangulation These methods are typically O(NlogN) processes.

Fig. 53 shows an illustration 5300 of the Delaunay triangulation (indicated by dashed lines) and Voronoi diagram (indicated by solid lines) of a set of nodes in a plane. The 690186_final c1 -53- Voronoi diagram is a segmentation of the page into regions that are closer to a given point than any other. The Delaunay triangulation is the dual of the Voronoi diagram that can be generated by connecting points together that share a boundary in the 00 Voronoi diagram. A typical point in a plane of randomly located points in the plane N 5 has around 5 points connected to it in this triangulation. These points can be considered as good candidates for neighbours in a grouping stage.

N The triangulation output lends itself to a method of forming groups of CC. Those CCs that are adjacent in the Delaunay triangulation are grouped together based on a pairwise comparison of their bounding boxes. This initial grouping is then followed by subsequent passes over the pairs of adjacent CCs to join these groups together, or place ungrouped CCs into existing groups. The process can also look for different types of grouping (text, table, etc) in a single pass through the data. Groups of text CCs are generally characterised by the following features: similar colour; similar size of bounding boxes; rough alignment along horizontal or vertical axis (depending on text alignment); and close together along axis of alignment relative to the size of the CCs.

In step 430, the groups of CCs are checked or verified to determine which groups of CCs are text characters. The information relating to the group contents and merging generated during the grouping stages is stored by a processor. The information relating to each individual group may be stored in a data structure that includes the colour, bounding box, and contents of a group. These structures are updated during the grouping stage when the contents of a group are changed. In an alternative embodiment, a group marker is included in the CC data structures and data such as group colour and bounding box can be reconstructed from the CC data. In step 430, the text character CCs are subjected to an alignment test as an extra cheek to ensure the CCs are text.

The groups formed generally include all of the text, but may also include parts of images, which are undesirable to classify as text. To reduce this problem, the groups 690186_final -54are examined to see whether the connected components in the group are arranged in neat rows (or columns) like text, or randomly like noise or similarly coloured areas of an image tend to be.

00 C1 5 This may be done mainly by forming four histograms of the bounding box edges, one N, for each side left, top, right or bottom edges). One of these should have full bins O where the baseline of the text is and be empty in other places. To check for this, the sum of the squares of the histogram bins may be found and compared to an expected value. If any of the four histogram bins are found to be much higher than would be expected for randomly arranged bounding boxes, the group is considered to be text. All four bounding box edges are used so as to allow for pages that are scanned in sideways or upside-down, or for text which is arranged in columns rather than rows.

3.1 Grouping CCs Fig. 20 illustrates in detail the step 420 of Fig. 4 which groups the set of CCs segmented by step 110. The process begins at step 2010 by obtaining a root CC. A root CC is one which has not been merged into another CC during the colour segmentation stage. In step 2020, the children CCs of the root CC are found. A list of children CCs for this root CC is formed. The children may be defined as CCs whose bounding boxes are completely contained within the bounding box of the current root CC and that touch the bounding box at a boundary. Step 2020 is described in greater detail in relation to Fig. 22.

From step 2020, processing moves to step 2030. In step 2030, a neighbour analysis on the children of the current CC is carried out. For each child, a set of neighbouring CCs are found which are close in some defined way. This may be achieved, for example, by finding the Delaunay triangulation of the centres of the bounding boxes of each child CC. The edges in the triangulation represent connections between neighbouring CCs.

Alternative methods may use different elements of the bounding box data and colour information for the list of CCs to define proximity. In step 2040, the neighbour data is 690186_final used to carry out an initial grouping. This processing step 2040 forms groups of objects of similar properties geometry colour) within the same child list to determine features of the document layout. Fig. 23 describes in further detail step 00 2040.

C N In decision step 2050, a check is made to determine if there are more root CCs remaining to be processed. If there are more root CCs, processing returns to step 2010, and the next root CC is obtained and subsequently processed. Otherwise, the grouping stage (420) terminates.

3.1.1 Finding Children for Parent CC Fig. 22 illustrates the step 2020 of finding children CCs and forming the list of children for a given parent CC. In step 2210, a touching CC is obtained from the list of touching CCs of the parent CC. In step 2220, the root CC is found. This is done using any merging information from the colour segmentation stage associated with the touching CC. In decision step 2230, classification of the CC from step 410 of Fig. 4 is checked to determine whether the CC satisfies the class test is it appropriate to store this CC in the child list). All CCs other than noise sized CCs may be stored, but alternative embodiments may store other combinations of classes, for example potential text only. If the class of the touching CC is appropriate, processing continues at step 2240. Otherwise, processing continues at step 2260. At decision step 2240, the CC is checked for containment with respect to the parent CC. The containment test involves checking that the bounding box of the CC is completely covered by the bounding box of the parent CC, but alternative methods are possible. If the containment test is satisfied, processing continues at step 2260. The CC can be included in the list of children in step 2250, processing then continues to step 2260. Any CC should only appear once in the list of children so that step 2250 includes a check that the CC is not already included in the list. This check may be carried out using a hash table.

Processing then continues at decision step 2260. Step 2260 tests whether there are any more elements in the touching list of the parent. If so, processing continues at step 690186_final cI -56- 2210. Otherwise, the list of children for the current CC is complete and processing terminates.

00 3.1.2 Initial Grouping N, 5 Fig. 23 illustrates the step 2040 of Fig. 20 of initial grouping carried out using the N neighbour data from step 2030 of Fig. 20. This method may be designed purely to group text, but in alternative embodiments, the method may also classify or group other document objects such as tables. The initial grouping may be a two-pass process in which the first pass joins CCs into groups and the second pass joins groups together into larger groups. In step 2305, the counter PASS is set to 1. In step 2310, a child is obtained. In step 2320, a neighbour of the child is obtained. At step 2330, a check is made to determine if the child and neighbour satisfy a grouping test. These objects are tested for grouping using a series of tests. The tests may be based on initial classification, geometry and colour, and are different between the first and second pass.

Fig. 26 described hereinafter provides an extract of an area of a document with which the grouping test of step 2320 is explained.

Referring to Fig. 23, if the grouping test is satisfied, the child and neighbouring CCs are grouped together in step 2340. Processing then continues at step 2350. Otherwise, if step 2330 returns false, processing continues at step 2350. If two CCs are grouped together, each is marked as belonging to the same group. The group that the CCs are marked with depends on prior grouping of the two CCs. If neither is already grouped, a new group is formed containing both CCs. If only one of the two has already been grouped, the other CC is included in that group. If both have already been grouped and the groups are the same, no action is taken. Finally, if both have already been grouped and the groups are different, the two groups are merged into a single group, and the other group is marked as empty.

At step 2350, a check is made to determine if there are more neighbours of the current CC. If more neighbours exist, processing continues at step 2320. Otherwise, 690186_final -57-

U

Sprocessing continues at step 2360. In step 2360, the process checks for more children Ci of the parent CC. If more children exist, processing continues at step 2310. Otherwise, processing continues at step 2370. In step 2370, a test is conducted to determine if both passes are complete (PASS If this is the case, the process terminates. If only the N, 5 first pass is complete, processing continues at step 2380 and the counter PASS is N incremented. Processing continues at step 2390. In step 2390, processing returns to the start of the list of children. Processing then returns to step 2310 and the second pass Cstarts.

In an alternative embodiment, the process 2040 loops through the edges of the triangulation data rather than pairs of child CC and neighbours. This is slightly more efficient as each pair of neighbours is considered only once.

3.1.2.1 Grouping Test for Two CCs To illustrate the preferred grouping test for two neighbouring CCs, Fig. 26 shows a simple extract from an area of a document containing just two objects the letters 'g' and The dashed lines represent the coordinates of the bounding boxes for each letter. The coordinates are positioned at x, y' and y' for the left and right x coordinates and top and bottom y coordinates, respectively, of the ith CC. The subscripts 1 and 2 refer to the 1 st and 2 nd CC respectively the letters and The processing of this embodiment also uses the colour of each CC in YUV space, u, v and the widths, wi, and heights, hi, of the bounding boxes.

The horizontal overlap distance for two CCs is defined as the length of the horizontal section covered by both of the CCs, or zero if the CCs do not overlap. The vertical overlap distance dyov is defined analogously, and is illustrated in Fig. 26. The horizontal overlap is zero in Fig. 26 and so is not marked. The overlap distances can be expressed as follows: dxY max(0, min(x, x) max(xc, x dy,, max(0, min(y, y2) max(y[, y2)).

dY o2 -1 2 690186 final -58- C The horizontal inner distance between two CCs is defined as the shortest distance between the left edge of one CC and the right edge of the other, or zero if the horizontal 00 Soverlap distance is non-zero. The vertical inner distance is defined in the same way N 5 using the top and bottom edges of the CC. The horizontal distance dxin is illustrated in Fig. 26, while for this example the vertical inner distance is zero and not represented.

O These inner distances can be expressed as follows: dx,, max(xt x,x (2) dyi, max(y' y' 0) In the first pass, two neighbouring CCs are grouped together as text if those CCs meet the requirements of three conditions based on colour, size, and alignment. The colour condition is satisfied if: V(yi y 2 _v 2 Tc (3) where the threshold parameter may be Tc =500.

The size test is satisfied if: m Wmi h m in max W> TR (4) SWmax hmax where Wmin is the minimum width of the two CCs, Wmax is the maximum width, hmin is the minimum height and hmax is the maximum height. The threshold parameter may be TR =0.55.

The alignment condition is satisfied if either of the following conditions is met: and (dy, /max(wn, or 0) and /max(w The threshold parameter may be Ts =0.65.

690186 final -59- _The second pass uses parameters based on groups rather than individual CCs. The mean colour, [Yi, U i V. width, W, and height, Hi, of the elements of each group may be used. For the case of an ungrouped CC, these values are set to the colour, width and height parameters for the individual CC. The test also uses the distance between the (-i centres of the CCs being considered, D, which is defined as follows: X- X, YI'+ Yl b Y2 (6) Dx x 2 (6) 2 2 2 2 As for the first pass, the groups are joined if a series of conditions are met. These conditions relate to colour similarity To, size TR, and separation To, and are described by the following equations: r _U -U) 2

V

2 2 <c max(W,n 1W.,'H x TR (7) m in /mx, H TR2 D max(Wni, Hmm, TD where the parameter values may be Tcg=500, TR =0.55, TR2 and TD =1 if either group contains 3 or less elements, and Tcg=lO0, TR=0.55, TR2 and TD =2 otherwise. No alignment test is used in the second grouping stage.

The thresholds may depend upon features of the CCs being tested for grouping, e.g., the pixel count of each CC.

3.2 Checking Groups Fig. 21 illustrates in detail step 430 of Fig. 4 for checking groups. This process determines whether or not each group consists of text. This decision is made mainly on the basis of whether the objects in the group are found to be aligned in either rows or columns. Groups are assumed to be text, tested for text-like properties, and rejected if the groups fail those tests.

690186_final In step 2110, the next of the groups formed in step 420 is obtained. In step 2120, the N, size of the text characters in the group is estimated. The estimated size is based on the statistics of the lengths of individual characters. These lengths may be defined as the 00 maximum of the width and the height of an object's bounding box. This measure is 5 reasonably insensitive to skew and the alignment of text on the page and is also sufficiently uniform over the set of characters within a typical font of a given size. In O alternative embodiments, bounding box area, pixel count and/or stroke width may be used as measures of the length. A histogram of character lengths may be formed, and the estimated size may be based on the largest length associated with a histogram bin with more than a threshold number of elements in the bin. The threshold used is at least 3 objects and at least 15% of the number of objects in the group. If no such bin exists, no estimate is returned.

In decision step 2125, a check is made to determine if the character size is found. If no suitable character size could be found, the group is rejected and processing continues at step 2160. Otherwise, processing continues at step 2130.

Step 2130 processes the CCs in a group and other suitable CCs that are contained within the bounding box of the group, but have not yet been allocated to any group.

This process 2130 is beneficial for adding text that may have been missed by the original grouping and small objects such as punctuation marks that may have been omitted from the initial grouping based on classification. Only objects that share a parent with the objects in the group and are of a sufficiently similar colour may be added to the group. The colour similarity condition for the group and contained CCs is satisfied if the following condition is met: /(yy)2 2 2 <Tcg 2 (8) where is the colour of the group, is the colour of the CC. The parameter value may be Tcg2=500.

690186_final -61- __Alternatively, geometric tests may be applied in step 2130, and the requirement that the N bounding box of the CC be fully contained by the bounding box of the group may be relaxed so that objects near to the group join the group. Other alternatives of the step 00 2130 may merge some objects to form characters. This is intended for scripts, such as N, 5 Chinese, with complicated characters that may have been segmented as more than one separate object, and is beneficial in improving the accuracy of alignment tests later in 8 processing. Two objects may only merge if their bounding boxes overlap. The N, merging is then limited to not occur if the merging would create an aspect ratio of more than 1.6, or create a merged object that is larger than the character size estimated in step 2120.

In step 2150, the alignment of objects within the group is checked. This test distinguishes text groups from other groups and is described in further detail below.

Following this step, a test is carried out in step 2160 to determine whether there are more groups to process. If there are more groups, processing returns to step 2110.

Otherwise, the process 430 ends.

3.2.1 Check Alignment Fig. 24 depicts in greater detail step 2150 of Fig. 21. During this step, a subset of the CCs in the group are defined as characters. Characters may be those objects that have a size more than half the character size estimated in 2120, and less than twice this size.

Steps 2430 to 2450 conduct acceptance tests for the group based on a histogram analysis of a sequence of parameters related to the group elements. These parameters are the left, top, bottom and right edge of each character's bounding box. Using multiple parameters allows text to be identified in a variety of alignments, since the alignment of text on the page depends on many factors such as the language and skew of text on the page. Alternatively, various combinations of the horizontal and vertical bounding box parameters may be used to identify a broader range of text alignments.

690186_final -62- In step 2430, a histogram is formed for the group element values of the next parameter.

The size of the bins in the histogram may be scaled according to the group character sizes. A value of 1/5 of the average height of the characters in the group (rounded up) 00 may be used for top and bottom bounding box edges, and 1/5 of the average width of C 5 the characters in the group (also rounded up) may be used for left and right bounding c, box edges. The range of bins in the histogram is set so that all of the data are included with non-empty bins at each end of the range. The lowest value covered by the histogram may be set to the lowest value of the parameter in the group.

Decision step 2440 tests whether the values in the histogram are well aligned, forming discrete clusters (ideally representing baselines of different lines of text) rather than spread randomly. Step 2440 tests whether the number of characters in the group, N, is larger than a threshold, T with a preferred value of T=7.

For small groups step 2440 examines three parameters ALl, AL2, and OV.

ALl is the count of the largest bin in the histogram. AL2 is the count of the second largest bin in the histogram. OV is the size of the largest subset of overlapping characters in the group. The pseudo-code in Table 2 describes the tests that are used for this group. The group passes the alignment test if the pseudo-code returns Y, and fails the test if the pseudo-code returns N.

TABLE 2 IF (N 3) return N ELSE IF (N 3) IF (ALl 2 and OV 3) return Y

END

ELSE IF (N 4) IF (AL1 2 and OV 4 and AL2 2) 690186_final 63 return Y ELSE IF (ALl>=3and OV>= 4) return Y 00

END

CI 5 ELSEIF (N IF (ALI 4andOV return Y

END

ELSE EF (N 6) IF (ALI 2and OV >--6and AL2 return Y ELSE E (ALl 4 and OV 6) return Y ELSE TF (ALl 3 and AL2 3) return Y

END

ELSE IF (N 7) IF (ALl 2 and OV 6 and AL2 2) return Y ELSE IF (ALl 6) return Y ELSE IF (ALlI 3 and OV 4 and AL2 return Y

END

END

return N 690186_final c- -64- U a, For large groups, a test is done comparing the sum of the squares of the histogram bins to the expected value for randomly arranged CCs within the group. The equation for this test is given below: 00 (1))n ,h >2x n+ =1 where m is the total number of histogram bins, n is the total number of characters, and Shi is the population of the ith bin of the histogram. The term on the right-hand side of the equation is twice the expected (mean) value, and approximates, for large enough m and n, the value for which there is a 0.1% chance of randomly arranged characters being accepted. An example of this processing is shown in Fig. 27, described hereinafter.

Referring to Fig. 24, if the group is accepted according to the test, and processing continues at step 2470. In step 2470, the group is kept, alignment checking finishes.

Processing then terminates. Otherwise, if step 2440 returns false processing continues at step 2450. Decision step 2450 checks whether there are more parameters to test. If there are, processing continues at step 2430 for the next parameter.

Otherwise, if all parameters have been tested, processing continues at step 2480, where the group is rejected. Processing then terminates.

The foregoing description discloses testing based on one parameter at a time and rejecting exactly those groups that fail every test. However, in view of this disclosure, it will be apparent to those skilled in the art that alternative ways of combining tests for different parameters may be practiced without departing from the scope and spirit of the invention, such as accepting groups which are nearly well enough aligned in two different but similar parameters (such as the top and bottom edges of the bounding boxes), or creating an overall score for the group based on many parameters.

690186_final c- 3.2.2 Alignment Example N, Fig. 27(b) shows a selection of irregularly arranged bounding boxes 2710, 2712, 2714, as might result from parts segmented from an image 2700. Fig. 27(b) shows 00 randomly arranged connected components. Fig. 27(a) shows the histogram 2720 of the C 5 values of the bottoms of the bounding boxes for this image. Fig. 27(d) shows the arrangement of bounding boxes 2740, 2742 on the page for a text group that has well O aligned connected components, and Fig. 27(c) shows the corresponding histogram 2730. As depicted, the histogram 2730 for the text has a few large clusters of values, while the other histogram 2720 has more evenly spread values. Using the sum of squares of histogram bin values as the measure, Fig. 27(a) gives a value of 19, and Fig.

27(c) gives a value of 47. The term on the right hand side of the acceptance test above for m=19 and n=13 is 45. According to this test therefore, the image data of Fig. 27(b) is rejected, while the text data of Fig. 27(d) is accepted. Alternatively, the acceptance test may be based on other statistics, such as the total number of values that are in bins of significantly larger than average count.

4. Generating Compressed Output Image The back end module uses inpainting to make the background image more compressible, by painting over the foreground areas with an estimated background colour. The inpainting is preferably performed on background images at a lower resolution 150 dpi). The inpainting algorithm is a single pass, tile based algorithm and seeks to enhance compressibility. The colour of each pixel is chosen by interpolating from surrounding pixels to the left and right instead of using one average colour for a large area.

The algorithm performs the following steps: 1) combine the masks for all the foreground components to make one foreground mask for the tile; 2) dilate the mask so that a small additional area around the foreground components is inpainted; 3) in raster order over the tile: a. If the pixel is not masked, update the tile's activity; and 690186_final -66b. If the pixel is masked, paint the pixel with a colour interpolated from the N, colours of the nearest non-masked pixels to the left and right; 4) if the activity of the non-masked areas is below a certain threshold, paint the whole 00oO tile with the mean colour of the non-masked pixels (this gives improved compression with ZLib compressed JPEG); and 5) if the whole tile is masked, paint the whole tile N, with the mean colour of the previous tile. Step 2) above eliminates bleeding effects and O improves compression as well as sharpens the output quality.

Fig. 5 illustrates in detail step 130 of Fig. 1. Steps 510 through 540 use a tile-based processing system similar to that described in Figs. 2 and 3. In step 510, the next tile to be processed is obtained. In step 520, an inpainting process is performed on the current tile to remove any foreground CCs as identified in step 120 of Fig. 1 and flatten any tiles which appear visually close to flat. In step 530, the current tile is compressed.

This may be done using JPEG in the YCrCb colourspace with the 2 chrominance channels subsampled by 2:1 horizontally and vertically. The tiles of the reduced resolution background image each comprise 16x16 pixels, which means that those pixels can be encoded directly into four 8x8 pixel JPEG blocks for the Y channel and one 8x8 JPEG block for each of the Cr and Cb channels without any buffering required between tiles.

In step 540, a check is made to determine if there are more tiles to process. If there are any more tiles to inpaint and compress, processing returns to step 510. Otherwise, processing continues at step 550. In step 550, the foreground is compressed, which involves compressing the foreground elements identified in step 120. The foreground elements are grouped according to colour and one binary image at the full input resolution is created for each similarly coloured group of foreground elements. Each image created may then be encoded in CCITT G4 Fax if the image is large enough that the encoding produces a compression advantage in the output document.

690186_final -67- In step 560, the output document is generated. The compressed background and C, compressed foreground images are stored in a compound compressed format. This format may be a PDF document, for example. The JPEG encoded background image 00 may be further compressed using Flate (Zlib) compression. This gives a significant 5 space saving on JPEG images containing a large number of repeated flat blocks as produced by steps 520 and 530. The composite document may be written containing O the Flate and JPEG compressed background images, and a page description containing N, details of the size, position, order and in the case of the binary foreground images the colour to render each of the images on the page.

4.1 Inpaint Tile Fig. 30 illustrates in detail step 520 of Fig. 5. This process modifies the downsampled background image to increase compressibility and enhance the sharpness of the foreground CCs by removing foreground CCs from the background and flattening image tiles with low visible activity. A small area surrounding the foreground CCs is also inpainted to remove bleeding effects to enhance the image and increase compressibility.

The process 520 shown in Fig. 30 removes the selected foreground CCs from the low resolution background by painting the CCs out with colours estimated by interpolating between the colours of pixels to the left and right of the foreground CC. This increases compressibility. A small area around the outside of the CC is also inpainted to increase further compressibility and to enhance the appearance of the image. Compressibility is increased still further by identifying tiles of the background image which have low visible activity and setting all their pixels to the same colour.

The input to the process of Fig. 30 is the downsampled background image created in step 250 of Fig. 2, the list of CCs from step 240 of Fig. 2, and the foreground or background selection information from step 120 of Fig. 1. In step 3010, the tile is checked to see if the tile is marked as being 'flat'. If so, processing continues at the tile 690186 final N -68flattening step 3040. Otherwise, processing continues at step 3020, where a full N resolution foreground bitmask is formed for the tile. This has a bit set in every location that corresponds to a foreground CC. In step 3030, the areas in the background image 00oO corresponding to the foreground CCs and a small area around them are removed by C1 5 inpainting over them with colours interpolated from the colours of pixels to the left and right of the CC. The activity of the non-inpainted pixels in the tile is also measured. In O step 3040, the entire tile is flattened to a constant colour if the tile activity is found to be low enough to be visually flat. Processing then terminates.

4.1.1 Form Tile Foreground Bitmask Fig. 31 illustrates in greater detail step 3020 of Fig. 30 for forming a tile foreground bitmask. In step 3110, an initial bitmask for the tile is created. This bitmask is the same width as the tile and one row taller than the tile. The first row of this bitmask is set to be the same as the last row of the tile above unless this tile is in the first band of the document in which case the first row of the bitmask is set to be blank. This allows the inpainted area to extend below foreground CCs that have a bottom edge that aligns with a tile boundary.

In step 3120, the next CC in the tile is obtained. In step 3130, the CC is checked to determine if the CC is a foreground CC (in step 120). If the CC is a foreground one, processing continues at step 3140. Otherwise, processing continues at step 3150. In step 3140, the bitmask corresponding to this CC and the current tile is combined with the tile bitmask created in step 3110, using a bitwise OR function. Processing then continues at step 3150. In step 3150, a check is made to determine if there are more CCs in the current tile. If so, processing returns to step 3120. Otherwise, if all the CCs in the tile have been processed, the result of step 3150 will be No and processing continues at step 3160. In step 3160, the last line of the mask formed is saved so that it can be used by step 3110 when processing the tile directly below this one on the page.

The bitmask created in the process 3020 of Fig. 31 is at the full resolution of the input image.

690186_final -69- 4.1.2 Inpaint Pixels and Measure Tile Activity Fig. 32 illustrates in greater detail step 3030 of Fig. 30. Each pixel is examined in 00 raster order until one that should be inpainted is found marking the start of a run of C'I 5 pixels to inpaint. Subsequent pixels are examined until the end of the run of pixels to N inpaint is found. The horizontal run of pixels is then painted with colours interpolated O linearly between the colours of the pixels to the left and right of the inpaint run.

Referring to Fig. 32, step 3210 gets a row of the tile. In step 3220, the start of the next run of pixels is found in terms of accumulated pixel activity of row. Each pixel in the row is examined in raster order and the pixel activity in the tile is accumulated until a pixel is found that should be inpainted. The pixel activity of the non-inpainted pixels is recorded by accumulating the pixel values and the squares of the pixel values and keeping a count of the number of pixels measured. To decide whether each pixel should be inpainted, the pixel location is compared with the corresponding locations in the tile foreground bitmasks created in step 3020 of Fig. 30. As the tile bitmask is at twice the resolution of the background image, there are 4 pixels in the mask corresponding to the area covered by one pixel in the background image. Also to improve compression and remove the bleeding edge effect from foreground CCs, a small additional area around the edge of foreground CCs is inpainted. To do this, 8 pixels in the full-resolution foreground mask are examined to decide whether to inpaint the current background pixel. Fig. 34 shows an example pixel 3410 in the background image tile 3430 and the corresponding full resolution mask pixels 3420 which are examined. If any one of these 8 pixels indicated by 3420 is set in the mask 3440, i.e.

corresponds to the location of a foreground CC, the pixel 3410 is inpainted. Since the bitmask 3440 is stored as an array of bit-vectors, these 8 pixels 3420 can be checked quickly using a bitwise AND operation. Alternatively, this may be implemented by dilating the set pixels in the bitmask 3420 and then downsampling. The colour of the last examined pixel that is not inpainted on each row is recorded and made available 690186_final when processing the next tile to the right so that the colour can be used as an N, interpolation value if the first pixel in the same row of the next tile is to be inpainted.

00 Referring to Fig. 32, in decision step 3230, a check is made to determine if any pixels to inpaint have been found before the row's end. If a pixel to inpaint has been found, processing continues at step 3250. Otherwise, processing continues at step 3240. In 8 step 3250, the remaining pixels in the row are examined in raster order to find the end of the run of pixels that should be inpainted in the row. The same test is used here to determine whether a pixel should be inpainted as the test used in step 3220. In step 3260, a check is made to determine if the end of the run of pixels to inpaint has been found before the row's end. If the end of the run of pixels to be inpainted is found before reaching the end of the row, processing continues at step 3270. In step 3270, each pixel in the run of pixels to be inpainted is set to a colour interpolated linearly between the colours of the nearest non inpainted pixels to the left and right. If the run of pixels to inpaint extended to the left hand edge of the tile, the last value saved for the same row in the previous tile is used as the left interpolation value. Processing then continues at step 3220, and the next run of pixels to inpaint is searched for. If in step 3250 the end of the row is reached without finding any pixels which should not be inpainted, the decision step 3260 directs processing to continue at step 3280. In step 3280, each pixel in the run of pixels to be inpainted is set to the value of the nearest non-inpainted pixel to the left, which may be a value saved for the same row in a previous tile. After step 3280, processing continues at step 3240.

In step 3240, a check is made to determine if there are more rows in the tile to process.

If there are more rows in the tile, processing continues at step 3210. Otherwise, processing ends if there are no more rows.

4.1.3 Inpainting Examples Fig. 35 shows some one dimensional inpainting examples. Before and after plots are provided for two examples 3510, 3520, where pixel intensity is plotted as a function of 690186_final -71- I' position X. In the first example 3510, the run of pixels to inpaint is completely within a N,1 tile, and the inpainted pixel values (indicated by diagonal hatching) are replaced with values interpolated linearly between the non-inpainted pixel values to the immediate 00 left and right. The inpainted area is slightly larger than the foreground component to remove any bleeding effect; this is depicted as the dilated mask area. The colour to inpaint each pixel is found by interpolating between the non-masked pixel colours to the left and right. In the second example 3520 of Fig. 25, the run of pixels to inpaint C, crosses a tile edge 3530. When processing the left hand tile, the values of the pixels to be inpainted are replaced with the value of the non-inpainted pixel to the immediate left. When processing the right hand tile of the example 3520, the inpainted pixel values are set to values interpolated linearly between the last recorded non-inpainted pixel value for this row and the first non-inpainted pixel value to the right.

4.1.4 Tile Flattening Fig. 33 depicts in greater detail step 3040 of Fig. 30. In decision step 3310, a check is made if all the pixels have been inpainted. In particular, the number of pixels that were inpainted as recorded in step 3220 is checked. If all the pixels in the tile were inpainted, the visible area of the tile is deemed to be low and processing continues at step 3320. In step 3320, all the pixels in the current tile are painted (set) to the mean colour of the previous tile in raster order. This significantly increases the compressibility of the background image when a block based compression technique like JPEG is used. Processing then terminates. If step 3310 determines that not all of the pixels in the tile were inpainted, processing continues at decision step 3330. In step 3330, the activity of the non inpainted pixels as measured in step 3220 is checked against a pre-determined threshold. If the activity is found to be less than this threshold, the activity visible area of the tile in the reconstructed image is deemed to be close to flat. In this case, processing continues at step 3340 and the tile is made completely flat by painting all the pixels in the tile the mean colour of the noninpainted visible pixels in the tile. This significantly increases the compressibility of 690186_final 72 the background image. The process of Fig. 33 ends after step 3340, or after step 3330 N if the activity in the tile is found to be higher than the threshold.

00 Hardware Embodiment Fig. 51 illustrates a system 5100 in accordance with a second embodiment of the invention. The system 5100 features an efficient data flow between different stages of processing pipelines. The design greatly reduces memory bandwidth and enables the system 5100 to run fast.

Scanners normally acquire scanned data in pixel raster order. The pixel data is then stored and often compressed for further image processing. In a conventional scan-todocument application, the scanned data normally needs to be retrieved from storage, decompressed, and then kept in memory for segmentation and layout analysis to process the image data. This is often the case for high-speed scanners, as the segmentation process simply cannot keep up with the speed of the scanner's page-bypage streaming raster data.

This not only requires a big memory buffer, but also high memory bandwidth as each image pixel needs to be written and read at least twice. First, the pixels have to be written to memory by the scanner and then a compressor reads the data from the memory and compresses the data. Later, a decompressor must read the compressed data and decompress the data into memory. Finally, an image processor can segment the decompressed data. There is at least one redundant memory read and write for each original image pixel, not mentioning the compressed data. For a high resolution scanner 600 dpi), this means over 200MB of extra data.

This embodiment of the invention employs high-speed, auto-segmentation which works directly on the real-time, page-by-page streaming raster data from a scanner 5105. As a result, the redundant memory read and write are completely eliminated and the size of memory buffer is also greatly reduced.

690186_final -73- N, Bus 5110 carries from the scanner 5105 the scanned raster data, which is written to module 5115, a line buffer. In this example, a 64-line buffer is used but other sizes 00 may be practiced, dependent on the height of a band as explained hereinafter. The line N, 5 buffer 5115 stores a band of data for processing by code segmentation module 5125 while simultaneously collecting a new band of incoming scanned data. The module O 5125 reads a tile of data via a bus 5120 from the line buffer 5115 and colour segments N,1 the data into connected components on a tile-by-tile basis. When the module 5125 finishes a band of data, a new band of data is ready from the line buffer 5115 for processing. The old band buffer is then used to collect the new incoming raster data.

The height of a band is determined by the height of a tile, and the line buffer 5115 requires double the height of a band. In this embodiment, the preferred tile size is 32x32.

Module 5125 is implemented in hardware in this embodiment, so that the processing speed of a band can keep up with the speed at which the scanner 5105 produces a new band of data. The output of module 5125 is compact Connected Components (CCs) on bus 5135 to a layout analysis module 5140 and a downsampled image either uncompressed or compressed on bus 5130 to the inpainting module 5150. The data on bus 5135 may be written to memory until a significant area of a page or a whole page of CCs is produced. Module 5140 performs layout analysis using only the compact connected component data. As the data is compact, the processing power required to perform layout analysis is small. Therefore, the layout analysis module may be implemented as software (SW) executed by an embedded processor in real time. The output of the layout analysis module is foreground information provided on bus 5145, which tags the data from bus 5135. The data on the bus 5130 also may be written to memory until a significant area of a page or a whole page of CCs are produced.

The inpainting module 5150 performs the removal of foreground on the downsampled image (provided via bus 5130) on a tile-by-tile basis. This module 5150 may be 690186_final -74implemented on the same embedded processor that runs the layout analysis software 5140, or alternatively it may be implemented by hardware Module 5150 00 decompresses the data from the bus 5130, if the data is compressed. The module 5150 then in-paints the foreground area with an estimated background colour on the downsampled image to produce a background image. The foreground removed background image is output on a bus 5155, and a foreground mask is generated on a O bus 5175.

Output generation module 5160 creates the layout analysed document, such as a PDF file, from the foreground images produced on the bus 5175 and background image produced on the bus 5155. The module 5160 may be implemented by software running on the same embedded processor.

In the second embodiment, modules 5115 and 5125 work on real time page-by-page scan data and produce compact connected components and a downsampled image on page N. Modules 5140, 5150 and 5160 work in sequential order using the data produced by module 5125 on page N-1. The system can therefore deliver layout analysed documents from the live data of high speed scanners in real time.

The modules 5125, 5140, 5150 and 5160 may be implemented in the manner described for the corresponding steps in the first embodiment.

5.1 Colour Segmentation Module Fig. 52 illustrates in detail the module 5125. The components of the Colour Segmentation into CCs module 5125 includes a dehalftone module 5220, which takes a stream of pixels in tile order and removes any artefacts caused by the scanning of material printed using a halftone system of printing, for example an Ink Jet Printer.

The dehalftone module 5220 is a hardware embodiment of the software embodiment described earlier. The module may internally use static RAM and pipelined processing to achieve the necessary speed.

690186_final c-

U

Pixels output from the dehalftone module 5220 are passed to the Colour Convert module 5230, which converts the pixels from the input colour space (often RGB) to the 00 YCbCr luminance/chrominance space. This module 5230 performs the necessary C' 5 multiplications and additions on each pixel according to the formula:- Y R 0.2989 G 0.5866 B 0.1145 Cb= R -0.1687 G -0.3313 B 0.5000 Cr R 0.5000 G -0.4183 B -0.0816 The arithmetic operations are performed in scaled, fixed point arithmetic to reduce the complexity and increase the speed of the module 5125. The output from the Colour Convert module 5230 is passed to two modules, the DownScan Module 5240, and the Connected Component Analysis Module 5260. The DownScan module 5240 performs a simple averaging of the colour of a set of 4 (in a 2 by 2 square) or 16 (in a 4 by 4 square) to form each output pixel. The pixels output from the DownScan module 5240 are then compressed by the Hardware JPEG compressor 5250.

6. Computer Implementation The methods according to the embodiments of the invention may be practiced using one or more general-purpose computer systems, printing devices, and other suitable computing devices. The processes described with reference to any one or more of Figs. 1-52 may be implemented as software, such as an application program executing within the computer system or embedded in a printing device. Software may include one or more computer programs, including application programs, an operating system, procedures, rules, data structures, and data. The instructions may be formed as one or more code modules, each for performing one or more particular tasks. The software may be stored in a computer readable medium, comprising one or more of the storage devices described below, for example. The computer system loads the software from the computer readable medium and then executes the software.

690186_final c-76- Fig. 7 depicts an example of a computer system 700 with which the embodiments of the invention may be practiced. A computer readable medium having such software 00oO recorded on the medium is a computer program product. The use of the computer C1 5 program product in the computer system may effect an advantageous apparatus for implementing one or more of the above methods.

In Fig. 7, the computer system 700 is coupled to a network. An operator may use the keyboard 730 and/or a pointing device such as the mouse 732 (or touchpad, for example) to provide input to the computer 750. The computer system 700 may have any of a number of output devices, including line printers, laser printers, plotters, and other reproduction devices connected to the computer. The computer system 700 can be connected to one or more other computers via a communication interface 764 using an appropriate communication channel 740 such as a modem communications path, router, or the like. The computer network 720 may comprise a local area network (LAN), a wide area network (WAN), an Intranet, and/or the Internet, for example.

The computer 750 may comprise a processing unit 766 one or more central processing units), memory 770 which may comprise random access memory (RAM), read-only memory (ROM), or a combination of the two, input/output (O10) interfaces 772, a graphics interface 760, and one or more storage devices 762. The storage device(s) 762 may comprise one or more of the following: a floppy disc, a hard disc drive, a magneto-optical disc drive, CD-ROM, DVD, a data card or memory stick, flash RAM device, magnetic tape or any other of a number of non-volatile storage devices well known to those skilled in the art. While the storage device is shown directly connected to the bus in Fig. 7, such a storage device may be connected through any suitable interface, such as a parallel port, serial port, USB interface, a Firewire interface, a wireless interface, a PCMCIA slot, or the like. For the purposes of this description, a storage unit may comprise one or more of the memory 770 and the storage devices 762 (as indicated by a dashed box surrounding these elements in Fig. 7).

690186_final c- 77ci Each of the components of the computer 750 is typically connected to one or more of the other devices via one or more buses 780, depicted generally in Fig. 7, that in turn 00 comprise data, address, and control buses. While a single bus 780 is depicted in Fig. 7, Ci 5 it will be well understood by those skilled in the art that a computer, a printing device, or other electronic computing device, may have several buses including one or more of O a processor bus, a memory bus, a graphics card bus, and a peripheral bus. Suitable ,i bridges may be utilized to interface communications between such buses. While a system using a CPU has been described, it will be appreciated by those skilled in the art that other processing units capable of processing data and carrying out operations may be used instead without departing from the scope and spirit of the invention.

The computer system 700 is simply provided for illustrative purposes, and other configurations can be employed without departing from the scope and spirit of the invention. Computers with which the embodiment can be practiced comprise IBM- PC/ATs or compatibles, laptop/notebook computers, one of the Macintosh (TM) family of PCs, Sun Sparcstation a PDA, a workstation or the like. The foregoing are merely examples of the types of devices with which the embodiments of the invention may be practiced. Typically, the processes of the embodiments, described hereinafter, are resident as software or a program recorded on a hard disk drive as the computer readable medium, and read and controlled using the processor. Intermediate storage of the program and intermediate data and any data fetched from the network may be accomplished using the semiconductor memory.

In some instances, the program may be supplied encoded on a CD ROM or a floppy disk, or alternatively could be read from a network via a modem device connected to the computer, for example. Still further, the software can also be loaded into the computer system from other computer readable medium comprising magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red transmission channel between the computer and another device, a computer readable card such as a 690186_final C'N -78- PCMCIA card, and the Internet and Intranets comprising email transmissions and N, information recorded on websites and the like. The foregoing is merely an example of relevant computer readable mediums. Other computer readable mediums may be 00 practiced without departing from the scope and spirit of the invention.

CK1 7. Industrial Applicability O The embodiments of the invention are applicable to the computer and data processing C industries. The foregoing describes only a small number of methods, apparatuses, and computer program products for processing and compressing a digital image in accordance with embodiments of the invention, and modifications and/or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

690186_final

Claims (18)

1. A method of inpainting a digital image comprising a plurality of pixels, 0 said method comprising the steps of: generating a plurality of blocks of pixels from said digital image; and changing in raster order pixel values of at least one run of pixels for at least one block, said changing step comprising the following sub-steps performed on each block: (Ni determining start and end pixels for a run of pixels in said block relating to an object, said run comprising adjacent pixels grouped together; modifying at least one pixel value of said object in said run dependent upon pixel values of pixels outside said run; and determining an activity measure for pixels that do not correspond to an object in said block; and changing all pixel values in each block having at least one run of pixels to a set value if said activity measure for said block is less than a predetermined threshold.

2. The method according to claim 1, further comprising the step of modifying at least one pixel value of a dilated object pixel outside said object dependent upon pixel values of pixels outside said dilated object.

3. The method according to claim 1 or 2, wherein said generating step comprises the sub-steps of: arranging said digital image into a plurality of bands, each band comprising a predetermined number of consecutive lines of pixels; and buffering and processing said bands one-by-one, wherein said processing step comprises the following sub-steps performed on each currently buffered band: arranging said current band into a plurality of blocks of pixels; and 690186 final N buffering and processing said blocks of said current band one- C by-one for said changing steps. 00

4. The method according to any one of claims 1 to 3, wherein said run N, 5 comprises adjacent pixels in a raster line of pixels of said block. The method according to claim 1, further comprising the step of compressing each block using a block-based compression method.

6. The method according to claim 5, wherein said block-based compression method is JPEG.

7. The method according to claim 6, further comprising the step of further compressing said block-based compressed blocks using another compression technique.

8. The method according to any one of claims 1 to 7, wherein the at least one pixel value of said object modified dependent upon pixel values of pixels outside said object uses a value interpolated from pixel values to the left and right of said run, or a value from a pixel to the left of said run.

9. The method according to any one of claims 1 to 8, wherein all pixel values in each block having at least one run of pixels are changed to an average value of a previously processed block or an average value of visible pixels in said block.

10. The method according to claim 1, further comprising the step of, if an end of said run of dilated object pixels is not found, setting colour values of pixels to a colour value of a pixel that does not correspond to an object to the left of said run of dilated object pixels. 690186 final -81-

11. The method according to any one of claims 1 to 10, wherein pixel values N, are colour values. 00oO

12. An apparatus comprising a processor and memory for inpainting a C 5 digital image comprising a plurality of pixels in accordance with the method of any one of claims 1 to 11. N 13. A computer program product comprising a computer readable medium having recorded therein a computer program for inpainting a digital image comprising a plurality of pixels in accordance with the method of any one of claims 1 to 11.

14. A method of changing pixels values of a digital image comprising a plurality of pixels, at least a portion of said pixels corresponding to an object in said image, said method comprising the steps of: arranging said digital image into a plurality of bands, each band comprising a predetermined number of consecutive lines of pixels; and buffering and processing said bands one by one in turn, said processing step comprising the following sub-steps for each currently buffered band: arranging the current band into a plurality of blocks ofpixels; and processing said blocks one by one in turn, said processing step comprises the following sub-steps for each block: determining an activity measure for pixels in said block that do not correspond to objects in the image; if said activity measure is less than a predetermined threshold, changing pixel values of all pixels in said block to a pixel value; compressing said block with JPEG; and compressing said JPEG compressed block using another compression method. 690186_final c- -82- The method according to claim 3 or 14, wherein said bands each cI comprise 16 lines of pixels of said digital image, said blocks comprise 16x 16 pixels, and said compressing steps are performed in a pipeline manner. oO

16. The method according to claim 14, further comprising the step of, setting colour values of pixels in said block to an average colour value of pixels that do 8 not correspond to an object in said block.

17. The method according to claim 14, further comprising the step of, setting colour values of pixels in said block to an average colour of a preceding block.

18. The method according to claim 14, wherein pixels of a dilated object are determined by dilating a mask defining a location of said object.

19. The method according to claim 14, wherein said other compression method comprises ZLIB. The method according to any one of claims 14 to 19, wherein pixel values are colour values.

21. An apparatus comprising a processor and memory for changing pixels values of a digital image comprising a plurality of pixels, at least a portion of said pixels corresponding to an object in said image, in accordance with the method of any one of claims 14 to 690186_final (N -83-

22. A computer program product comprising a computer readable medium c having recorded therein a computer program for changing pixels values of a digital image comprising a plurality of pixels, at least a portion of said pixels corresponding to 00 an object in said image, in accordance with the method of any one of claims 14 to 9 CI DATED this Twenty-first Day of December, 2004 Canon Kabushiki Kaisha C Patent Attorneys for the Applicant SPRUSON FERGUSON 690186_final