KR100896668B1 - Method of processing image data, system for processing images, software module and machine-readable medium - Google Patents

Abstract

A method, system, and machine-readable medium for classifying an image element into one of a plurality of categories is provided, wherein the image is based on a ratio between an unoccluded perimeter of the image element and an occluded perimeter of the image element. Assigning the elements and coding the image elements according to a coding scheme associated with the category in which the image elements are classified. Example applications include image compression, and categories include image foreground and background layers.

Foreground, background, category, coding, perimeter

Description

Method of processing image data, system for processing images, software module and machine-readable medium}

Field of invention

FIELD OF THE INVENTION The present invention relates generally to image processing, and more particularly to methods, systems, and machine-readable media for classifying image elements.

Background technology

Many images are created using computerized methods that do not rely on a pixel-based representation of the image. For example, text processing software displays an image using structured page information that describes high-level elements of the image, such as text, fonts, colors, embedded images, and the like. This structured page information comes from a wide variety of file formats, such as MSWord ™ doc, Adobe ™ PDF, or PostScript ™ files. When printed or otherwise rendered, the information may be converted into a sequence of overlayed images that gradually comprise the image.

Often these images need to be compressed, i.e. encoded. In general, image elements are first classified as either foreground or background based on several classification criteria. After classification, the foreground is encoded at a higher resolution because it contains elements of interest. In contrast, the background is typically encoded at a lower resolution because it contains less interesting elements. Such coding schemes are well known in the art, such as MPEG, JPEG, and the like. Thus, the quality of the element classification greatly affects the compression rate and video quality of these images. For this reason, it is important to perform classification effectively.

Current element classification approaches to images rendered from structured page information classify all text in the image as foreground and all other details as background, and all monochrome elements and background as foreground. Classifying all other things as, and classifying all other things as a foreground and a first element drawn as a background. However, all these approaches are particularly ineffective for geographic maps, since interesting elements are sometimes rendered to meet the criteria for background classification when they are actually foreground elements. As a result, these interesting elements are incorrectly encoded at lower resolutions. As such, the compression efficiency and video quality of these elements are significantly degraded.

Accordingly, there is a need in the prior art for an effective method of classifying image elements, in particular image elements rendered from structured page information, for example electronic documents.

Summary of the Invention

The present invention provides a method for classifying image elements as one of a plurality of categories. The method includes classifying the image element based on the ratio between the unoccluded perimeter of the image element and the perimeter of the image element occluded by other image elements. The image element may then be coded according to a coding scheme associated with the category in which the image element is classified.

Embodiments of the present invention provide a technique for classifying image elements as one of a plurality of categories based on the ratio between the unoccluded perimeter of the element and the occluded perimeter of the element. The occluded perimeter of an image element is typically different from the unblocked perimeter of the element when other image elements are overlaid thereon. Therefore, the ratio can provide a good estimate of the amount that the image element is blocked by other image elements. "Pushing" coding errors for highly occluded image elements can effectively improve the perceived quality of a highly compressed image.

The invention may be applied to image compression, where image elements may be classified as belonging to one of the foreground and background layers. 1A is a block diagram of an image compression system embodying the present invention. The system includes a rendering engine 10, classifier 20, foreground coder 30, and background coder 40. The rendering engine 10 generates an image to be compressed. The rendering engine 10 may be, but is not limited to, a print driver or text processing software. Classifier 20 processes the image by classifying the image elements as either foreground or background elements. Elements classified as foreground may be coded according to any coding technique by foreground coder 30. Similarly, elements classified as background may be coded according to any coding technique by background coder 40. Exemplary coding techniques are well known in the art. Such techniques are described, for example, independently or in connection with coding techniques described in "High Quality Document Image Compression with DJVu" by Journal of Electronic Imaging, 7 (3): 410-425, 1998, L. Bottou et al. May be used. Typically, foreground and background coding techniques will be chosen to provide compression of foreground image elements with low loss of image quality. Background image elements, on the other hand, will be coded with higher compression and relatively higher loss of image quality.

It will be appreciated that the image compression application is for illustrative purposes only, and the present invention may also be used for any application where image element classification is performed. It will also be appreciated that the number of categories is not limited to two.

In typical image compression, the low ratio between the unoccluded perimeter and the occluded perimeter of the image element may indicate whether the element is classified as belonging to the background layer, and any coding errors for the boundaries of the elements may result from the foreground layer. It is likely to be obscured by other occlusion factors. Therefore, coding errors will not be noticed. In contrast, a high ratio between the unoccluded perimeter and the occluded perimeter of the image element may indicate that coding errors for the boundaries will be observed because they will not be obscured by other elements. In this case it may be suitable for coding an element in the foreground layer. Therefore, the present invention provides a high compression ratio and improved visual quality of the compressed image.

The number of bits for encoding the foreground elements may be proportional to the perimeter P _occluded of the visual portion of the element (ie, after removing the element type portions occluded by other elements). The edges of the background elements resulting from occlusions by the background elements may be defined by the boundary of the foreground elements that occlude. Thus, the number of bits for encoding the background elements may be reduced by excluding the occluded portions of the background elements, which may already be encoded as part of the foreground elements. Therefore, the number of bits for encoding the background elements may be proportional to the length P _unoccluded of the perimeter segments that do not result from occlusions by the foreground elements. The proportional coefficient also depends on the color differences along the element boundary.

1B is a flowchart of a method according to an embodiment of the present invention. According to the invention, the image classification system _{calculates the} P _unoccluded 100 of the image element and the P _occluded 110 of the image element when occluded by other image elements. The system compares the ratio of these calculated values to a predetermined threshold T 120:

(1)

(One)

If the ratio of P _unoccluded to P _occluded exceeds the threshold, then the system classifies the image element as belonging to foreground layer 140. Otherwise, the system classifies the image element as belonging to the background layer 130. Typical threshold T is 80%. The system then codes the image elements using either a foreground or background coding scheme based on the element classification 150.

Optionally, the system assigns some predetermined types of image elements to the foreground or background layer. For example, the system may pre-assign text and symbols to the foreground layer. Since the foreground may be encoded at a higher resolution, the encoded text and symbols as foreground improve their readability.

As some aspects may be optional, it is understood that all or some of the invention may be used in connection with other methods for classifying image elements. For example, if text can be automatically assigned to the foreground layer, the method need not be applied to the image elements of the text.

An image element processed according to the present invention may initially be displayed as structured page information describing the components of the image element and its coordinates in the image. Most of these components are simple actions such as "rectangle fill", "draw line", or "draw single text". These operations assign a solid color to a set of specific parts called an element shape. Later operations may overwrite image data generated by faster operations. When the print operations are complete, they produce a completed printed output (often a page).

In the present invention, the system processes the image element from its structured page information so that the image element coordinates are read and compared with the foreground and background layers. From this comparison, the system determines the occluded portions of the image element. Next, the system moves the image element to either the foreground or background layers.

2 is a flowchart of an exemplary method for calculating the perimeter of an image element. This method performs a number of boolean operations between foreground and background layers and image element shapes. In this embodiment, these layers and shapes may be represented using run length encoding. In run length encoding, each scan line (or row) of layers and image elements is represented by a sequence of lengths describing successive runs of black and wavy bits. A run is a contiguous group of zeros (white bits) or ones (black bits) that are encountered during the left-to-right scan of the scan line. For illustrative purposes, black bits represent image elements and white and white represent non-element space. Thus, the system scans each scan line of the layer or element shape from left to right to calculate the perimeter of the run length encoded layer or element shape by making a single pass on the “black bit” runs.

For each "black bit" run, the system calculates 205 the circumference r of the run by adding twice the run length, i.e., the number of bits and twice the width. The length and width values are doubled to account for the upper and lower lengths and the terminal widths, respectively. The system then calculates 210 the length l of each contact portion between the run and the next run in the next scan line, i.e., the run bits in the next scan line that are adjacent to the run and are part of the same layer or element shape.

After processing all runs, the system sums 220 all perimeters r for calculating R and all contact lengths l for calculating L (220). Since the perimeters of adjacent runs contain the same contact length, the system multiplies the sum L by 2 to account for overlapping inclusion. The system calculates 230 the perimeter of the layer or element shape as P = R-2L.

3 shows the perimeter calculation in FIG. 2 for a typical image element. In this example, image element 300 consists of five scan lines. Except for the third scan line, each scan line has a single "black bit" run, separated into two by a "white bit" run. The system determines the circumference r of each "black bit" run 310. Next, the system calculates the sum R of the circumferences r of the runs 310. The system detects the contact portions 320 between adjacent runs and sums all their lengths l and multiplies the sum by two to yield 2L. The perimeter P of the image element 300 is equal to R-2L.

It will be appreciated that the perimeter calculation is for illustrative purposes only, and that the perimeter calculated by other techniques may be used in the present invention. Such techniques include, but are not limited to, contour-mapping and region-growing.

4 illustrates an example for detecting unoccluded or occluded portions of image elements for which the system calculates perimeters such that the image element is classified as belonging to one of the foreground or background layers, using the method of FIG. 2. Representative method. First, the system creates two empty layers F and B to display image elements to be classified as foreground and background. The system then does the following for all image elements starting at the top element and going toward the bottom element.

The system calculates 400 the perimeter P _original of the image element. This is the perimeter of the original shape of the image element because the image element can appear without occlusion. The system then determines 405 the portion of the image element shape occluded by the background image elements drawn over the current image element. This is achieved by calculating the intersection of the image element shape and the current background (B). For example, for a given image element portion i, if the current background portion B (i) has a value, then the image element portion i is designated as occluded and removed.

For the processed first image element, ie the top element, the background B is empty so that there are no occluded parts to be removed. For subsequent elements, if the background includes previously processed occlusion elements, the system removes the occluded portions from the image element shape (410).

Next, the system determines 420 the resulting portion of the element shape occluded by the foreground image elements drawn over the current image element. This is accomplished by calculating the intersection of the image element shape and the current foreground F. For example, for a given image element portion i, if the current foreground portion F (i) has a value, then the image element portion i is designated as occluded and removed.

For the first image element processed, ie the top element, the foreground F is empty so that there are no occluded parts to be removed. For subsequent elements, if the foreground includes previously processed occlusion elements, the system removes the occluded portions from the image element shape (425). The system then calculates (430) the perimeter (P _occlusion ) of the portions occluded by the foreground and background elements.

The resulting image element shape includes only the visible parts of the image element. The system calculates (440) an element's _occluded perimeter (P _occluded ) and an element's unoccluded perimeter (P _unoccluded ), where

P _occluded = R-2L (2)

As mentioned above, the perimeter of the visible parts, and

(3)

(3)

The perimeter of the visible boundary, where the sum of the occluded perimeter and the original perimeter is equal to two times the sum of the unoccluded perimeter of the image element and the perimeter of the obstructed portions to be removed.

5A-5C show exemplary occluded and unoccluded portions of an image element detected in accordance with the method of the present invention. 5A shows a polygonal image element 700 occluded by an 'ab' image element 710. The _occluded perimeter P _occluded of the polygon 700, ie the perimeter of the visible portions with the occluded portions removed, is shown in FIG. 5B. The unoccluded circumference P _unoccluded of the polygon 700, ie discontinuous visible boundaries (indicated by bold black lines), is shown in FIG. 5C.

FIG. 6 shows the calculation from equation (3) used to determine the unoccluded perimeter of polygon 700. The sum of the _original perimeters and occluded perimeters P _original and P _occluded is equal to two times the sum of the perimeter _occluded of the _unoccluded perimeter P _unoccluded and the occluded portions.

It should be understood that the method of detecting occluded and unoccluded portions of the image elements is merely exemplary, and numerous variations of the method may be used by the present invention.

Optionally, the color difference δ may be included in the following ratio.

(4)

(4)

The color difference can be used to determine how close the color of the image element matches the background color. If these colors match closely, then the edges of the image element are not sufficiently distinguished, which increases the likelihood of the element belonging to the background.

In general, multiple passes for an image element are used to determine whether the element belongs to the foreground or background. This is done because elements located below the current element may be identified as foreground elements rather than background elements, and thus may affect the perimeter of the current element. In the present invention, the calculation of the color difference allows the estimation of the likelihood of the current element to be assigned to the background, thus allowing the determination of whether a single pass for the image element should be foreground or background.

Since the calculation of the color difference may be very expensive, two exemplary unifications may be used. 7A is a flowchart of the first example. The system calculates 500 the color difference between the element boundary and each adjacent background element. The system then selects 510 the largest of these color differences as δ.

7B shows a second example. The system selects a predetermined number of background elements (550). The system heuristically determines these background elements during a preliminary pass to the image elements depending on the overall size and shape of the background elements. That is, an image element, such as a solid rectangle, having a large area and a large area overlapping the current image element compared to a bounding box (ie, a virtual boundary around the element) is a potential background element. As such, image elements can be used to calculate color differences. In contrast, an image element, such as a long thin curve, having a small area compared to a bounding box and a small area overlapping with the current image element is not a potential background element. As such, image elements may not be used to calculate color differences.

After the system selects the background elements, the system calculates 560 the color difference of each selected adjacent background elements and element boundary. The system then averages the color difference as δ.

It will be appreciated that the methods described for calculating color differences are for illustrative purposes only, and many techniques for calculating color differences may be used in the present invention. Such techniques include, but are not limited to, gradient measurements.

8 is a flowchart of a method according to another embodiment of the present invention. According to the present invention, the image classification system initializes 580 so that the foreground layer F and the background layer B are empty. Thereafter, the system repeatedly classifies each image element (582).

For each image element, the system does the following. The system determines 583 the portion of the image element that intersects the foreground and background layers, and then identifies and removes the intersection as the occluded portions of the image element (584). The system then calculates an unoccluded perimeter calculation (P _unoccluded) elements of the image, and 585, the two rail (P _occluded) clogging of the image element (586). The system compares the ratio of the predetermined threshold T and their calculated perimeters, as in equation (1). If the ratio of P _unoccluded to P _occluded exceeds the threshold, the system classifies the image element as belonging to the foreground layer (589). Otherwise, the system then classifies the image element as belonging to the background layer (588). An exemplary threshold T is 80%. Optionally, the system includes the color difference of the ratio, as in equation (4).

After all the image elements have been sorted, the system then codes 590 the foreground and the background using their respective coding schemes.

9 illustrates an example electronically generated image that may be separated in accordance with the present invention. The system decomposes the image 600 into a sequence of basic image elements 610 through 655 drawn on top of each other, and the front 610 and back 655 of the sequence are respectively the top and bottom of the sequence. Represents the bottommost image element. That is, the system prints the front image element last and the back image element first. The system will individually print the basic image elements 610-655 as shown in FIG. 9 using printing software that processes a given image format.

According to the invention, the system first prepares empty foreground and background layers F and B representing image elements to be classified into foreground and background. Thereafter, the system processes the first element 610 according to the method of the present invention. That is, the system calculates the circumference P _original of the original shape of the first element using, for example, the circumference calculation of FIG. 2. Thereafter, the system determines portions of the first element shape occluded by the background and foreground image elements shown above the current image element, for example as in FIG. 4. Since the layers F and B are empty at this point, the first element is not blocked and the parts are not removed. Thus, P _occlusion of these non-existent occlusion portions is zero.

Now, the system calculates the perimeters (P _occluded , P _unoccluded ) of the first element from equations (2) and (3). Optionally, the system can calculate the color difference between the first element and the background, for example according to the method of FIG. 7A or 7B. The system then calculates the ratio of occluded and unoccluded perimeters. Based on the ratio, the system classifies the first element as belonging to one of the foreground and background layers, for example, as shown in FIG. 1B. Since the first element is not occluded, the occluded perimeter and the unoccluded perimeter are the same. Thus, the first element has a ratio of 1 or has a color difference and is larger than that. If the threshold is set to 80% (ie 0.8) then the first element is then assigned to the foreground. The system then updates the foreground layer F to include the first element.

In the case of occlusion image element 650, occluded by elements 610, 615, 620, 630, and 635, layers F and B are formed by elements 610-645 prior to element 650. It is processed and classified so it is not empty. Thus, the system determines and removes the occluded portions of element 650. The _occluded perimeter P _occluded of element 650 may be greater than the _unoccluded perimeter P _unoccluded . As such, the ratio between the perimeters may be less than the threshold. In this case, the system classifies element 650 as a background.

The system repeats the method of the present invention for image elements 610-655 so that classified image elements are generated, after which the system effectively encodes.

The mechanisms and methods of the present invention may be implemented using a general purpose microprocessor programmed according to the present disclosure. Thus, the present invention may also include a machine-readable medium containing instructions that can be executed by a processor to perform a method according to the present invention. This medium may include, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, or any type of media suitable for storing electronic instructions.

10 is a block diagram of one embodiment of a computer system that may implement the present invention. System 700 may include, but is not limited to, a bus 710, a system memory module 730, and a storage device 740 in communication with processor 720 in accordance with embodiments of the present invention.

It will be appreciated that the structure of the software used to implement the present invention may take any desired form, for example, the illustrated method may be implemented in a single program or multiple programs.

Many modifications and variations of the present invention are possible in light of the above teachings. Accordingly, within the scope of the appended claims, the invention may be practiced otherwise than as specifically described herein.

Through the present invention, a technique is provided for classifying image elements as one of a plurality of categories based on the ratio between the unoccluded perimeter of the image element and the occluded perimeter of the image element.

1A is a block diagram of a system embodying the present invention.

1B is a flow diagram of an embodiment of the method of the present invention.

2 is a flowchart of exemplary steps for calculating an image element perimeter.

3 shows an exemplary perimeter calculated according to the method of FIG. 2.

4 is a flow chart of exemplary steps for calculating occluded and unoccluded perimeters.

5A-5C illustrate exemplary occluded and unoccluded perimeters calculated according to the method of FIG. 4.

6 shows circumference calculation without occlusion.

7A and 7B are flowcharts of example steps for calculating color difference of an image element.

8 is a flow chart of another embodiment of the method of the present invention.

9 illustrates an exploded electronically generated image as a sequence of exemplary image elements drawn on top of each other, to which the methods of FIG. 1B or 8 may be applied.

10 is a block diagram of one embodiment of a computer system that may implement the present invention.

Explanation of symbols on the main parts of the drawings

10: rendering engine 20: classifier

30: foreground coder 40: background coder

720: Processor 730: Memory

Claims (26)

In the way of processing image data: Assigning an image element to one of a plurality of categories based on a ratio between an unoccluded perimeter of the image element and an occluded perimeter of the image element; And Processing the image element based on assignment to one of the plurality of categories. The method of claim 1, Processing the image element further comprises coding the image element in accordance with a coding scheme associated with one of the plurality of categories. The method of claim 1, Determining the shape of the image element. The method of claim 3, wherein Determining the shape of the image element further comprises removing the occluded portions of the image element. The method of claim 1, And wherein the plurality of categories comprise a foreground layer and a background layer. The method of claim 1, Before assigning the image element: Determining a foreground layer and portions of the image element that intersect with the background layer; Identifying the intersection portions as occluded portions of the image element; Removing the occluded portions to determine the shape of the image element; And Calculating the unoccluded perimeter and the occluded perimeter of the image element. The method of claim 1, And the unoccluded perimeter is calculated based on the original perimeter of the image element, the occluded perimeter of the image element, and the perimeter of the occluded portions. The method of claim 1, And the occluded perimeter is calculated based on the sum of the perimeters of each horizontal scan line of the image element, the sum of the lengths of the contact portions between each horizontal scan line of the image element and the next horizontal scan line. The method of claim 1, The ratio is based on a color difference between the image element and at least one adjacent image element, the unoccluded perimeter, the occluded perimeter. The method of claim 9, And the color difference is calculated to be the largest of the color differences between the image element and the at least one adjacent image element. The method of claim 9, And wherein said color difference is calculated to be an average of color differences between said at least one adjacent image element and said image element. The method of claim 1, If the ratio exceeds a predetermined threshold, the image element is classified as one of the plurality of categories. The method of claim 5, wherein The coding schemes associated with the plurality of categories are respective image compression schemes, and the compression scheme of the background layer realizes a coded output having a higher compression ratio and image loss than the compression scheme of the foreground layer. Way. In a system that processes images: Means for classifying an image element as one of a plurality of categories based on a ratio between the unoccluded perimeter of the image element and the occluded perimeter of the image element; And Means for processing the image element based on classifying the image element as one of the plurality of categories. The method of claim 14, And means for generating an image element. The method of claim 14, Wherein the plurality of categories comprise a foreground layer and a background layer. The method of claim 14, Means for coding the image element in accordance with a coding scheme associated with one of the plurality of categories. delete A computer readable medium, the medium controlling a computing device to perform classifying an image element as one of a plurality of categories based on a ratio between the unoccluded perimeter of the image element and the occluded perimeter of the image element. And program instructions for executing the computer-readable medium. The method of claim 19, And the program instructions further comprise coding the image element in accordance with a coding scheme associated with one of the plurality of categories. The method of claim 19, Wherein the plurality of categories comprises a foreground layer and a background layer. The method of claim 21, The coding schemes associated with the plurality of categories are respective image compression schemes, and the compression scheme of the background layer realizes a coded output having a higher compression ratio and image loss than the compression scheme of the foreground layer. Media available. The method of claim 19, The ratio is based on a color difference between the image element and at least one adjacent image element, the unoccluded perimeter, the occluded perimeter. The method of claim 23, And wherein said color difference is calculated to be the largest of color differences between said image element and said at least one adjacent image element. The method of claim 24, And wherein said color difference is calculated to be an average of color differences between said at least one adjacent image element and said image element. The method of claim 19, If the ratio exceeds a predetermined threshold, the image element is classified as the one of the plurality of categories; otherwise, the image element is classified as another one of the plurality of categories. Media available.

Images