DE3881392T2 - System and method for automatic segmentation. - Google Patents

Description

The invention relates to a system and method for automatically segmenting a scanned document in an electronic document processing device.

A document to be processed in an electronic document processing device, such as an electronic copier, an optical character recognition system or a data compression system, may contain different types of information that must be processed in different ways to achieve sufficient copy quality and compression and/or to enable image manipulation. For example, a document may contain continuous tone photographs, rasterized or dithered images, hereinafter referred to as halftone images, and black and white information such as text or graphics. When the data obtained by scanning such a document is subjected to image processing and image storage in an electronic copier and printed, the information representing text or graphics is usually thresholded to obtain a binary image, while periodic information (rasters) is subjected to gray-scale thresholding and continuous tone information is dithered. Consequently, it is necessary to locate and identify areas or segments of the document that contain different types of information. This process is called "segmentation".

An example of a conventional automatic segmentation system capable of separating text information from halftone image information is described in EP-A2-0 202 425. In this system, the scanned image of the document is divided into a matrix of blocks or sub-images of 4 x 4 pixels in size. Then each of these blocks is labelled either as TEXP or IMAGE. Since the labelling of the relatively small blocks is subject to statistical fluctuations, the resulting matrix of labels will often contain short chains of text blocks in an area where image blocks predominate, and vice versa. In a final step of the segmentation process, the label matrix is smoothed by eliminating such short chains of blocks. In other words, a context rule is applied that requires that the identifiers of isolated blocks or isolated short blocks chains be switched to match the identifier prevailing in the environment.

In general, an automatic segmentation system has to meet two contradictory requirements. On the one hand, it should be fast enough so that the document can be processed at high speed. On the other hand, it should be sufficiently robust so that it can also process documents for which it is difficult to distinguish between different types of information, for example because the characters of a text are printed in light colors on a dark background or because, for example, photographs contain light areas. In order to improve the robustness of a conventional segmentation system, a larger number of criteria must be checked in the recognition step and/or in the smoothing step, so that the processing time increases.

It is an object of the present invention to provide an automatic document segmentation system and method having improved robustness and speed.

This object is achieved by the system specified in claim 1 and the method specified in claim 17.

According to the invention, the number of different identifiers that can be chosen in the initial identification step is larger than the number of different types of information between which a decision must ultimately be made. Consequently, it is not necessary to identify the type of information with certainty in the initial identification step. For this reason, the initial identification step can be carried out in a relatively short time, even if the size of the partial images is chosen to be relatively large in order to reduce statistical fluctuations. The type of information indicated by the initial identifiers is finally determined in the smoothing step on the basis of context rules. It has been shown that context rules suitable for this purpose do not require too much computing time, so that a net time saving is achieved. Furthermore, since the initial identifiers represent a differentiated classification, certain errors that occurred in the initial labeling step can be corrected in the smoothing step, even if these errors relate to relatively large chains of sub-images. This contributes to an improved robustness of the system.

Useful details and further improvements of the inventive system are specified in the dependent claims.

In the following, preferred embodiments of the invention are explained in more detail with reference to the drawings. They show:

Fig. 1 is a block diagram illustrating the general structure of an automatic segmentation system;

Fig. 2 is a graphical representation of a classification tree used in the initial labeling step;

Fig. 3(A) - 3(E) are diagrams of context rules used in the smoothing step;

Fig. 4(A) and 4(B) show an example of an initial identification matrix and a smoothed identification matrix obtained therefrom;

Fig. 5 is a block diagram of a possible hardware implementation of the system according to the invention; and

Fig. 6 and 7 are block diagrams of modified examples of the segmentation system.

As shown in Fig. 1, a document segmentation system includes an initial characterization module 10 and a smoothing module 12. A signal representing the scanned image of the entire document is transmitted from an original scanner (not shown) to the initial characterization module 10. The scanned image is considered to consist of a matrix of sub-images. In a representative example An A4 document is scanned with a scanning resolution of 500 dpi and the size of the partial image is 64 x 64 pixels. The gray value of each pixel is specified by an 8-bit word corresponding to one of 256 gray values.

In the initial labeling module 10, each individual sub-image is analyzed using a classifier that has a number of routines for extracting characteristic features from the sub-image. On the basis of the extracted features, a specific initial label is assigned to the sub-image. The result is an initial label matrix that represents the entire document and whose matrix elements are the labels of the individual sub-images.

The initial label matrix is further processed in the smoothing module 12. A number of context rules are applied to change the initial labels depending on the labels assigned to the neighboring sub-images. The context rules are designed such that some of the initial labels are completely eliminated during the smoothing process.

The result is a smoothed identifier matrix, which in this embodiment consists of only two different identifiers that form segments that correspond to areas with continuous toning (e.g. photographs) or black and white areas (e.g. text or graphics) of the scanned document.

The classifier used in the initial labeling module may be based on conventional image analysis methods such as histogram evaluation, spatial analysis, differential operators, spectral analysis, or a combination of these methods. The classifier may be a classification tree in which the checking routines to be applied each depend on the result of the previous check, or alternatively a one-step classifier may be used. As a preferred embodiment, Fig. 2 shows a classification tree that evaluates a gray-value histogram of the partial image. Each branch point of the tree diagram shown in Fig. 2 corresponds to a specific criterion for which the histogram data is checked. For example, the the following criteria should be considered.

- The abscissa position of the highest peak of the histogram, i.e., the gray value that occurs most frequently. This criterion provides a rough measure of the overall brightness of the sub-image.

- The number of peaks in the histogram. In particular, a histogram with two separate peaks can indicate text or graphics.

- The height difference between two peaks. For text or graphic information, in most cases there will be a large height difference between the highest and second highest peak.

- The gray value difference between two peaks. In black and white images, this difference can be large.

- The height of the minimum between two dominant peaks. In a continuous tone image this value will be high.

- The height difference between the highest peak and the minima on one or both sides of it, as a kind of "signal-to-noise ratio".

- The number of pixels below the minimum value of the valley between the two main peaks. This number will be large in halftone images.

- The widths of the highest peak or the two highest peaks. Narrow peaks may indicate text or graphics.

If the criteria used in the classifier have a wide range of possible results, the results are thresholded to obtain a manageable number of branches. The structure of the tree, the criteria used in it and the thresholds for the results can be optimized by adapting them to statistical results obtained from a number of calibration documents. The greater the variety of calibration documents, the greater the robustness but also the required complexity of the classifier.

In the example shown in Fig. 2, the classification tree provides four different identifiers as possible classification results, which are denoted by BW, BIM, BG and U. In this example, the features of the images marked with these identifiers can be described as follows:

BW: Two dominant gray values with a high contrast; a candidate for text or graphics (BW stands for black/white ),

BIM: An image that also has two dominant gray values, but is not a strong candidate for text or graphics based on other criteria (BIM stands for "bimodal"),

BG: A typical background area; relatively bright and with low contrast; can occur in text or graphic segments, but also in halftone segments,

U: An area with a diffuse gray value distribution (U = "undefined"); a candidate for images with continuous tones.

An example of an initial label matrix obtained with the classifier described above is shown in Fig. 4(A). It can be seen that this initial label matrix still has a number of fluctuations that must be eliminated in the smoothing step.

The context rules used in the smoothing procedure are described below with reference to Fig. 3.

To compare the matrix elements with their respective neighbors, the matrix elements are grouped into arrays A, A' of 3 x 3 elements. Four context rules are applied to each 3 x 3 array, which are illustrated in Fig. 3(A) to 3(D).

A so-called "LOCAL" context rule shown in Fig. 3(A) is used to eliminate isolated identifiers in a uniform environment. This rule can be formulated as follows:

If an identifier X is surrounded by upper, lower, right and left neighbors with identifier Y, change X to Y. In this rule, X and Y are any of the initial identifiers BW, BIM, BG, U.

The context rules shown in Fig. 3(B) and 3(C) can be called “weak expansion” rules and have the following structure:

If at least one element in a 3 x 3 array A' has the identifier BW (weakly expanding identifier) and the array contains no identifiers from a certain group, then expand the identifier BW over the entire array.

The expansion rule shown in Fig. 3(B) transforms combinations of BW and BG into BW and can be written briefly as BW/BG--> BW. In this rule, the "specific group" of identifiers that should not be contained in the field consists of the identifiers BIM and U. If any of these identifiers are contained in the field, the field is left unchanged by this context rule.

In Fig. 3(C), the "specific group" of forbidden identifiers includes the identifiers BG and U. Consequently, this context rule only transforms fields consisting of combinations of BW and BIM, and it can be written as BW/BIM-->BW.

It is possible to set up further context rules with the same structure by defining other groups of identifiers that must not be included in the field. For example, it is possible to convert the entire field to BW in one step if the field consists of a combination of BW, BG and BIM.

Furthermore, these context rules can be modified by requiring that the field must contain at least two, three or more elements with the identifier BW.

An EXPAND context rule shown in Fig. 3(D) requires the following:

If the array A' has at least one element with the identifier U (strongly expanding identifier), the identifier U is expanded over the entire array.

This rule does not impose any restrictions on other identifiers that appear in the original field.

Fig. 3(E) illustrates a context rule called FILL that is not restricted to the 3 x 3 fields. This rule can be defined as follows:

1) Where the identifier U forms intersecting vertical and horizontal chains 14,16, fill the entire rectangle 18 spanned by these chains with the identifier U (the term "chain" denotes an uninterrupted sequence of identifiers U in a row or a column of the matrix);

2) Check all combinations of horizontal and vertical chains to maximize the area filled with U;

3) If the height of the maximized area is less than four elements or its width is less than four elements, change all identifiers of this area to BW.

According to an extension of the context rule FILL, the spanned rectangles are only filled with the identifier U if they contain a number of U identifiers that is greater than a specified ratio (Umin/U) and/or the shape of the rectangle is bound to certain conditions, for example larger than a specified minimum or smaller than a specified maximum.

The smoothing module 12 is programmed to apply the context rules illustrated in Fig. 3 in an order listed below.

1) LOCAL

2) BW/BG--> BW

3) LOCAL

4) BW/BIM--> BW

5) LOCAL

6) BW/BG--> BW

7) LOCAL

8) EXPAND

9) LOCAL

10) FILL

In each of these steps, the context rule is applied to the entire matrix before the next step is executed. In the case of the context rule LOCAL, the entire matrix is scanned in single-element steps with a 3 x 3 window, so that each element is considered once as the central element of the 3 x 3 array A.

The same procedure can be applied in steps 2), 4) and 6). Alternatively, the matrix can be divided into a rigid grid of 3 x 3 fields A'.

In step 8) a rigid grid of 3 x 3 fields A' is used. Alternatively, the floating field method can be used, but then it should be required that each field contains at least two identifiers U since otherwise the expanded area would be too large.

Step 1) starts with the initial identifier matrix generated in module 10. All other steps are applied to the modified matrix obtained as a result of the previous step. Note that the application rule LOCAL is executed several times, alternating with other context rules. The rule BW/BG-->BW is applied in step 2) and again in step 6).

At the end of step 7, the identifiers BG and BIM will be largely eliminated and the matrix will have areas that are homogeneously filled with the identifier BW, while other areas contain the identifier U in combination with other identifiers. In these areas, the Identifier U is expanded in steps 8) and 10) so that at the end of step 10) the entire matrix is composed of rectangular regions that are homogeneously filled with either BW or U. However, the FILL rule dictates that regions filled with U are converted to BW if they are too small.

The identification matrix obtained at the end of step 10) thus consists of the identifiers BW and U, which form large rectangular segments representing black/white areas and areas of the scanned document with continuous toning, respectively. This matrix corresponds to the desired smoothed identification matrix. To distinguish this matrix from the initial identification matrix, the identifiers BW and U are renamed smoothed identifiers T (for "TEXT") and P (for "PHOTO"), respectively.

Fig. 4(B) shows the smoothed label matrix obtained from the initial label matrix shown in Fig. 4(A). These figures reflect experimental results obtained by applying the segmentation method described above to a test document containing a text region with different text formats and two photograph regions. The actual edges of the photograph regions of the document are indicated by dashed lines 20.

It can be seen that the P-segments in Fig. 4(B) match the actual boundaries of the photographic areas within the resolution of the matrix of sub-images.

As shown in Fig. 4(A), the photographer's liver moose contain relatively large contiguous areas filled with the labels BW, BIM and BG, which could just as well have been interpreted as text areas. In the smoothing process, these ambiguities were successfully eliminated using the context rules.

Fig. 5 shows a possible hardware implementation of the automatic segmentation system.

The document is scanned in a scanner 22 and the digital values indicating the gray levels of the individual pixels are stored in a bit field. These values are also transmitted to a histogram unit 24 which creates histograms for the individual sub-images of the document. The histogram data are evaluated in a classifier 26 which corresponds to the initial labelling module 10 in Fig. 1. The classifier 26 contains a feature extraction unit 28 for checking the features of the histogram and a classification tree 30 which selects the features to be checked and finally assigns one of the initial labels to the sub-image under examination.

The initial identifiers are further processed in a context processor 32, which corresponds to the smoothing module 12 in Fig. 1. The context processor contains processing modules 34 for sequentially applying the context rules (steps 1 to 10) as well as buffers 36 for storing the initial identifier matrix, the intermediate results and the smoothed identifier matrix.

In a modified hardware implementation, multiple histogram units 24 and classifiers 26 may be provided so that multiple sub-images can be processed in parallel in the initial labeling phase.

In the embodiment shown in Fig. 1 to 4, the segmentation system only distinguishes between two different types of information, namely black/white information (identifier T) and continuous tone information (identifier P). The photograph segments can contain both continuous tone information and periodic information such as raster or dithered images. The solution according to the invention is also applicable to segmentation systems that further distinguish between continuous tone information and periodic information. This can be achieved, for example, by modifying the segmentation system in the manner shown in Fig. 6 or 7.

In both figures, halftone indicators are used to detect halftone information. Halftone information can be detected by using the following criteria for each sub-image:

- The distance between the first peak in the spectrum that is not at zero frequency and the origin of the spectrum.

- The ratio between the peak corresponding to zero frequency and the first non-zero peak in the spectrum.

In Fig. 6, the initial labeling process and the smoothing process are carried out in the same way as in Fig. 1, and then photograph areas (identifier P) are further analyzed to distinguish between continuous tone information and periodic information. This can be achieved by checking one of the previously mentioned raster indicators, which is done in the periodicity module 38. The system shown in Fig. 6 has the advantage that the time-consuming checking for periodic information is limited to the segments identified as a photograph area.

Alternatively, the check for periodic information can be performed in the initial labeling phase, as shown in Fig. 7. In this case, the initial labels will contain at least one label indicating a strong candidate for raster images, and the context rules in the smoothing module will contain rules for expanding this label so that the smoothed label matrix contains three different labels corresponding to continuous tone information, periodic information and black/white information.

The context rules for finding areas containing halftone information can be similar to the described context rules FILL and EXPAND (Fig. 3), which are intended for finding areas with continuous toning. Instead of the target identifier U, the identifier indicating halftone information will then be used.

In the example shown in Fig. 7, an edge analysis module 40 is additionally provided to improve the congruence between the segments (P and T in Fig. 4B) and the actual edges 20 of the photograph areas of the document.

Edge analysis can be accomplished, for example, by shifting the grid of subimages vertically and horizontally by a certain fraction of the subimage size (e.g. 1/4, 1/2, 3/4) and repeating the initial labeling and smoothing procedures for the shifted grids. A comparison of the different results then provides more detailed information about the actual position of the edges of the photographic area.

Optionally, the edge analysis for vertical and horizontal edges can be limited to the parts of the document where vertical or horizontal edges of the photographic area are expected.

In an alternative approach, edge analysis can be performed by examining specific target regions centered on the coordinates of the label transitions in the smoothed label matrix. For example, the target regions are divided into subwindows that provide a higher resolution than that used in the initial labeling, and each subwindow can then be classified as an edge subwindow or a non-edge subwindow.

The analysis of the target areas can be limited to isolated locations on the transition lines in the smoothed label matrix. If the edge lies exactly within these target areas, the exact location of the entire edge can be found by extrapolation.

While specific embodiments of the invention have been described above, a person skilled in the art will be able to envisage various modifications, all of which fall within the spirit and scope of the invention as set out in the claims.

For example, the context rules mentioned in connection with Fig. 3 can be executed with matrices that are larger than the described size of 3 x 3 subimages.

Claims (17)

1. System for automatically segmenting a scanned document in an electronic document processing device to separate document areas (T, P) containing image information of different types, such as black and white images, continuous tone images and the like, comprising - matrix generating means for dividing the scanned image representing the entire document into a matrix of partial images, - identification means (10) for analyzing the information contained in each partial image and for assigning an initial identifier (BW, BIM, BG, U) to the partial image so as to obtain an initial identifier matrix (Fig. 4(A)), and - smoothing means (12) for smoothing the initial identifier matrix by changing the identifiers according to context rules in order to obtain a pattern of uniformly labelled segments representing the different document areas, characterized in that the marking means (10) are arranged to select the initial identifiers from a first set of identifiers (BW BIM, BG, U), and the smoothing means are arranged to convert the initial identifiers into smoothed identifiers (T, P), which smoothed identifiers are selected from a second set of identifiers which is numerically smaller than the first set. 2. System according to claim 1, wherein the context rules implemented in the smoothing means (12) include rules for expanding some of the initial identifiers (BW, U) and for eliminating other of the initial identifiers (BIM, BG) and the expanded identifiers are ultimately identified with the smoothed identifiers (T, P). 3. System according to claim 2, wherein context rules contain at least one rule having one of the following structures: "if in a certain array (A') of matrix elements in the initial identifier matrix at least n elements have the identifier BW and this array contains no identifiers from the group G, then change all identifiers in this array to BW"; where n is a given number, BW is a given initial identifier and G is a given subset of the set of initial identifiers, b) "if in a given array (A') of matrix elements in the initial identifier matrix at least m elements have the identifier U, then change all identifiers in this array to U"; where m is a given number and U is a given initial identifier. "c1) where the identifier U forms intersecting vertical and horizontal chains (14,16), fill the entire rectangle (18) spanned by these chains with the identifier U; c2) check all combinations of horizontal and vertical chains to maximize the area to be filled with the identifier U; c3) if the height of the maximized area is less than hmin elements or the width is less than wmin elements, change all identifiers within this area to BW"; where U and BW are given initial identifiers and hmin and wmin are given numbers. 4. System according to claim 3, characterized in that the context rules further contain a rule with the following structure: c4) "if the rectangle mentioned in c1 contains a number of identifiers U that is greater than a given number Umin, fill the entire rectangle with the identifier U", c5) "if the shape of the rectangle mentioned in c1 satisfies the conditions: width/height > min and width/height < max, then fill the entire rectangle with the identifier U", where min and max are given numbers. 5. A system according to claim 3 or 4, wherein the predetermined fields (A') to which the context rules (a) and (b) are applied have a size of 3 x 3 sub-images. 6. System according to claim 3, 4 or 5, wherein the smoothing means comprise several stages (34) for gradually modifying the initial identifier matrix by applying each context rule at least once according to a predetermined order. 7. The system of claim 6, wherein the context rule with the structure is applied before the context rule with the structure (b) and the context rule with the structure (b) is applied before the context rule with the structure (c). 8. A system according to claim 6 or 7, wherein the context rules include a local rule requiring that "if a given matrix element is surrounded by upper, lower right and left immediate neighbors, all of which have the same identifier X, then the identifier of the given matrix element is also changed to X", where the local rule is applied immediately before any of the rules with structure (a), (b) or (c). 9. System according to one of the preceding claims, in which the size of each partial image is at least 16 x 16 pixels, preferably 64 x 64 pixels and the scan resolution is greater than 100 dpi. 10. System according to claim 9, wherein the marking means (10) include a histogram unit (24) for generating gray value histograms of each partial image. 11. System according to one of the preceding claims, in which the identification means (10) comprise a tree classifier for determining the identifier to be assigned to a given partial image by checking features of the input data according to a tree structure. 12. System according to one of the preceding claims, in which the identification means contain a device (38) for detecting raster or dithered information. 13. System according to one of claims 1 to 11, with context rules for finding halftone areas. 14. System according to one of claims 1 to 11, with a device (38) for detecting rasters or dithered information only in photograph areas which have been segmented by the smoothing means (12). 15. System according to one of the preceding claims, with several identification means (10) for parallel processing of the data from several partial images. 16. A system according to any preceding claim, comprising edge line analysis means (40) responsive to the output values of the smoothing means (12) for locating the edges of the segmented regions with greater resolution. 17. A method for automatically segmenting a scanned document in an electronic document processing device to separate document areas (T, P) containing different types of image information, such as black and white images, continuous tone images and the like, wherein - the scanned image representing the entire document is divided into a matrix of partial images, - the information contained in each partial image is analyzed to assign an initial identifier (BW BIM, BG, U) to the partial image in order to obtain an initial identifier matrix, and - the initial identifier matrix is smoothed by changing the identifiers of individual matrix elements according to context rules in order to obtain a pattern of uniformly labelled segments corresponding to the different document regions, characterized in that the initial identifiers are selected from a first set of identifiers (BW, BIM, BG, U) and that in the smoothing step the initial identifiers are converted into smoothed identifiers (T, P) selected from a second set of identifiers which is numerically smaller than the first set.