JP2818448B2 - System and method for automatic document segmentation - Google Patents

Description

The present invention relates to a system and method for automatically segmenting a document scanned by an electronic document processing device.

Documents to be processed by an electronic document processing device, such as an electronic copier, optical character recognition system or data compression system, require sufficient copy quality and sufficient compression to be obtained.
And / or consists of different types of information that must be processed in different ways to enable image manipulation.
For example, a document may be a continuous tone photograph, or white or black information such as text or diagrams, or rasterized or dithered, hereinafter referred to as "halftone". May consist of images. When the data obtained by scanning such a document undergoes image processing, image storage and printing in an electronic copier, information representing text or charts is usually thresholded to obtain a binary image. On the other hand, the period information (raster) is thresholded to a gray value (grey-value), and the continuous tone information is dithered. As a result, it is necessary to find and identify locations of regions or sections of the document that contain different types of information. This process is called "segmentation."

One example of a conventional automatic segmentation system capable of distinguishing text information from halftone image information is disclosed in EP-A2-0 202 425. In this system, a scanned image of a document is subdivided into a matrix of blocks or subimages having a size of 4 × 4 pixels. Further, each of these blocks is classified with a label of either TEXT or IMAGE. Because the labeling of relatively small blocks suffers statistical variability, the resulting matrix of labels often contains short runs of TEXT blocks in the region where the IMAGE block predominates, The reverse is also true. In the final stage of the segmentation process, the label matrix is relaxed by removing such short runs of blocks. In other words, a context rule is applied that dictates that the label of an isolated block or an isolated short sequence of blocks be transformed to be identical to the dominant label in the environment.

In general, automatic segmentation systems must satisfy two conflicting requirements. On the one hand, the system must be fast to allow for fast processing of documents. On the other hand, it is difficult to distinguish different types of information, for example, because the text-printed characters are printed in light colors or on a dark background or because, for example, the photograph contains light areas. The system must be powerful enough to handle even the most important documents. In order to improve the performance of conventional segmentation systems, it is necessary to look at a number of criteria in the labeling and / or relaxation stages, thus increasing the processing time.

It is an object of the present invention to provide a system and method for automatic document segmentation with improved power and speed.

This object is achieved by a system as set forth in claim 1 and a method as set forth in claim 17.

According to the invention, the number of different labels that can be selected in the initial labeling stage is greater than the number of different types of information that must ultimately be distinguished. As a result, there is no need to reliably identify the type of information at the initial labeling stage. For this reason, the initial labeling stage can be completed in a relatively short time, even if a relatively large sub-image size is chosen to reduce statistical variability. The type of information represented by the initial label is ultimately determined in the mitigation phase based on context rules. It has been found that a context rule suitable for this purpose does not require much computation time, and thus provides a net time savings.
Furthermore, because the initial labels give a differentiated classification, some errors that occur in the initial labeling stage are corrected in the relaxation stage, even if these errors occur in relatively large series of subimages. Is possible. This contributes to the improved strength of the system.

Useful details and further refinements of the system according to the invention are set forth in the appended claims.

In the following, preferred embodiments of the present invention will be described with reference to the accompanying drawings.

As shown in FIG. 1, the document segmentation system comprises an initial labeling module 10 and a mitigation module 12. Signals representing a scanned image of the entire document are transmitted from a document scanner (not shown) to the initial labeling module 10. The scanned image is considered to be composed of a matrix of sub-images. In a typical example, an A4 document is scanned at a scanning resolution of 500 dpi, and the size of the sub-image is 64 × 64 pixels. The gray level of each pixel is represented by an 8-bit word corresponding to one of the 256 gray levels.

Within the initial labeling module 10, each individual sub-image is analyzed by a classifier consisting of several routines that extract unique features from the sub-image. Based on the extracted features, a particular initial label is assigned to the sub-image. The result is an initial label matrix, which represents the entire document,
The matrix elements are the labels of the individual sub-images.

Further, the initial label matrix is processed in the relaxation module 12. Some context rules are applied to transform the initial labels that depend on the labels assigned to adjacent sub-images. The context rule is designed so that some of the initial labels are completely removed during the relaxation process.

Consequently, in this embodiment, the scanned document consists of only two different labels that separately form segments corresponding to the continuous tone area (eg, a photograph) and the white / black area (eg, a text or chart). A relaxed label matrix is obtained.

The classifier used in the initial labeling module is
It may be based on conventional methods of image analysis, such as histogram evaluation, spatial analysis, differential operator, spectral analysis or a combination of these methods. The classifier may be a tree classifier in which the test routine applied is dependent on the results of the preceding test separately, or alternatively a one-shot classifier is used. Is also good. In a preferred embodiment, FIG. 2 illustrates a tree classifier that evaluates a gray level histogram of a sub-image. Each of the branches of the tree diagram shown in FIG. 2 corresponds to a particular criterion against which the histogram data is examined. For example, the following criteria are taken into account:

The highest peak position of the histogram on the abscissa, ie the most frequently occurring gray level. This criterion gives a rough indication of the overall brightness of the sub-image.

The number of peaks in the histogram; In particular, a histogram with two separate peaks can be a text or chart indicator.

The height difference between the two peaks. In most cases of Texas or chart information, there is a large difference in height between the highest peak and the second highest peak.

The difference in gray level between the two peaks. For white / black images, this difference will be large.

The height of the minimum level between the two dominant peaks. In continuous tone images, this level will be high.

A height difference between the highest peak and the lowest level on one or both sides, such as a kind of "signal-to-noise suspension".

The number of pixels below the minimum level of the valley between the two main peaks. This number will be large for halftone images.

The width of the highest peak or the two highest peaks. The narrow width can be a text or chart indicator.

If the criteria used in the classifier can yield a wide range of results, the results are thresholded to get a practical number of branches. The structure of the tree, the criteria used for it, and the thresholds that represent the results may be optimized by matching them to statistical results obtained from several standard documents. The greater the variety of standard documents, the more powerful the classifier is, but the more complexity it requires.

In the example shown in FIG. 2, the tree classifier gives four different labels as a result of its possible classification, which are labeled BW, BIM, BG and U. In this example, the features of the images displayed by these labels can be described as follows.

BW: two dominant gray levels with high contrast; an image (BW stands for black / white), which is a candidate for text or charts, BIM: two dominant gray levels, but with respect to other criteria Images that are not strong candidates for text or diagrams (BIM stands for "bimodal"); BG: typical background area; relatively bright and low contrast; It can occur not only in chart segments but also in halftone segments. U: Region with diffuse gray level distribution (U stands for "undefined"); candidate for continuous tone image.

An example of an initial label matrix obtained using the above classifier is shown in FIG. 4 (A). It will be appreciated that this initial label matrix still contains some variation that must be removed in the relaxation step.

The context rule used in the mitigation phase is the third
It will be described below with reference to FIGS. 3A to 3E.

The matrix elements are combined into a 3.times.3 element array A, A 'to compare the matrix elements with the elements adjacent to each. From FIG. 3 (A) to third
(D) The four context rules shown in FIG.
Applies to x3 arrays.

The so-called "LOCAL" shown in Fig. 3 (A)
Context rules have the purpose of removing isolated labels in a homogeneous environment. This rule is formulated as follows.

If label X has an upper, lower,
If surrounded by right and left neighbors, then convert X to Y.

In this rule, X and Y represent the initial labels BW, BIM, BG and U.

The context rules shown in FIG. 3 (B) and FIG. 3 (C) may be referred to as “weak extension” rules, and have the following structure.

If at least one element in the 3x3 array A 'has a label BW (weakly spreading label) and the array does not contain a label from a predetermined group, then spread the label BW over the entire array. .

The extension rule shown in FIG. 3 (B) converts the combination of BW and BG into BW, and may be simply written as BW / BG...> BW. Under this rule, the "predetermined group" of labels that must not be included in the array consists of the labels BIM and U. If any of these labels are included in the sequence, the sequence is left untransformed by this context rule.

In FIG. 3C, the "predetermined group" of the forbidden labels consists of the labels BG and U. Thus, this context rule only converts sequences consisting of a combination of BW and BIM, which may be written as BW / BIM...> BW.

By defining other groups of labels that must not be included in the array, it is possible to create other context rules with the same structure. For example, if the array is
If it consists of a combination of BW, BG and BIM, it is possible to convert the entire sequence into BW at once.

In addition, these context rules may be modified by requiring that the array must include at least two, three, or more elements with the label BW.

The "EXPAND" context rule shown in FIG. 3 (D) specifies as follows.

If the array A 'contains at least one element with a label U (a strongly spreading label), the label U is spread over the entire array.

This rule places no restrictions on the appearance of other labels in the initial sequence.

FIG. 3E shows a context rule called "FILL", which is not constrained by a 3.times.3 array. This rule is defined as follows:

1) If the labels U form vertical and horizontal runs 14, 16 that intersect, the entire rectangle 18 of these runs is filled with labels U (the term “run” means , Indicating the uninterrupted order of the labels U in the rows or columns of the matrix).

2) Examine the combination of horizontal and vertical runs to maximize the area filled with U.

3) If the height of the maximized area is less than four elements, or if its width is less than four elements,
Then convert all labels in this area to BW.

As an extension to the context rule FILL, a rectangle whose length and width are the continuation of U is a predetermined ratio (U minimum number
/ U) if it contains a greater number of labels U and / or the shape of the rectangle is constrained to certain conditions, for example greater than a predetermined minimum or less than a predetermined maximum. If done, the rectangle is only filled by the label U.

The mitigation module 12 applies the context rules shown in FIGS. 3A to 3E in the following order.

1) LOCAL 2) BW / BG ……> BW 3) LOCAL 4) BW / BIM ……> BW 5) LOCAL 6) BW / BG ……> BW 7) LOCAL 8) EXPAND 9) LOCAL 10) FILL In each of the steps, the context rules are applied to the entire matrix before the next step is performed. In the case of the context rule LOCAL, in one element step the entire matrix is scanned using a 3 × 3 window, so that each element is
It is considered the central element of the × 3 array A.

In steps 2), 4) and 6), the same procedure may be applied. Alternatively, the matrix may be divided into fixed grids in a 3 × 3 array A ′.

In step 8), a fixed grid of 3 × 3 array A ′ is used. Alternatively, a floating array method may be employed, but each array must be required to include at least two labels U. Otherwise, the expanded area would be too large.

Step 1) starts with the initial label matrix generated in module 10. All other steps are performed on the modified matrix resulting from the preceding step. It will be appreciated that the application of the rule LOCAL is performed several times, interleaving other context rules. Rule BW / BG ……> BW
Is applied to step 2) and again to step 6).

At the end of step 7), most of the labels BC and BIM have been removed and the matrix will show the area uniformly filled by the label BW, while other areas have been combined with other labels Includes the label U in the form. Within these regions, the label U will be spread out in steps 8) and 10), so at the end of step 10)
The entire matrix is made up of rectangular areas uniformly filled with BW or U. But the rule FILL mandates that if the area filled with U is too small, that area will be converted to BW. The label matrix thus obtained at the end of step 10) consists only of the labels BW and U which form a large rectangular segment representing each of the white / black and contone areas of the scanned document. This matrix corresponds to the required relaxation label matrix. To distinguish this matrix from the initial label matrix, each of the labels BW and U is represented by the relaxed labels T (for "TEXT") and P (for "PHOTO").
You can change the name.

FIG. 4 (B) shows a relaxed label matrix obtained from the initial label matrix shown in FIG. 4 (A). Text area with several text formats and 2
This figure shows an experimental result obtained by applying the above-described segmentation processing to a test document including two photograph regions. The actual boundaries of the photographic area of the document are shown by dashed lines 20.

It will be appreciated that the P segment of FIG. 4 (B) matches the actual boundaries of the photographic region within the resolution of the sub-image matrix.

As shown in FIG. 4 (A), the photo area has labels BW, BIM and B, which can be interpreted as text areas.
Includes a relatively large coherent region filled with G. In the mitigation process, these ambiguities have been successfully removed by context rules.

FIG. 5 shows an example of a hardware device of the automatic segmentation system.

The document is scanned in scanner 22 and a digital number representing the gray level of each pixel is converted to a bitmap.
Is stored within. Further, these numbers are transmitted to a histogram unit 24 which produces a histogram for the individual sub-images of the document. The histogram data is evaluated in a classifier 26 corresponding to the initial labeling module 10 of FIG. The classifier 26 comprises a feature extractor 28 for examining the features of the histogram, and a tree classifier 30 for selecting the features to be examined and ultimately assigning one of the initial labels to the examined sub-images.

Further, the initial labels are processed in a context processor 32 corresponding to the mitigation module 12 of FIG. The context processor determines the context rule (step 1)
It comprises a processing module 34 for sequentially applying steps 10)) and a buffer 36 for storing the initial label matrix, the intermediate result and the relaxed label matrix.

In the improved hardware device, a plurality of histogram units 24 and a classifier 26 may be arranged so that a plurality of sub-images can be processed in parallel in an initial labeling stage.

In the embodiment shown in FIGS. 1 to 4, the segmentation system determines only two different types of information: white / black information (label T) and continuous tone information (label P). Just do it. Photo sections may include continuous tone information as well as periodic information such as rasterized or dithered images. The invention according to the present invention is also applicable to a segmentation system for further discriminating between continuous tone information and periodicity information. This can be achieved, for example, by modifying the segmentation system as shown in FIG. 6 or FIG.

In both figures, the halftone index is used to detect halftone information. By using the following criteria for each sub-image, it is possible to detect halftone information.

The distance between the first non-DC peak value in the spectrum and the origin of the spectrum.

The ratio between the DC peak value in the spectrum and the first non-DC peak value.

In FIG. 6, the initial labeling process and the relaxation process are the first
This is done in the same way as in the figure, after which the photographic area (label P) is further analyzed to determine continuous tone information and periodicity information. This is achieved by examining one of the aforementioned raster indices and is performed by the periodicity module 38. The system shown in FIG. 6 has the advantage that the time-consuming examination of periodicity information is limited to only those sections identified as photographic areas.

Alternatively, a check on the periodicity information may be performed at the initial labeling stage as shown in FIG. In this case, the initial label indicates a strong candidate for a raster image, such that the relaxed label matrix includes three different labels corresponding to continuous tone information, periodicity information, and white / black information. It will include at least one label, and the context rules in its mitigation module will include rules to extend this label.

The context rules for finding areas containing halftone information can be similar to the previously described context rules FILL and EXPAND (FIG. 3) intended to find continuous tone regions. In this case, instead of the target label U, a label indicating the halftone information will be used.

In the example shown in FIG. 7, a border analysis module 40 is added to improve the match between the segments (P and T in FIG. 4B) and the actual border 20 of the photographic area of the document.

For example, the size of the subwindow (ie, 1/4, 1/2,
Boundary analysis is performed by vertically and horizontally moving the sub-image grid by a fixed percentage of 3/4), and by repeating the initial labeling and relaxation procedure on the moved grid. Is also good. Furthermore, a comparison of the different results provides more detailed information regarding the actual location of the photographic area boundaries.

Optionally, the boundary analysis for the vertical and horizontal boundaries is
The vertical and horizontal boundaries of the photographic region may each be limited to those portions of the document that are expected to be present.

In an alternative method, a boundary analysis may be performed by further analyzing a particular target area centered on the label transition coordinates in the relaxed label matrix. For example, the target area can be further subdivided into sub-windows that provide higher resolution than that used in the initial labeling stage, and each sub-window can be further classified as a bounded sub-window or a non-bounded sub-window. is there.

Analysis of the target area may be limited to isolated locations on the transition line in the relaxation label matrix. If the position of the boundary line is precisely defined within these target areas, the exact position of the entire boundary line may be found by extrapolation.

Although specific embodiments of the present invention have been described above,
Those skilled in the art will be able to devise various changes that fall entirely within the scope of the inventions specified in the claims.

For example, the context rules mentioned with respect to FIG. 3 can be implemented using a matrix of 3 × 3 sub-images, which is larger than the aforementioned size.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a general configuration of an automatic segmentation system, FIG. 2 is a diagram showing a tree classifier used in an initial labeling stage, and FIGS. 3 (A) to 3 (E) are shown in FIGS. The context rule diagrams used in the mitigation step, FIGS. 4 (A) and 4 (B) show an example of the initial label matrix and the resulting relaxed label matrix, and FIG. 5 shows the hardware of the system according to the invention. Block diagram of an example of a wear device,
6 and 7 are block diagrams of an improved example of the present segmentation system. 10 ... initial labeling module, 12 ... relaxation module, 22 ... scanner, 24 ... histogram unit, 26 ... classifier, 28 ... feature extractor, 30 ... tree classifier, 32 ... context processor, 34 Processing module, 36 Buffer, 38 Periodic module.

Claims (17)

(57) [Claims] 1. A document area (T, P) containing various types of image information such as white / black images, continuous tone images and the like.
Automatically segmenting a scanned document in an electronic document processing device to classify a scanned image representing a whole document into sub-image matrices. And a labeling means for analyzing the information contained in each sub-image and assigning initial labels (BW, BIM, BG, U) to said sub-images in order to obtain an initial label matrix thereby. And relaxing means for relaxing the initial label matrix by transforming the labels according to context rules, with the aim of obtaining a uniformly labeled segment pattern representing the document area of the document area. Adapted to select an initial label from a first set of labels (BW, BIM, BG, U); The moderator is adapted to convert the initial label to a moderated label (T, P), and the moderated label is selected from a second set of labels that is less in number than the first set. System. 2. The context rule executed in said mitigation means comprises rules for extending some (BW, U) of said initial labels and removing other initial labels (BIM, BG), The system of claim 1, wherein the extended label is also finally identified using the relaxed label (T, P). 3. The context rule: a) where n is a predetermined number, BW is a predetermined initial label, and G is a predetermined subset of the initial label set, If at least n elements in the predetermined array (A ') of matrix elements in the initial label matrix have a label BW and this array does not include a label from group G, then this array B) convert all the labels in B into a predetermined array of matrix elements in the initial label matrix (A), where m is a predetermined number and U is a predetermined initial label. '), If at least m elements have a label U, then convert all labels in this array to U, c) c1) the vertical where label U intersects When forming horizontal and horizontal continuations, the entire rectangle having these continuations in length and width is filled with the label U. c2) In order to maximize the area to be filled with the label U, the vertical and horizontal continuations are formed. Check all combinations, c3) height of the maximized area when U and BW are predetermined initial labels and the minimum height and minimum width are predetermined numbers. From at least one rule having one of the following structures: if is less than the height minimum element, or if its width is less than the width minimum element, convert all labels in this area to BW. 3. The system according to claim 2, comprising: 4. If the context rule further comprises: c4) if the rectangle referred to in c1 includes a number of labels U greater than a predetermined minimum number of U, the entire rectangle is filled with the label U; c5) When the minimum and maximum are predetermined numbers, the rectangular shape referred to in c1 has width / height> minimum and width /
4. The system according to claim 3, comprising a rule of construction: if the condition height <max is satisfied, the entire rectangle is filled with a label U. 5. The system according to claim 3, wherein the predetermined arrangement (A ′) to which the context rules (a) and (b) are applied has a size of 3 × 3 sub-images. 6. The method of claim 3 wherein said mitigation means comprises a plurality of steps for modifying said initial label matrix stepwise by applying each of said context rules at least once according to a predetermined order. 6. The system according to claim 5. 7. A context rule having said structure (a) is applied before a context rule having said structure (b), and said structure (b) is preceding a context rule having said structure (c). 7. The system of claim 6, wherein context rules apply. 8. If the context rule is that a given matrix element is surrounded by elements having the same label X immediately adjacent to the top, bottom, right and left, 8. A label according to claim 6 or claim 7, wherein the label also comprises a local rule ordering to be converted to X, and wherein the local rule is applied immediately before each of the rules having the structure (a), (b) or (c). System. 9. The method of claim 1, wherein each subimage has a size of at least 16
A system according to any one of the preceding claims, wherein the system is x16 pixels, preferably 64x64 pixels, and the scanning resolution is greater than 1000 dpi. 10. The system of claim 9, wherein said labeling means comprises a histogram unit for generating a gray level histogram for each of the sub-images. 11. The method according to claim 1, wherein said labeling means comprises a tree classifier for determining a label to be assigned to a given sub-image by examining characteristics of the input data according to the tree structure. A system according to claim 1. 12. The system according to claim 1, wherein said labeling means comprises means for detecting rasterized or dithered information. 13. The system according to claim 1, comprising a context rule for finding halftone regions. 14. A method according to claim 1, further comprising means for detecting rasterized or dithered information only in a photographic area already segmented by said relaxation means. system. 15. The system according to claim 1, comprising a plurality of labeling means for processing data from a plurality of sub-images in parallel. 16. The apparatus of claim 1, further comprising a boundary analysis means responsive to an output of said mitigation means for determining a boundary position of the segmented region with higher resolution.
The system according to any one of claims 15 to 15. 17. A document region (T, T) containing various types of image information such as white / black images, continuous tone images and the like.
P) A method for automatically segmenting a scanned document in an electronic document processing device to classify a document, wherein a scanned image representing the entire document is subdivided into a matrix of sub-images and an initial label matrix The information contained in each sub-image is analyzed to give initial labels (BW, BIM, BG, U) to the sub-images for the purpose of obtaining the same, and a uniform label representing the various document areas is provided. The initial label matrix is relaxed by transforming the labels of the individual matrix elements according to the context rules to obtain the segment pattern given the first label (BW, BIM, BG, U), and wherein, in the mitigation phase, the initial label has a smaller number of labels than the first set. Wherein the be converted to relaxed labels being selected from (TP).