JP2615051B2 - Document image area division and identification method - Google Patents

Description

Description: TECHNICAL FIELD The present invention relates to division and recognition of a region of a document image.

2. Description of the Related Art In order to "understand" a document image ("obtain information sufficient to answer any structural or semantic questions related to the document image"), the structure of the document ("text,
Understanding the structure of titles, figures, tables, etc. ") is the first step. Further, in the document image processing system proposed by the present applicant, it is required to define an abstracted image processing operation at a so-called higher level so as to enable expression of a more advanced image processing operation. . From these viewpoints, it can be said that the area division (identification) operation is one of very important image processing operations.

Below is a summary of conventional segmentation (however,
Maeda, Matsuura, Nambu (Mitsubishi): "Structural analysis of handwritten documents" IEICE General Conference (1986) 1517, density characteristics of horizontally long small rectangular areas Are used to combine character lines and blocks. Hino, Fukuda, Tabata (Hitachi): "Office Work Station by Multimedia Processing (Part 2)-Separation and Extraction Method of Document Structure-" National Convention of IPSJ (Showa 61)
3K-2, and Iwaki, Kida, Arakawa (NTT): The "functionally distributed document recognition system" PRU86-32 extracts character lines from the adjacent state of the circumscribed rectangle of the black connected component. In addition, we make approximate division using ruled line and blank space,
Akiyama, Hagita, Masuda (NTT): "Automatic reading of unknown format documents" PRU86-33 shows a method for identifying areas based on features such as periodicity of the marginal distribution and line density. In these schemes, the following three feature values are used to identify the category of the region: (1) Marginal distribution feature: Indicates the positional spread and periodicity of black pixels.

(2) Line density feature: Represents graphic complexity (3) Bounding rectangle feature: Represents the size of each element However, knowledge of how to use these feature amounts is stored in a program as a processing procedure. It has a disadvantage that it cannot be flexibly coped with a change or addition of a processing method.

On the other hand, systems that express processing procedures using production rules, which are AI techniques, have also been proposed. Iwaki, Kubota, Toshiro, Arakawa (NTT): "Study on introduction of production system in character / graphic separation processing" PRL83 −6
3. Here, a system is adopted in which a series of rule sets is activated every time one feature value is calculated, and the result is written to a blackboard. However, even in this system, the outlines of the division processing method and the area identification method are fixed.

In order to process multimedia data, it is desirable for the system to automatically perform media separation by document image understanding. It is important to divide the input document image into regions and identify the categories in order to understand the structure of the document image. The present applicant has proposed a method for accurately and rapidly performing this region division and identification, but the method is a flexible method in which the global structure is grasped by extracting a white region from an image pyramid, and the division process and the identification process are separated. The present invention is based on the introduction of a certainty factor for a judgment method and a feature amount and a method of synthesizing the same. Thus, in the above-described method, certainty factors for each category are empirically given to each feature amount and used for the region identification processing. However, when there are a plurality of input documents and the type of the input document is switched to another, if the assignment method of the certainty is fixed, there is a possibility that correct identification cannot be performed. Therefore, it is necessary to flexibly change the method of assigning the certainty according to the change in the type of the input document.

Objective The present invention has been made in view of the above-mentioned circumstances, and based on a dialogue with an operator, whether the identification result is correct or not,
The purpose is to change the function that gives certainty, that is, to have a learning effect.

Configuration In order to achieve the above object, the present invention performs region division on an input image using an image pyramid, and extracts a plurality of feature amounts for region identification from each divided layer,
By synthesizing the certainty factors assigned to the respective feature amounts, a more reliable classification result can be obtained based on the classification result of the input image in the region division and classification method of the document image for identifying the category of the region. The value of a function for assigning certainty to each feature amount is changed. Hereinafter, a description will be given based on examples of the present invention.

The area division and identification method of a document image to which the present invention is applied has the following two features.

(1) Region division by global information using an image pyramid.

(2) Complete separation between feature quantity extraction processing and area identification processing.

In particular, in (2), certainty is brought into the determination of the category of the area, and a weighted average or a calculation method based on the Dempster-Shafer theory is employed to make a final determination. Also,
It is also possible to provide detailed analysis means such as backtracking or weight change when reliable determination cannot be made.

FIG. 1 is a diagram for explaining the overall processing flow of the present invention, FIG. 2 is a diagram for explaining an image pyramid, and FIG. 3 is a diagram for explaining a conversion rule. Reference numeral 1 denotes an original image, which is intended for a document image in which character lines, tables, figures, photographs, and the like are mixed. This is input from a scanner (2), and preprocessing such as skew correction is performed (3). The input image is expressed as a set of binary data of 0 (white) and 1 (black). An image pyramid is created from this data (4). Is a kind of scaling the image pyramid, as shown in FIG. 2, the image viewed P ₂ since i.e. P ₁ viewed from afar, corresponding to viewing P ₃ from P ₂₀ (coarse-grained) . This makes it easy to grasp the structure of the image of the entire page. FIG. 3 is a diagram for explaining rule conversion in the image pyramid. As shown in FIG. 3, looking at four adjacent pixels (FIG. 3 (a)), if a certain condition is satisfied, four pixels are set. The black one
And the four black (or white) pixels are newly regarded as one black pixel (FIG. 3 (b)). In this example, if any one of the four is black, it is converted to black. Using such an image pyramid, a rectangular white area (blank portion) is extracted from the original image (5). According to the above-described creation method, it can be said that the portion of the upper layer that is a white pixel is all the layers below it, and the corresponding region is a completely white region (not including the black pixel). In this example, a method is used in which white rectangular areas are extracted in order from the upper layer using layers from the uppermost layer to a certain level. The image from which the white area has been extracted has several areas (including black pixels inside)
The boundary of this area is extracted (6). In order to extract the white region, there are as many region division methods as the number of layers of the image pyramid, depending on which layer is used. Under certain conditions, it is first determined which divided layer (hereinafter referred to as a divided layer) is to be subjected to area identification (to be classified into categories). In this example,
Among divided layers in which the number of regions having a size equal to or more than a certain value is equal to or more than a certain number, a divided layer using the lowest layer is analyzed first. When the divided layer to be processed first is thus determined, an original image region corresponding to each region in the layer is cut out, and some feature amounts are calculated for each cut out region (7). In this example, the following 12 types of feature values are calculated.

Feature amount: The following 12 image features are used to identify regions (text, titles, tables, figures, ruled lines / outer frames, etc.).

1. Size distribution of the circumscribed rectangle of the black connected component: [size; length of the diagonal of the rectangle] ・ Maximum value ・ Average value ・ Mutation rate (%)… (Standard deviation / Average value) × 100 ・ Maximum area ratio Size: The size value that has the largest area ratio as a result of adding the area for each rectangle size.

Character density (%): Area ratio when black concatenation of a certain size (set to 1 to 70) is regarded as a character.

2. Peripheral distribution of black pixels-Existence of periodicity due to space between character lines-Existence of columns: vertical or horizontal alignment 3. Existence of straight lines-Number of horizontal straight lines-Number of vertical straight lines 4. Frame (frame) 5. Average linear density: A value obtained by normalizing (× 100 / S) the number of intersections with the stroke when a perpendicular line is drawn from each point on the X and Y axes by the circumscribed rectangular area S of the region.

6. Degree of expansion: A / W ² [A; area of polygon area] W; polygon completely disappears by degeneration operation (here, operation of erasing black pixels in which white pixels are present in the vicinity of 8-connections) Value (= width) of the number of steps required for (2) (= width)] For each feature value as described above, if the value of the feature value is large, it is certain that the region belongs to each category (confidence) Degree) is given and step 8
With a certainty factor assignment. The assignment method of the certainty factor is empirically determined, for example, as follows.

Confidence Settings ... to the value d _i of the feature amounts, functions to provide a confidence (0-100) for each region Category alpha [g _i (d _i,
α)] was heuristically determined as shown in Tables 1 to 4.

However, it is assumed that the end point of the section is included in the section that appears first in the table.

Peripheral distribution of black pixels ZERO_PERIOD (label_poly, direction): existence of periodic zero (white) area (Table 5) Existence of straight line: Number of horizontal and vertical straight lines (Table 6) Existence of a frame (Table 7) Existence of columns (Table 8): Horizontal writing: presence / absence of vertical character alignment (column) Vertical writing: presence / absence of horizontal character alignment (column) COLUMN (label_poly, intensity) (Table 8) For example, in the case of horizontal writing, FIG.
The column 2 is shown in FIG. 5 (b), and the column 3 is shown in FIG. 5 (c) or (d).

Character density (Table 9) Average linear density (Table 10) Elongation REDUCE (label_poly, elongatedness) (Table 11) For categories, in this example, the text, title, table,
Although the figure, the ruled line / frame, and the other six types are considered, it is also possible to consider a gray image such as a photograph. In terms of a symbol, for a feature amount i, the degree of certainty that the region belongs to the category α according to the value d _i of the feature amount i ▲ g ^α _i ▼
(D _i ) [0 ≦ ▲ g ^α _i ▼ (d _i ) ≦ 100, i = 1,..., N / α = 1,
, M] is assigned in step 8.

Since the certainty assignment described above is fixed to one type, when the input document is switched to another one, it is necessary to assign an appropriate certainty accordingly. There is. The present invention, through the interaction with the operator,
The system recognizes whether the identification result is appropriate or not, and attempts to change the assignment of the certainty factor (function system ｇg ^α _i ▼ (d _i )) accordingly.

The specific method will be described below. Feature i
Takes a value d _i , the likelihood (confidence) that the region belongs to the category α is determined by the step function ▲ g ^α _i ▼
(D _i ). This example is shown in Table 2. The size in Table 2 is the length of the diagonal line of the minimum rectangle circumscribing the black connected component. When the present invention, the correct answer categories is section belongs d _i at alpha ₀ which was Dj (the total number of intervals and the l),
Since ▲ g ^αo _i ▼ (d _i ) is a constant value in the section Dj, ▲ g ^αo
_i ▼ (Dj)], ▲ g ^αo _i ▼ (Dj) becomes △ ▲ g
If the maximum value is obtained among ^αo _i ▼ (Dj)｝ (j = 1,..., l), the assignment of the certainty factor is determined to be valid, and the functional system ｛▲ g ^α _i ▼ (d _i ) ｝ Does not change. However, ▲ g
^{If αo} _i ▼ (Dk) (k ≠ j) is the maximum value, then ▲ g ^αo _i
▼ (Dk) is reduced by h (for example, h = 5 (%)), and ▲ g
^αo _i ▼ (Dj) is increased by h. However, 0 ≦ ▲ g ^αo
_i ▼ (Ds) (s = 1,..., l) ≦ 100.
With such a change, the certainty factor certainty distribution {｛g ^αo _i ▼ (Ds)} (s = 1,..., L) indicates that the actual value of the feature i is d _i ∈Dj and the maximum value is It changes in the direction you take.

To explain a specific example with reference to Table 2, it is assumed that the feature value i is the average value of the size, the correct answer category α ₀ is the text, and the feature value d _i = 30. At this time, {▲ g ^αo _i ▼ (Ds)} =
{10,95,10,0}. The interval Dj including d _i (= 30) is (2
It is a _{^{6,41), ▲ g αo i ▼}} (Dj) = 10 is {▲ g ^.alpha.o
_i ▼ (Ds)｝ is not the maximum value, so that ▲ g ^αo _i ▼ (Dj) → ▲ g ^αo _i ▼ (Dj) + b (h =
5) = 10 + 5 = 15 (%), The maximum value is Becomes In the above _{^{description, ▲ g αo i ▼ (d}} i) is _{^{{▲ g αo i ▼ (Ds}} )} (s = 1, ..., l) when a maximum value in the function system {▲ g ^.alpha.o _i ▼ was with (d _i)} does not change the, in accordance with the frequency ^{_{▲ g αo i ▼ (d i}} ) and the large, other ▲ g ^.alpha.o _i ▼ value of (Ds) (s ≠ j) Making it smaller is also effective.

By combining the certainty factors obtained as described above with respect to all n feature amounts, it is possible to estimate the certainty that the region belongs to the category α (step 9). There are the following two methods for this combining method.

[Method 1] Calculation method using weighted average-Confidence (0 to 0) that target region belongs to category α
100): 0 ≦ a _i (α) ≦ 1 0 ≦ g _i (d _i , α) ≦ 100 i; feature quantity of type d _i; characteristic amount value n; the number of feature quantity _{g i (d i, α)} ; confidence belonging to the category alpha when the value of the feature amount i is d _i, however, g _i , _j (d _i , d _j , α) also need to be considered in the future.

The category that takes the maximum value of the certainty factors synthesized in this way is the final result of the area identification. Degree ”is defined.

Determining degree of area identification (0 to 100): In FIG. 6, H indicates the degree of determining area identification. Find more. However, m; number of categories in the area] [Method 2] A calculation method based on the combination rule of the basic probabilities of Dempster and Shafer.

In this method, the final decision result is classified into the following three cases according to the values of the upper and lower bound probabilities: [Classification of decision result] i) The region is composed of a single category, and the category can be specified. .

ii) The region is composed of a plurality of categories, and the candidate categories can be specified.

iii) Although the region could not be identified, the candidate category with a certain degree of reliability can be presented.

The above three cases are respectively based on the following three determination rules.

[Determination Rule] The upper limit probability (%) of the region category α is P1 (α), the lower limit probability is Cr (α), and appropriate threshold values γ ₁ and γ ₂ [where 0 <γ ₂ <50 <γ ₁ <100 (%)] is set.

i) If there is a category α such that P1 (α)> γ ₁ and Cr (α)> 50 (see FIG. 7 (a)), the target region consists of a single category α.

If, P1 (β)> γ ₂ categories set satisfying {beta}
Everything is mixed.

(in _{α)> γ 1, [α} ; give max of P1] Cr (α) ≦ 50 ( FIG. 7 (c) iii) P1 reference), then category set that satisfies P1 (β)> γ ₂ { β｝
All are considered as the leading candidate categories.

Hereinafter, each of these rules will be described.

[Explanation of Judgment Rules] i) If there is a category α that satisfies Cr (α)> 50,
For any category β where β ≠ α, Holds, such a category α is uniquely determined, and both Cr and P1 take the maximum value of (true) in the category α. In addition, (Especially the latter) means that evidence that affirms that the target area belongs to the category α is more powerful than evidence that denies it.

P1 (α)> γ ₁ becomes a reference when it is provided in order to exclude the case of ii), which is also to be satisfied at the same time Therefore, there is a difference greater than γ ₁ -50 between the strength of the evidence that affirms that the target area belongs to the category α and the strength of the evidence that denies it.

Thus, the absence of a category with a certain degree of P1 (γ ₁ ) means that there is not a small amount of evidence supporting other categories. It is considered that this is because images belonging to several categories are mixed, and the characteristics of each are extracted.

iii) The state in which the preconditions are satisfied indicates a lack of information, and it is considered that the state moves to the state i) if the evidence supporting the category α increases sufficiently. Also, considering that it is dangerous to specify a category only from the information of P1, a slightly looser standard [γ ₂
= 10 (%)]. Another threshold is
γ ₁ = 70 (%) is set.

Which category each region belongs to is determined by the above two methods, but it cannot be determined (case of certainty lower than a certain value in method 1 / case of case iii) in method 2 or 2 in method 2 A detailed analysis such as re-weighting or using the next lower layer is performed on the area determined to have two or more categories mixed (11). further,
At 12, small areas are combined in accordance with rules in adjacent large areas. For example, "a character area having a small area surrounded by a figure area is regarded as a character in the figure and combined with the figure area".

Effects As is clear from the above description, according to the present invention, it is not necessary to change the function form that gives certainty for each input document. There are advantages such as being able to handle many types of document images because it is automatically changed to a more plausible way of contact.

[Brief description of the drawings]

FIG. 1 is an overall flow chart for explaining an embodiment of the present invention, FIG. 2 is a chart for explaining an example of an image pyramid, and FIG. 3 is a chart for explaining an example of a conversion rule. FIGS. 4 (a) to 4 (d) are diagrams for explaining the operation principle of the present invention, FIG. 5 is a diagram for explaining an example with columns in the case of horizontal writing, and FIG. FIG. 7 is a diagram for explaining the degree of determination of area identification, and FIG. 7 is a diagram for explaining an example of a determination rule.

Claims (1)

(57) [Claims] An input image is divided into regions using an image pyramid, a plurality of feature amounts for region identification are extracted from each divided layer, and a certainty factor assigned to each feature amount is synthesized. Thus, in the method of segmenting and identifying a document image that identifies the category of the area, the value of a function that assigns certainty to each feature based on the identification result of the input image so as to obtain a more reliable identification result And a method for segmenting and identifying a document image.