JPH096907A - Logical discrimination method of document element - Google Patents

Abstract

PROBLEM TO BE SOLVED: To logically identify document elements in a composite column document picture. SOLUTION: Main white area allowable structure (pattern) dividing the columns of a document picture is previously defined by using a normal expression based upon a cross of main white (background) areas. After extracting horizontal and vertical main white areas from the document picture, the sequence of the extracted main white areas is compared with a finite state machine (automaton) by approximate normal matching to determine a set (i.e., a pattern) of main white areas most close to a required pattern. Consequently a column layout is determined and document elements can logically be identified.

Description

Detailed Description of the Invention

[0001]

FIELD OF THE INVENTION The present invention relates generally to a method and apparatus for extracting a major white region from a composite column document image to identify columns of document elements.

[0002]

2. Description of the Related Art A method for dividing a text area is disclosed in, for example, Baird.
"Image Segmentation by Shape Directed Cover
s, "10th International Conference onPattern Recogn
ition, pp. 820-825, 16-21 June 1990. Many conventional document segmentation methods such as Baird assume that all elements (components) in a document are rectangular. However, considering that there are various kinds of document layouts in reality, this assumption is not always correct. Such a conventional method has a problem in dividing and identifying a non-rectangular element area. Further, in the above Baird, the white area of the document image is analyzed. However, the method of dividing the column of the document element by extracting and analyzing the main white area from the result of analyzing the white area of the document image is I couldn't see it. Here, the “white” area is an area that does not include any connected elements and is the background area of the document.

Japanese Patent Application No. 7-243212 discloses a division method for extracting a non-rectangular document element by extracting a main white area of an input document image and finding a closed loop of the main white area in the document. There is. However, as shown in FIG. 40, the document segmentation system of Japanese Patent Application No. 7-243212 may accidentally detect a white area that is structurally insignificant. In that case,
The problem arises that one document element is originally divided and identified as two or more individual document elements. In the example of FIG. 40, a single document element C is divided and extracted as two elements C-1 and C-2 by misidentifying the document division system area A as a main white area. Inadequate detection of major white areas in a document image is due to the fact that the document segmentation system does not always handle only acceptable column layouts that meet the criteria. To avoid such misidentification problems, the document segmentation system should be provided with a model of one or more allowed column layouts that defines the major white areas that segment the document elements within a document image column.

A structural model is a representation of a particular column layout that may exist in the original document. The structural model is either generated offline by the user or provided by the author of the original document, prior to the actual document segmentation process.

In the document segmentation system disclosed in Japanese Patent Application No. 7-243212, some loss occurs in extracting the main white area. In other words, the document segmentation system uses a fixed threshold to sort out the main white regions, so that all the main white regions actually existing in the document cannot be extracted. Specifically, as shown in FIG. 41, a main white region that is small in size but very important may be overlooked. In the example of FIG. 41, the small main white area E in the center of the page is not extracted because its height is smaller than the threshold height of the main white area in the vertical direction. As a result, in the document element extraction, the document element regions F and G that should be identified as two separate document elements are extracted as one document element. Therefore, there is a demand for a segmentation system that can properly match the main white region of the input document with the structural model to avoid data loss.

Co-authored by E. Myers and W. Miller, "Approximate Matching of Regula
r Expressions, "Bulletin of Mathematical Biology,
vol.51, No. 1, pp. 5-37, 1989 discloses a matching method for approximate regular expressions. If the sequence is A and the regular expression is R, then the sequence matching R with the highest score among all sequences matching R is found. This solution is the highest-scoring sequence match, or delete, insert, when converting A to R elements,
It is based on the minimum cost value of an edit operation such as replacement. Myers are also at least O
(MN) shows some algorithms for finding the best matching sequence. Here, M and N are the lengths of A and R, respectively.

[0007]

SUMMARY OF THE INVENTION The present invention provides a system for identifying document elements by analyzing only white areas in a document image, ie, background areas that contain no text.

The present invention also provides an apparatus for accurately and effectively segmenting a document element by comparing a main white region extracted from an input document image with a description of an allowable pattern of a column model document. The allowed column image can include a combination of several rectangular document elements,
Therefore, it may be non-rectangular. The document element division system of the present invention divides a document element into columns from an input document image based on a main white area pattern. The dominant white region pattern is based on the dominant white region type, sequence, and intersection.

In the present invention, a regular expression based on the intersection of the main white areas is used to predefine an allowable structure (pattern) of the main white areas that divides a column of a document image. After extracting the main white regions in the horizontal and vertical directions from the document image, the sequence of the main white regions extracted is compared with a finite state machine (automaton) by approximate regular expression matching, and the set of main white regions separating columns ( That is, the pattern) is determined. Thus, the document element segmentation system of the present invention uses the approximate matching method and outputs the closest match to the desired pattern in the main white region,
Data loss and misidentification can be avoided.

Once the document image has been divided into columns, the document elements of the input document image can be further processed. For example, based on a comparison of input document images,
Assign logical tags related to the matching column-structured document element to the corresponding document image document element,
Performs logical identification of document elements, extracts and processes divided document elements with an optical character recognition unit,
For example, output to a printer.

With the present invention, it is not necessary to analyze the portion of the document image containing the document elements to detect which connected elements form a coherent group, or document element. In the present invention, the image on the document is first scanned to produce an electronic or digital representation of the input document image. The main white area is a rectangular area of “white space” and has a predetermined minimum size. In this specification, the white space means a background area, and in the document image,
An area where no image or text exists. Since a normal document is one in which a black or color image is formed on a white background area, these background areas will be referred to as “white space”, but in the interpretation of the present invention, the document is formed on the color background area. Is formed, or even if text is formed on a black or color background with an outline, a color (or black) background area that does not include these images is defined as “white space”.

A document element is a rectangular area containing information such as a headline, text, graphic (graphic) in a document image, and is separated from each other by a main white area. An area containing a document element and separated from another area containing the document element by a main white area of a predetermined size is an individual document element.

One aspect of the invention is a method for logically identifying document elements of a composite column document image, the method comprising the step of identifying a major background area in the document image. Generating an ordered data string corresponding to the arrangement of the finite state machine, comparing the ordered data string with a finite state machine, and selecting a finite state machine from the at least one candidate path for the ordered data string. Determining the optimal path that best matches the column layout, and identifying the column layout based on the determined optimal path.

Other objects and effects of the present invention will become more apparent from the following detailed description of the preferred embodiments with reference to the drawings.

[0015]

1 illustrates a preferred embodiment of a document element segmentation system 100 of the present invention. The document element division system 100 includes a document white area extraction system 110, a main white area selection unit 120,
Memory 130, processor 140, string conversion means 150, comparison means 160, column layout identification means 1
70, a logical tag assigning unit 180, and a document element extracting unit 190, all of which are connected to each other via the bus unit 105. Also, one or more printers 2
00, scanner 210, user interface 22
0, remote interface 230, non-volatile memory 27
0 is connected via the bus means 105. The remote interface can connect to a LAN, wide area network (WAN), or another computer. As shown in FIG. 1, the document element identification system 100 is preferably implemented on a general purpose computer 300, but may be a dedicated computer, a microprocessor-based or microcontroller-based system, an integrated circuit such as an ASIC, a hard-wired such as a discrete device circuit. Electronic circuit,
It can also be implemented by a programmable logic device (PLD) such as a field programmable gate array.

FIG. 2 shows a document white area extraction system 110 of the document element division system 100 of FIG.
2 illustrates a preferred embodiment of As shown, the document white area extraction system 110 includes a connected component identifying unit 260,
It includes a bounding box generating means 250 and a main white area extracting means 240, all of which are connected by the bus means 105. First, the scanner 210 and the non-volatile memory 2
The document image data is input to the connecting element identifying unit 260 from the remote interface 70, the remote interface 230, the memory 130, and the like. The memory 130 may be an internal memory of the general-purpose computer 300, a floppy disk and a disk drive, a hard disk drive, a CD-ROM, an EPR.
It may be a well-known external memory such as an OM attached to the outside of the general-purpose computer 300. The document image data read by the scanner 210 may be temporarily stored in the memory 130 before being input to the connected component identifying unit 260. The document image data is input to the connecting element identifying means 260 in the form of a binary (binary) image or a digital signal of a plurality of bits. Each bit indicates whether a particular pixel of the document image is on (ON) or off (OFF).

Upon receiving the document image data, the connected element identifying means 260 detects all connected elements in the document image. FIG. 4 is an example of the document image 400. The connecting element 410 is composed of a series of adjacent "on" pixels (black pixels) surrounded by "off" pixels (white pixels). Systems for detecting connected elements 410 in document image 400 are well known in the art and will not be described here.

When the connected element 410 of the document image 400 is detected, the bounding box creating means 250 creates a bounding box 420 for each connected element 410. As is well known in the art, bounding box 420 includes connecting element 410.
Is the smallest rectangular frame that completely encloses. Systems for generating bounding box 420 from connected elements 410 are also well known in the art.

The document image data including the bounding box information is sent to the main white area extracting means 240. As shown in FIGS. 5 and 6, the main white area extracting means 240 extracts main white areas in the vertical direction and the horizontal direction of the document image 400.

In the preferred embodiment of the document white area extraction system 110, the primary white area extraction means 240.
Is divided into two sections shown in FIG. 3, namely a vertical extraction section 241 and a horizontal extraction section 242. The vertical extraction unit 241 and the horizontal extraction unit 242 respectively include a primary (primitive) white area extraction unit 243, a comparison unit 244, and
An erasing means 245 and a grouping means 246 are provided, and these are connected to the bus means 105. The vertical extraction unit 241 and the horizontal extraction unit 242 include the same components and operate in the same manner.

As shown in FIG. 5, the horizontal extraction unit 242 first extracts the primary (primitive) white areas 430-1 to 430-10 extending beyond the threshold with respect to the horizontal primary (primitive) white area. , Which together form a major white region 460 that extends horizontally.
Similarly, as shown in FIG.
The primary white regions 430-11 to 430-19 that extend above the threshold value for the next white region are extracted, and these are collected to form a main white region 460 that extends in the vertical direction.

The formation of the horizontal major white area 460 from the horizontal white area is determined by the horizontal primary white area 430-1.
.About.430-10 adjacent regions are merged into a group according to specific rules to form one or more primary white regions grouped in the horizontal direction. Similarly, the formation of the vertical main white region 460 from the vertical white region also includes the vertical primary white region 4
This is done by merging adjacent regions of 30-11 to 430-19 into a group according to a specific rule, which results in one or more vertically grouped primary white regions. When the vertical and horizontal primary white areas are grouped and merged as described above, the horizontal primary white areas 430 and
Of the primary white areas that are grouped in the horizontal direction, a width wider than the threshold width 440 and a threshold height 45
Identify areas with height greater than zero. Similarly, in the vertical primary white region 430 and the primary white regions grouped in the vertical direction, the threshold height 4
Regions having a height greater than 50 'and a width greater than the threshold width 440' are identified. These areas will be identified as the major white areas.

As shown in FIG. 7, the identified major white area 460 surrounds the document element 470 and divides the document element into pieces. Thus, the size and orientation of the main white area 460 can be used to define the logical structure of the document.

Document white area extraction system 11 described above
0 is an example of various embodiments that can perform extraction of a white area, and is not limited to this.

Once the main white area 460 has been identified by the document white area extraction system 110, the horizontal main white area 460 and the vertical main white area 4 are identified.
The position of the intersection 480 where 60 intersects is specified. The main white area 460 that forms this intersection is then
Is converted into a one-dimensional data string by the string conversion means 150. Further, the main white area intersection 480 is classified by type and stored in the memory 130 or the non-volatile memory 270 together with the one-dimensional data string obtained by the conversion. The comparison means 160 refers to the stored information and makes a comparison. The type of main white region 460 that extends between two intersections 480 is the intersection type of the end of the main white region and the main white region 46.
Classified or identified based on the type of intersection within 0. That is, in the string conversion means 150,
The location of a particular major white area 460 and the intersection 480 of this major white area 460 with another major white area 460 classifies and identifies each major white area 460 to identify the major white areas appearing in the document image. Generate a one-dimensional data string corresponding to a sequence according to the rules. In the preferred embodiment, the major white regions 460 are concatenated according to the rules of top to bottom and left to right to produce a one-dimensional data string.

The comparison means 160 is the string conversion means 1
Compare the sequence of major white regions 460 as a one-dimensional data string generated by 50 with the allowed column layout defined in the structural model. It is preferable that a part of the comparing unit 160 is a finite state machine expressing an allowable (appropriate) document column division area.
The comparison means 160 matches the data string with the finite state machine using the approximate matching method.
The column layout identifying means 170 identifies the column layout of the document image based on the comparison result by the comparing means 160.

The logical tag assigning means 180 is based on the column layout of the identified matching document image, the document elements 470 that are between the major white areas 460 (ie, separated by the major white areas 460).
Add a logical tag to the area. The document element extraction unit 190 uses the logically tagged document element 470.
To extract. As another configuration example, the document element extracting unit 190 may be used instead of the logical tag assigning unit 180 or may be used in combination with it. Logical tag assigning means 180 and document element extracting means 19
Details of No. 0 are described in Japanese Patent Application Nos. 7-243212 and 7-243213.

As shown in FIG. 7, document image 400 includes any number of major white areas 460. These major white areas have any number of intersections 480 and divide the document image into any number of document elements. Compares the spatial (geometric) relationship between the document elements of the input document image and the main white areas existing between the document elements and the spatial (geometric) relationship defined by the allowed column layout of the original document. By doing so, the document element 470 of the document image 400 is logically identified. If the geometric relationship between the main white areas 460 of the document image matches the geometric relationship of the allowed column layouts, the document image column layout of the document element 470 of the document image 400 will be identified. In the example of FIG. 7, columns 405-1 and 405-2 of the document element are specified. However, in order to identify the column layout of the document elements, a model that defines the spatial geometric relationship between the document element types must be supplied to the document element division system 100 in advance.

The structural model is a depiction of the structure of the original document that represents the allowed column layouts within the original document. In the example of FIG. 7, the input document image has a two-column layout, but the present invention is not limited to a two-column layout.
The system and method of the present invention can be applied to 3-column, 4-column, and more generalized n-column layouts.

In the two-column layout, the document image is
When a table, a description, etc. are inserted, it is necessary to consider three states. The first state is a state in which a figure, table, or supplementary sentence is contained in a column. In this case, the pattern of the main white area is not affected.

The second state is a state in which a figure, table, or explanatory text (hereinafter collectively referred to as an insertion portion) having a width wider than one column width and narrower than the entire document image width is inserted. is there. In this case, the text area on one side or both sides of the insertion portion is cut according to the size of the insertion portion. Due to the presence of the insertion portion, a different main white area pattern is formed and the main white area pattern is changed.

The third state is that the figure, table, and supplementary sentence have a width equal to the width of the entire document image, that is, a portion extending over both columns of a two-column document is inserted. Also in this case, the main white area pattern changes because different types of main white areas are formed depending on the insertion portion.

The document element dividing system 100 is based on the premise that the document elements inserted in a page do not overlap each other. Therefore, when a single element is inserted, all the states of the insertion pattern can be covered by combining the above three states.
Based on this assumption, across one or more columns,
A layout in which multiple document elements are inserted is also possible.

The space dividing the column is represented by an alternating sequence of horizontal and vertical major white areas 460 that intersect each other. Also, in a preferred embodiment,
Sequences can be matched even if there are no vertical paths. Further, before extracting the main white area 460 by the document white area extracting system 110, four margins, that is, an upper margin, a lower margin, a left margin, and a right margin are removed. In this case, after extracting the main white region, the four margins are replaced to form a closed loop of the main white region and the column of document elements is identified.

Whenever an insert is inserted in a column, at least one major white area 460 is also inserted to distinguish the insert from the surrounding text. When the insertion section is aligned with one of the vertical ends of the column, a main white area 460 is provided along the opposite vertical side to distinguish it from the rest. When the insertion portion is located at the center of the column, horizontal main white regions are inserted above and below the insertion portion to separate it from the text above and below. If the insert is aligned with the top of the column, a horizontal major white area is provided adjacent to the bottom of the insert, and if the insert is aligned with the bottom of the column, insert A horizontal main white region is formed along the upper part.

In the preferred embodiment, the location of the major white regions forming major white region intersection 480 allows for classification into the six types of major white regions shown in FIGS.

HL: Horizontal main white area that intersects only the left margin white area HR: Horizontal main white area that intersects only the right margin white area HF: At least one vertical major white area that does not intersect either the left margin or the right margin Horizontal main white area VC: Vertical main white area including at least a part of the vertical centerline of the document image VL: Vertical main white area located to the left of the vertical centerline of the document image VR: Vertical image of the document image Vertical main white area located on the right side of the center line In addition to these 6 forms, HLR: horizontal main white area intersecting both left and right margins on both sides of the column, ε: empty symbol is used.

The string conversion means 150 classifies the main white area 460 based on the above type. The vertical major white area 460 is classified by the horizontal location in which it is located. That is, VC is a vertical main white area at a position including the horizontal center (vertical centerline), VL is a vertical main white area 460 on the left side of the horizontal center, and VL is a vertical main white area on the right side of the horizontal center. Region 460 is classified as VR.
The horizontal main white area 460 is also segmented by its position, and if the horizontal main white area 460 does not cross the vertical centerline of the document at all, the horizontal main white area is erased by the string conversion means 150.

The string conversion means 150 also applies a threshold to the major vertical and horizontal white areas 460. As shown in FIG. 11, the HL main white area 4
60 intersects the VC major white region 460, but H
If the horizontal width of the L main white area 460 does not significantly exceed the width between the VC main white area 460 and the left margin (does not exceed the threshold value), this HL main white area 460 is set in the column. Treat as a separator that separates document elements. Therefore, this HL major white area 460 is incorporated into the sequence. Similarly, the HR major white region within the threshold that separates the elements in the column is also incorporated into the sequence.

In the document white area extraction system 110,
Once the main white area 460 of the input document image has been identified, the location of the intersection 480 between the horizontal main white area and the vertical main white area is located and the main white area 460 associated with this intersection is identified by the string conversion means. 150
Is converted into a one-dimensional data string by. The major white regions 460 extending between the two intersections 480 are classified according to their intersection type and end point position to produce a one-dimensional data string corresponding to the ordered sequence in which the major white regions appear in the document image. To do. In this embodiment, the main white regions 460 are concatenated from top to bottom and left to right to generate a one-dimensional data string.

Also, as shown in FIG. 10, if there are horizontal and vertical major white areas in which the topmost positions of each are very close to each other (within a threshold value) in the document image, the string The conversion means 150 treats the uppermost positions of these main white areas 460 as the same position. In the example of FIG. 10, the uppermost positions of the HF main white region and the VC main white region are regarded as the same position, and the HF main white region is first analyzed and incorporated into the one-dimensional data string according to the rule from left to right. Then, the VC major white region is analyzed. In this way, the main white region 460 is sequentially converted into a one-dimensional main white region sequence. One-dimensional string (ODS: one-dimensiona) generated from the document image of FIG.
l string) is represented by the following equation (1).

ODS = VL-VR-HF-VC-HLR (1) The document element segmentation system 100 can also process an iterative sequence of major white regions 460 as shown in FIGS. The one-dimensional data string corresponding to the ordered sequence of major white regions 460 in the document images of FIGS. 12 and 13 is given by the following equation (2):
It shows with (3).

ODS = VC-HL-VR-HL-VC-HL-VR-HL-VC (2) ODS = VC-HR-VL-HR-VC-HF-VL-VR-HF-VC (3) FIG. FIG. 14 shows an example of a finite state machine that can be used to compare two-column format document image divisions. The finite state machine determines if the input string belongs to the string set of the comparison model. As shown in FIG. 14, the column division sequence (that is, the data string that identifies the main white area 460 in the document image) is
Corresponds to the input string. The comparison string set corresponds to a two-column format allowable layout defined by the structural model, and in the present embodiment, the finite state machine of FIG. 14 becomes the structural model.

The transition from state to state in the finite state machine of FIG. 14 corresponds to the matching of the main white area determined by the comparison means 160 with the structural model. The main white area one-dimensional data string of the equation (3) representing the sequence of the document image shown in FIG. 13 and the transition of the finite state machine of FIG. 14 match. This matching is
"Start state, state 1, state 3, state 10, state 1
1, state 12, state 3, state 17, state 18, state 1
9, state 20, state 3, end state ”is generated. Therefore, the one-dimensional data string in equation (3) is a valid column segmentation sequence for the main white region of a two-column document image.

FIGS. 15-26 show the document image of a permissible (proper) column split sequence defined by the finite state machine of FIG. 14 in a two column layout embodiment.
Although FIGS. 15-26 show only a single-event permissible column splitting sequence based on a single insert, multiple events such as FIGS. 12 and 13 can also be treated as a series of additional major white region patterns. .

The comparison means 160 produces a cost matrix for choosing the best match between the finite state machine and the input string. Each row of the cost matrix is each character in the input string. The input string is a sequence of major white regions from top to bottom, left to right in this embodiment, so if, for example, the top positions of two vertical white regions 460 are the same:
The vertical main region located on the left is selected first. on the other hand,
Each column of the cost matrix corresponds to each state from the start state to the final (end) state of the finite state machine as shown in FIG. The number attached to each state functions as an identification name. The edit cost is the sum of the primary costs of edit operations such as "insert", "delete", and "replace". In this embodiment, the costs of the insert, delete, and replace operations are set to "-1". If the total edit cost is "0" as a result of the one-dimensional data string (or column division sequence), the column division sequence is valid and is considered to have matched. A path cost of "0" means that the path passes through the finite state machine completely from the start state to the end state. The cost of a path that does not go completely through a finite state machine is
Since the primary editing cost is defined as a negative value, the total cost of the column division sequence also becomes a negative value. In the present embodiment, the optimum path has the maximum cost, that is, it is interpreted as a value that is closest to "0".

In the present embodiment, these primary costs are predetermined by the document element division system 100 and are stored in the primary cost storage area of the memory 130. Moreover, the values of each primary cost need not all be equal. For example, the cost of the replace operation may be set to -2, which is twice the cost value of the insert or delete operation. After determining the overall cost matrix, the optimal path with the highest cost can be obtained by traversing the cost matrix from the last row to the first row.

In FIG. 40, V is added to the document image 400 before analysis.
It is an example of an insertion operation in which an L major white region has been extraly inserted and detected. In FIG. 40, the column division sequence generated by the string conversion means 150 is “VR-HL-VL-VC”. Comparing this column splitting sequence with the finite state machine of FIG. 14, the highest cost path generated is
State 1, state 4, state 5, state 3, end state ". Therefore, the VL major white region is an illegal major white region in the sequence, and the finite state machine indicates that the VL major white region is an improper insertion into the string.

FIG. 41 is an example of a missing operation detected by missing the vertical main white area E from the document image 400 before analysis. From here, the string conversion means 1
50 is a column division sequence "VR-HL-HR-V.
L "is generated. This column division sequence is shown in FIG.
Compared to the finite state machine of, the highest cost path generated is "start state, state 1, state 4, state 5,
"Missing, state 13, state 14, state 3, end state". That is, the highest cost path through the finite state machine shows a gap between states 5 and 13,
Shows the lack of a major white area in.

When the comparing means 160 determines the best sequence of the main white areas 460 in the document image as described above, the column layout identifying means 170 determines the document element of the document image based on the detected best sequence. Determine the column layout.

First, the column layout identifying means 170.
Determines if there are extra insertions, omissions, or substitutions in the best sequence. If the cost value of the best sequence is the maximum value “0”, the main white area of the best sequence is used as it is as the column division area. However, if there are extra insertions in the best sequence, as in FIG. 40, the insertion white areas must be deleted by identifying the matching major white areas around the insertion white area. Then, the remaining main white area is used as a spatial area for dividing the document image column. As shown in FIGS. 40 and 41, when the main sequence has a main white area inserted or missing, a matching main white area is detected to detect an extra white area or a missing white area. Replace is treated the same as missing. The document white region extraction system 110 then processes the document image again with varying thresholds (less than or greater than the initial setting) to extract missing major white regions from the best sequence, Remove extra major white areas.

If the document white area extraction system 100 fails to achieve the above operation, the column layout identification means 170 determines the second best path based on the cost value of the possible paths through the finite state machine. select.
In the present embodiment, this is applied only when there are a plurality of optimum paths having the same cost value. Then, the above-described operation is executed using the second optimum path selected instead of the first optimum path.

In the document white area extraction system 110,
If we were able to successfully accomplish the process of erasing the extra regions and extracting the missing regions using the second optimal pass, we would put four margins back into the main white region sequence on the second optimal pass, The document element of the document image is divided into column images. In this embodiment, the document image is divided into two document columns, and these columns may be non-rectangular.

The cost matrix generated from the input column partition sequence and the finite state machine is based on the approximate regular expression described in the above-mentioned Myers reference. In this embodiment, a cost matrix is generated using the method disclosed on page 20 of Myers to determine the optimal path.

27 to 29 and 30 to 33 are diagrams showing an example of generating a cost matrix based on another finite machine. The finite state machine of FIG.
Document images having the column layouts shown in 7, 18, and 19 can be handled. In this example, the edit cost values for insertion, deletion, and replacement are set to "-1" in advance. FIG.
The finite state machine of 7 has three obvious paths connecting the start and end states. That is, the paths are (1) state 1 → state 2 → state 4 → state 6, (2) state 1 → state 6, and (3) state 1 → state 3 → state 5 → state 6. Any major white region sequence that matches any of these three passes has a total cost of "0". This finite state machine is independent of the cost matrix.

FIG. 28 shows the lower path “state 0 →→” corresponding to the layout of FIG. 30 in the finite state machine of FIG.
It is a cost matrix of “state 1 → state 3 → state 5 → state 6”. According to the finite state machine of FIG. 27, the first transition in the column layout of FIG. 30 is ε. The row “0” of this cost matrix shows the cost when moving from the state 0 to each of the states 1 to 6. According to this, the cost of moving from state 0 to state 1 is zero. Row "1" shows the transition cost from state 1 to states 0, 2 to 6 in the input sequence "start, VR". Row "2" is the transition cost from state 3 to states 0-2, 4-6 in the input sequence "VR, HL". Line "3" contains the input sequence "VR, HL, V
"C" shows the cost of moving from state 5 to states 0-4 and 6. As shown in the cost matrix of FIG. 28, the transition cost of state 0 → state 1, state 1 → state 3, state 3 → state 5, state 5 → state 6 is 0, which means that the input sequence “VR, HL , VC ”.
Therefore, the path “state 0 → state 1 → state 3 → state 5 → state 6” is a valid path through the finite state machine of FIG. 27, and “VR, HL, VC” is the document layout of FIG. As shown by, the main white area 460
Is an effective column division sequence of.

FIG. 31 shows the column sequence "VR, V
It is a document image having "C". The document image of FIG. 31 can be analyzed by two different paths (path A and path B) of the cost matrix of FIG. Sequence "V
32 and 33 are diagrams obtained by analyzing the document image of FIG. 31 having “R, VC” using paths A and B, respectively.

FIG. 32 is an analysis using the path “state 0 → state 1 → state 3 → state 5 → state 6” (path A in FIG. 29) passing through the finite state machine of FIG.
L ”is missing from the path. Input sequence "VR,
In “VC”, when the path A passes through the finite state machine of FIG. 27, the total cost is zero (0) and the state 0 to the state 1
And further to state 3. However, as shown in row “1” of the cost matrix of FIG. 29, in the input sequence “VR, VC”, there is no transition from the state 3 of the finite state machine of FIG. The best transition to 5 is the transition at cost "-1". This is as shown in FIG.
Improperly missing state 5 (HL) from R, VC ", the best sequence is" VR, HL, VC "
Is shown. The best sequence “HL, VC” that compensates for the missing HL is the same cost “−1” as shown in rows 1 and 2 of the cost matrix of FIG.
, The transition cost from state 5 to state 6 will eventually be zero. Therefore, the total cost of the path A is "-1". Thus, by comparing the matching major white regions VR and VC of the proximity,
The column layout identifying means 170 identifies “HL” as a missing transition. On the other hand, FIG. 33 is an analysis of the VR-VC column separation sequence using the path “path of state 0 → state 1 → state 6” (path B of FIG. 29) passing through the finite state machine of FIG. As shown in row 1 of the cost matrix of FIG. 29, instead of moving from state 1 to state 3, the finite state machine stays in state 1 at cost "-1". This is because the input sequence "VR, VC"
It means that state 3 (VR) is improperly inserted, indicating that the best sequence is "VC". Also, as is clear from row 1 and row 2 of the cost matrix of FIG. 29, the transition from state 1 of row 1 to state 6 of row 2 is a transition between costs of the same value, so this transition cost is zero. is there. Therefore, it can be seen that "VR" is an extra insertion part in the path B. Comparing the path "VR-VC" with the cost matrix of Fig. 29, the state 1 in row 0 to the state 1 in row 1 is compared. The transition cost is -1, and the "VC" transition cost from state 1 in row 1 to state 6 in row 2 is 0. As a result, the total cost of pass B is −
Becomes 1.

In summary, FIGS. 28 and 29 show two cost matrices corresponding to the two column layouts shown in FIGS. 30 and 31, respectively, of the finite state machine shown in FIG. Comparing Figures 28 and 30, the sequence "VR, HL, VC" successfully matches the finite state machine and the cost of the optimal path is zero. On the other hand, FIG.
Comparing 9 and 31, there are two possible optimal paths, each with a value of "-1" in the matrix of FIG. Of these, in FIG. 32, HL is missing from the optimal path that obtains the sequence of FIG. 31, and in FIG. 33, an extra VR is inserted in the optimal path that obtains the sequence of FIG.

In this way, the document element 4 of the document column obtained by dividing the document image 401 before analysis based on the matching.
70 is a memory 13 together with the matched column representation
0, or output to the processor 140 for further processing. Examples of subsequent processing include classifying document elements by analyzing the structure of a document image column and logically tagging those document elements.

Next, referring to the flowchart in FIG. 34, a method of dividing the document element 470 using division of a composite column by matching the main white area pattern will be described. After starting the operation in step S200, the main white area of the document image is extracted in step S300. Once the major white regions have been extracted, the one-dimensional data string representing the major white regions of the input document image is determined in step S400 by analyzing the intersections of the major white regions.

In step S500, the data string representation of the major white area is input to the finite state machine as an input string. The output of the finite state machine is the optimal path that shows the best interpretation of the main white region extracted in the input document image. The finite state machine is a representation of the proper allowed major white region 460 extending between any two identified major white region intersections 480. The optimal path is based on a given editing cost of inserting, replacing, and deleting the major white regions from the path through the finite state machine, as determined by the cost matrix.
It is the lowest cost path through the states of a finite state machine.

In step S600, the identified optimal path is used to determine the column layout of the input document image. In step S700, the document element of the input document image is processed by dividing, logically tagging, or outputting the document element identified from the column layout. In step S800, the process ends.

FIG. 35 shows a document image main white area 46.
It is a flow chart which shows the detailed process of Step S300 which extracts 0. After starting the operation in step S300, the document image 400 is input in step S310. In step S320, the connected component 410 of the document image 400 is identified. In step S330, a bounding box 420 is generated for each connected element 410 identified in step S320. In step S340, the main white area 460 is extracted, and in step S350, the process returns to step S400.

FIG. 36 shows the detailed process of step S340 of extracting the main white region. Step S3
After the operation is started at 40, the primary (primitive) white area 430 is extracted at step S342. FIG.
As shown in, the primary white area 430 is a rectangular white space area between the bounding boxes 420. In step S343, the width and height of each horizontal primary white area 430 are compared with a threshold width 440 and a threshold height 450, respectively, and the width and height of the vertical primary white area 430 are determined with a threshold. Value width 440 'and threshold height 4
Compare with 50 'respectively.

The threshold width of the horizontal area is the document image 400.
Is set to 1/3 of the horizontal length of the document image (that is, the width of the document image), and the threshold height 450 of the horizontal region is set to a value exceeding the line spacing of the text (text) of the document image. preferable. On the other hand, the threshold height of the vertical region 450 '
Is set to 1/3 of the vertical length of the document image 400 (that is, the height of the document image), and the threshold width 44 of the vertical region is set to
It is preferable to set 0'to a value exceeding the line spacing of the text of the document image. Threshold height (450, 45
0 ') and horizontal and vertical primary white areas of a size exceeding the threshold width (440, 440') are identified as major white areas.

In step S344, the horizontal primary white area 430 having a width smaller than the threshold width 440 of the horizontal area is set.
And erase the vertical primary white areas below the vertical area threshold height 450 '. In step S345, the remaining primary white areas 430 are grouped into a main white area 460. There are many possible methods for grouping the remaining primary white areas into main white areas. For example, the method disclosed in Japanese Patent Application No. 7-243212 executes grouping into the main white areas.

In step S346, the major white area is erased if at least one of the vertical and horizontal sizes of the major white area is smaller than the corresponding vertical and horizontal thresholds. Alternatively, a method may be adopted in which only the main white areas whose both vertical and horizontal sizes are smaller than the corresponding thresholds are erased. In step S347, operation returns to step S350.

FIG. 37 is a one-dimensional data string (column division sequence) of the main white area based on the position and intersection of the main white area extracted from the document image.
It is a detailed process of step S400 for determining. After starting the operation in step S400, the main white area intersection 480 of the document image 400 is identified in step S410. In step S420, the type (HL, VC, etc.) of the main white area is detected based on the position of the intersection. In step S430, a sequence according to the rule of the main white area is detected based on the detected position and type of the main white area. In the preferred embodiment, the sequence according to the rules is a sequence of transitions from top to bottom of document image 400 and then from left to right of document image 400. When there are a plurality of main white regions at the same vertical position (height) in the document image, the determinant of the sequence shifts from the left edge to the right edge of the document image. However, if there is a horizontal main white area and a vertical main white area at the same vertical position in the document image, the horizontal main white area is given priority, and then the vertical main white area is incorporated in the sequence from left to right. In step S440, a one-dimensional data string representing an ordered sequence of major white areas of document image 400 is detected. In the present embodiment, the one-dimensional data string is a series of strings in which the types of main white areas are sequentially arranged according to a rule. In step S450, the process returns to step S500.

FIG. 38 shows the detailed process of the preferred embodiment of step S500. Step S500 compares a one-dimensional data string with a finite state machine (automaton) to determine a cost matrix for each of the possible paths that divide the document image into columns, and to identify the optimal path from the possible paths. It is a process to do. Step S5
Operation is started at 00, and at step S510,
Determine (or create) a finite state machine and define the allowed column layout. The finite state machine is preferably supplied as part of the structural model of the original document. In step S520, the finite state machine and the one-dimensional data string are compared for each pass that divides the document image and the cost matrix is determined using appropriate relative expression matching. In step S530, candidate paths through the cost matrix are evaluated and the optimal path is selected. The optimum path is selected based on the allowable column layout and a predetermined editing cost (insertion, deletion, replacement) for a possible path, and an appropriate path having a total cost of zero (0) is obtained through the document image. In step S540, the process returns to step S600.

FIG. 39 shows a preferred embodiment of step S600, which identifies the column layout of the document image from the optimal path. Starting in step S600, in step S605 all optimal paths with the maximum cost are searched. In step S610, the optimum path is checked to see if there are any paths that have not been evaluated.
If it is determined that all paths have been evaluated, step S
Proceed to 655, and return to step S700.

In step S610, if there is an optimum path that has not been evaluated yet, the process proceeds to step S615, the path that has not been evaluated is selected as the current optimum path, and the process proceeds to step S620. In step S620, the current optimum path is checked to see if there is an extra insertion as shown in FIG. If an extra insertion is detected, it is erased in step S625, and step S6
From 25, jump directly to step S670. In step S620, if there is no extra insertion in the current optimum path, the process proceeds to step S630.

In step S630, the current optimum path is checked to see if there is any omission as shown in FIG. If no omission is detected in the current optimum path in step S630, the process proceeds to step S660. If there is a missing part in the current optimum path, the process advances to step S635 to identify a matching main white region around the missing part. In step S640, the type of missing major white region is identified based on the adjacent matching major white regions identified in step S635. In step S645, the document white area extraction system 110
The document image is reprocessed with the reduced threshold to locate missing major white areas.

In step S650, the newly extracted main white area is checked to determine if the missing main white area is properly located. Step S65
If 0, it is determined that the missing major white area is not located properly, operation proceeds to step S610.
Return to If it is determined in step S650 that the missing main white region is properly located, step S6
Go to 70.

In step S660, the current optimal path is checked to see if there is a replacement. If there is no replacement, the process proceeds to step S670, while if there is a replacement, the replacement is deleted in step S665, and the process proceeds to step S635.

In step S670, four margins (margins) on the upper, lower, left and right sides are added to the main white area matched in the current optimum path. In step S675, the document elements divided in the main white area by the optimum path are extracted into columns. Then, in step S680, the process returns to step S700.

Although the present invention has been described based on the specific embodiments, it goes without saying that various substitutions and modifications can be made by those skilled in the art. The preferred embodiments of the present invention described above are merely examples, and the present invention is not limited thereto and various modifications can be made within the principle and scope of the present invention.

[0078]

According to the present invention, an approximate matching method with a finite state machine is used to output a sequence that is closest to a desired pattern in a main white area. False positives can be avoided. Therefore, even in a compound column document image that includes a non-rectangular document element column, the document column can be accurately and effectively extracted without erroneously extracting extraneous white space areas or overlooking important white space areas. Can be divided and identified.

[Brief description of drawings]

FIG. 1 is a block diagram showing a preferred embodiment of a document element division system of the present invention.

2 is a diagram showing a document element dividing system of FIG.
FIG. 3 is a block diagram illustrating a preferred embodiment of a document white area extraction system.

FIG. 3 is a block diagram showing a preferred embodiment of main white area extracting means in the document white area extracting system of FIG.

FIG. 4 is a diagram showing an example of a sample document image.

FIG. 5 is a diagram of a document image having an extracted horizontal primary white area.

FIG. 6 is a diagram showing a document image having an extracted primary white area in the vertical direction in the document image of FIG. 5;

FIG. 7 is a diagram of another document image having an extracted main white area.

FIG. 8 is a sample document image showing types of intersections of major white regions extending vertically.

FIG. 9 is an illustration of a sample document image showing types of major white region intersections that extend horizontally.

FIG. 10 is a diagram showing a sample document image and thresholding criteria for a main white area.

FIG. 11 is a diagram showing a sample document image and thresholding criteria for a main white area.

FIG. 12 is a sample document image showing a repetitive arrangement of document elements within a document column.

FIG. 13 is a sample document image showing a repetitive arrangement of document elements within a document column.

FIG. 14 is a diagram showing each state of the finite state machine of the preferred embodiment of the present invention.

15 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

16 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

FIG. 17 is a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

FIG. 18 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

19 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

20 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

FIG. 21 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

22 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

FIG. 23 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

FIG. 24 is a diagram of a sample document showing one of the major white area models defined by the state-to-state transitions of the finite state machine of FIG.

25 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

FIG. 26 is a diagram of a sample document showing one of the major white region models defined by the state-to-state transitions of the finite state machine of FIG.

FIG. 27 illustrates another sample finite state machine.

28 is a diagram of a cost matrix corresponding to the sample finite state machine of FIG. 27.

FIG. 29 is a diagram of a cost matrix corresponding to the sample finite state machine of FIG. 27.

FIG. 30 is a diagram of a sample document image of a major white area layout corresponding to the cost matrix of FIG. 28.

FIG. 31 is a sample document of a major white area layout corresponding to the cost matrix of FIG. 29.

32 is a diagram showing that there is a lack of a main white area as a result of analysis of the sample document of FIG. 31.

FIG. 33 is a diagram showing that there is an extra insertion of a main white area as a result of analysis of the sample document of FIG. 31.

FIG. 34 is a flow chart illustrating a preferred embodiment of a method for splitting a composite column document element by mapping a main white area pattern.

35 is a flow chart detailing a preferred embodiment of the step of identifying a major white region from an input image of the flow chart of FIG. 34.

FIG. 36 is a flowchart detailing a preferred embodiment of the main white region extraction step shown in FIG. 35.

37 is a flowchart detailing a preferred embodiment of the step of generating a data string representing a main white region in the flowchart of FIG. 34. FIG.

38 is a flow chart detailing a preferred embodiment of the step of processing a data string of the flow chart of FIG. 34.

FIG. 39 is a flowchart showing in detail a preferred embodiment of a step of identifying a column layout of an input document image corresponding to an optimum path in the flowchart of FIG. 34.

FIG. 40 shows an input image with extra insertion of structurally insignificant major white regions extracted by the document segmentation system.

FIG. 41 is an illustration of an input document image with structurally significant missing major white areas that were not extracted by the document segmentation system.

[Explanation of symbols]

 100 Document Element Dividing System 110 Document White Area Extraction System 130 Memory 140 Processor 150 String Converting Means 160 Comparing Means 170 Column Layout Identifying Means 180 Logical Tag Assigning Means 190 Document Element Extracting Means 400 Document Image 460 Main White Space 470 Document Elements

Front Page Continuation (72) Inventor Moody Jane USA 85721 Towson, Arizona University of Arizona Department of Computer Science Inside Science

Claims (1)

[Claims] 1. A method for logically identifying document elements of a composite column document image, the method comprising the step of identifying a major background area in the document image, the method corresponding to the placement of the major background area in the document image. Optimizing the ordered data string by comparing the ordered data string with a finite state machine and best matching the finite state machine among the at least one candidate path for the ordered data string A method for logically identifying document elements, the method comprising the steps of: determining a path of a document element, and identifying a column layout based on the determined optimal path.