JP2008217833A - Document image recognition system and storage medium for document image recognition program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a document image recognition system having a function of extracting a line and a column of a document image, which can extract a line and a column of a document image having horizontal lines and vertical lines mixed therein at high speed and accuracy. <P>SOLUTION: A document image skew detection section 1 extracts an aggregate of letter components referring to the document image, extracts horizontal and vertical line candidates, based on the extracted aggregate, estimates the reliabilities of the respective line candidates, extracts an aggregates of lines having a higher accuracy, based on the estimated line reliabilities, and estimates the skew, based on the result, using the arrangement of the letter components of the aggregate of lines having a higher accuracy. A layout analyzing section 5 extracts an aggregates of lines having their directions defined based on the aggregate of basic elements forming the document image, and executes column extraction using association between lines, and line extraction using a column as constraint to extract a line and a column. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a document image recognition apparatus and a document image recognition program for recognizing a document image by extracting character lines and columns in a document image such as a document read by an image scanner or a document received by a facsimile apparatus. It relates to the medium.

  In order to expand the target document of the document optical reading (OCR) engine, it is essential to have a layout analysis function for a longitudinal and lateral mixed document as seen in newspapers. The present invention provides a new technique of line extraction processing and column extraction processing for document image recognition as an elemental technology necessary for the layout analysis of this vertical and horizontal mixed document.

(1) Detecting document image tilt To read a general print document, it is first necessary to capture it as a document image using an image input device such as an image scanner. In order to use electronic filing or document recognition, it is necessary to detect and correct the amount of inclination of the document image.

  In the conventional tilt detection technique, the fact that characters are regularly arranged in a text area which is a main component in a document image is used.

  For example, Nakano et al. In the IEICE Transactions D, vol.J69-D, No.1, pp.1833-1834 (Non-patent Document 1) entitled “One Method for Document Image Tilt Correction”. Focusing on the fact that the reference line of the character string is almost constant as described, the lower end coordinate value of the character block is Hough transformed, and the peak value in the Hough space is detected to detect the inclination of the character string. A first method for estimating is proposed.

  Further, Mizuno et al. Extract connected components constituting a character as described in Japanese Patent Application Laid-Open No. Hei 7-92085 (Patent Document 1) entitled “Document Image Inclination Detection Device” and connect adjacent connected components to each other. Has been proposed to generate a temporary character line, and to estimate the inclination of the character string by obtaining a straight line in contact with the temporary character line.

  Saito et al., As described in Japanese Patent Laid-Open No. 2-170280 (Patent Document 2) entitled “Document tilt correction device”, temporarily corrects a document image while sequentially changing the tilt angle θ. A third method is proposed in which an angle θ that minimizes the area of a circumscribed rectangle including all black pixels in an image is obtained as an inclination angle.

(2) Layout analysis (row / stage extraction)
Conventionally, in an OCR apparatus or the like, the following method has been proposed as a method for extracting character lines and columns in a document image in which horizontal and vertical lines are mixed.

  For example, Enomoto et al. Project a character line of an input document in a certain direction as described in Japanese Patent Application Laid-Open No. 1-183784 (Patent Document 3) entitled “Document Image Processing Device”, and its peripheral distribution. A fourth method is proposed in which the column is automatically determined from the input document.

  Mizutani et al. Perform stage extraction using blank areas of components of an input document as described in Japanese Patent Laid-Open No. 5-174179 (Patent Document 4) entitled “Document Image Processing Device”. A fifth method is proposed.

Also, Hiramoto et al., As described in Japanese Patent Application Laid-Open No. 10-31716 (Patent Document 5) entitled “Character Line Extraction Method and Apparatus”, character line directions are mixed and character size is increased. A sixth method for extracting character lines from a document in which regions having different sheath pitches are mixed has been proposed.
Japanese Patent Application Laid-Open No. 7-192085 JP-A-2-170280 Japanese Patent Laid-Open No. 1-183783 JP-A-5-174179 Japanese Patent Laid-Open No. 10-31716 Nakano et al., "A method for correcting the tilt of document images", IEICE Transactions D, vol.J69-D, No.1, pp.1833-1834

  Many general printed documents include both vertical writing and horizontal writing. When a document is recognized, it is necessary to appropriately extract character lines and paragraphs.

  However, the above conventional method has the following problems.

(1) Problems concerning document image inclination detection The first method described above is based on the premise that the line direction is constant, and thus cannot be applied to a document in which horizontal and vertical lines are mixed like a newspaper. In addition, even for a document with a fixed line direction, the lower end of all characters does not necessarily exist on the reference line, so it is inevitable that errors are included. Furthermore, there is a problem that the Hough conversion process requires a huge amount of calculation.

  In addition, in the second method, in the case of a document in which horizontal and vertical lines are mixed, such as a newspaper, there is a case where a horizontal temporary line is erroneously extracted from a column of vertical lines. Large errors may occur.

  Although the third method is designed to detect the inclination of a mixed document of horizontal and vertical lines, angle detection is performed based on a small amount of information such as the area of a circumscribed rectangle including black pixels of the document image. Therefore, there is a problem that the accuracy of the detected inclination is unstable. In addition, it is necessary to repeat the process of rotating the image itself and extracting the rectangular area many times, which results in a huge amount of calculation.

(2) Problems related to layout analysis In the fourth method for extracting character lines and columns, the character lines in the columns are subdivided because the character lines are extracted in advance and the columns are extracted based on the extracted character lines. There is a problem that the irregularly shaped steps are subdivided.

  Further, in the fifth method, since a column is extracted using a blank area, there is a possibility that an erroneous column extraction may be performed in the case of a document having a portion where the step is narrower than the line spacing.

  This problem becomes a big problem in document images in which vertical and horizontal sentences are mixedly mixed. For example, as shown by a rectangular frame under the upper left photo area of the newspaper shown in FIG. 26, in the case of a document image in which the caption between the vertical article text and the horizontal picture caption is narrow, these are all set. As a single paragraph, there is a problem that characters in each line of horizontal writing caption are recognized as the first two characters of the vertical text.

  Further, in the sixth method, since the step region is extracted as preprocessing for performing high-precision line extraction, the irregular stage where the character lines in the stage are subdivided is subdivided. As a result, there is a problem that erroneous line extraction is performed.

  That is, in the prior art, [1] (basic element set) → row extraction process → stage extraction process → (layout analysis result) or [2] (basic element set) → stage extraction process → row extraction process → (layout analysis) The result is either bottom-up processing or top-down processing. All of these conventional technologies consider that line extraction processing and stage extraction processing are independent, and they are trying to achieve both row and stage extraction by executing both of them sequentially. There are major causes of spot generation.

  Based on such technical background, the present invention can extract lines and columns with high accuracy even when recognizing a document image having a complicated document structure in which horizontal and vertical lines are mixed. An object of the present invention is to provide a document image recognition apparatus.

  FIG. 1 shows a basic configuration example of the present invention.

  When a document image 2 to be recognized, which may include both vertical and horizontal lines, is input to the document image recognition apparatus 100, the document image inclination detection unit 1 first detects an inclination angle 3 for inclination correction. The layout analysis unit 5 extracts a basic row for the set 4 of basic elements extracted from the corrected document image corrected by the inclination angle 3 and repeats the row and column extraction so that the horizontal and vertical rows are obtained. The step structure of the mixed document image 2 is recognized, and the layout analysis result 6 is output.

(1) Document Image Inclination Detection The document image inclination detection unit 1 refers to the document image 2 and extracts character component extraction means for extracting a set of components constituting characters, and refers to the set of character components for horizontal and vertical line candidates. A line candidate extracting means for extracting a line reliability, a line reliability estimating means for estimating a reliability for each line candidate, a line extracting means for extracting a set of rows with high accuracy based on the row reliability, Inclination estimation means for estimating an inclination using the arrangement of character components in a set of high lines.

  Since the subsequent processing is performed using the set of character components extracted by the character component extraction means and the image is not directly manipulated such as rotation of the document image, the calculation amount can be greatly reduced. In order to extract horizontal and vertical line candidates by line candidate extraction means and to extract a set of horizontal and vertical lines with high accuracy by line reliability estimation means and line extraction means, In contrast, tilt detection can be performed. Furthermore, since the inclination estimation means estimates the inclination using only the character components constituting the line with high accuracy, it is possible to realize the inclination estimation that is strong against noise and highly accurate.

  By using such a technique related to the present invention, it becomes possible to detect the inclination of a mixed document of horizontal and vertical lines, which has been difficult with the prior art, at high speed and with high accuracy.

(2) Layout analysis (row / stage extraction)
The layout analysis unit 5 mutually executes basic line extraction means for extracting a set of lines whose direction has been determined from the set 4 of basic elements constituting the document, line extraction based on association between lines, and line extraction with a stage as a constraint. And a row / stage mutual extraction means for extracting rows and stages. Here, as the set 4 of basic elements to be input, for example, a black pixel connected component in the document image or a circumscribed rectangle overlapping rectangle of the black pixel connected component in the document image can be used.

  The major difference from the prior art is that the row extraction process and the stage extraction process are performed independently, such as the stage extraction process after the line extraction process or the line extraction process after the stage extraction process. According to the present invention, there is provided means for associating row extraction and column extraction, reflecting the row extraction result in the column extraction processing, and reflecting the column extraction result in the row extraction.

  By using the present invention, the row and column that cannot be achieved by the prior art are mixed by enhancing each other's result by the interaction between the row extraction and the column extraction, while using the row in which the row direction is determined as a basis. Extraction of lines and columns is also possible for document images with complex document structures that have irregular columns where the character lines in the columns are subdivided, and where there are narrower spaces between the lines. It can be performed with high accuracy.

  A program for realizing the above processing means by a computer can be stored in an appropriate storage medium such as a portable medium memory, a semiconductor memory, or a hard disk that can be read by the computer.

  According to the technology related to the present invention, it is possible to detect the inclination of a document image in which horizontal writing (horizontal row) and vertical writing (vertical row) are mixed, which is difficult in the prior art, at high speed and with high accuracy. According to the above, extraction of character lines and columns in a document image having an intricate document structure in which the character lines in the columns are subdivided, or there are portions where the spaces are narrower than the line spaces, is present. Can be performed with high accuracy.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[1] Hardware Configuration FIG. 2 shows a hardware configuration example to which the present invention is applied. The document image recognition apparatus 100 shown in FIG. 1 includes, for example, a CPU 20, a memory 21, a keyboard and other input devices 22, a display and other output devices 23, a hard disk and other external storage devices 24, as shown in FIG. An image scanner 25 for reading an image and a bus 26 for connecting them are configured. The functions of the processing means shown in FIG. 1 are realized by the CPU 20 executing a program stored in the memory 21 shown in FIG.

[2] Document Image Inclination Detection Unit FIG. 3 shows a basic configuration example of the document image inclination detection unit 1. The document image inclination detection unit 1 inputs the document image 2, detects the inclination, and outputs an inclination angle 3. The document image inclination detection unit 1 includes character component extraction means 11, line candidate extraction means 12, line reliability estimation means 13, line extraction means 14, and inclination estimation means 15.

  4 is a process flowchart of the document image inclination detection unit 1, FIG. 5 is a diagram for explaining extraction of character components, FIGS. 6 and 7 are diagrams for explaining extraction of line candidates, and FIG. 8 is an explanation of estimation of row reliability. It is a figure to do.

  The processing of the document image inclination detection unit 1 will be described in detail according to the flowchart shown in FIG. First, when the document image 2 is input in step S1 shown in FIG. 4, the character component extraction means 11 refers to the document image 2 and extracts a set of components constituting the character. In the extraction of character components, a set of connected components of black pixels in the document image 2 is extracted by a technique such as labeling (step S2). At this time, the circumscribed rectangle of each connected component is obtained and held, and in the following processing, only the coordinate value of the circumscribed rectangle is used, thereby reducing the amount of calculation.

  Next, the circumscribed rectangles of the connected components that overlap each other are integrated. This process is called overlapping rectangle integration. As a result of overlapping rectangle integration, a set of overlapping rectangles is obtained. Next, a set of character component circumscribed rectangles corresponding to character components is output by removing an extremely large overlapping rectangle corresponding to a region such as a figure or a table by taking a histogram of the size of the overlapping rectangle. .

  As shown in FIG. 5A, for example, for a character “day”, one connected component 40 of a black pixel is extracted as one character component. As shown in FIG. 5B, the character “old” is extracted as two character components because it is composed of connected components 41 and 42 of two black pixels. As shown in FIG. 5C, the character “o” is composed of connected components 43a and 43b of two black pixels. When these circumscribed rectangles 44a and 44b are obtained, they overlap each other. It turns out that it is a rectangle. Therefore, two connected components 43a and 43b are extracted as one character component by performing overlapping rectangle integration.

  The line candidate extraction unit 12 refers to a set of character components such as the example of the arrangement 50 of character components in the document image shown in FIG. 6 and extracts a set of horizontal and vertical line candidates (steps S3 and S4). . In addition, like the character component 51, one character component may belong to two line candidates of horizontal and vertical lines.

  More specifically, the line candidate extraction unit 12 outputs a candidate set of horizontal and vertical lines from the adjacent relationship between the character component circumscribed rectangles. In the case of a horizontal line, a part of the character component included in the horizontal line takes into account that the lower side is almost along the reference line of the line, and the lower side of the circumscribed rectangle of the character component is a certain range in the horizontal direction. The traversal candidates are extracted by sequentially searching for whether they exist within.

Consider a case where a horizontal character component adjacent to the circumscribed rectangle 60 of the character component shown in FIG. Paying attention to the lower side 62 of the circumscribed rectangle 60 of the character component, within a range of a predetermined angle (for example, ± 5 degrees) from the right end of the lower side 62 to the right, and within a range of a certain distance k · X _W (however, , K is a predetermined value, and X _W is the width in the X direction of the circumscribed rectangle), it is checked whether the lower side 63 of the circumscribed rectangle 61 of the adjacent character component exists. If the lower side 63 of the circumscribed rectangle 61 of the character component exists within this range, a right link is extended from the circumscribed rectangle 60 to the circumscribed rectangle 61. This search is repeated for each circumscribed rectangle. Note that the right link from each circumscribed rectangle is either zero or one so that the right link does not branch from one circumscribed rectangle. As a result, for example, as shown in FIG. 7B, a circumscribed rectangle group with a right link in the horizontal direction is obtained, and a set of traversing candidates is extracted by following the right link.

  Similarly, the vertical line candidates are obtained by attaching a lower link to a character component whose right side is adjacent to the lower side of each character component circumscribed rectangle and tracing the lower link.

  The row reliability estimation means 13 estimates the reliability for each row candidate (step S5). First, the likelihood of each line candidate is calculated quantitatively. For example, the amount by the length of the line that the number of character components included in the line candidate is a certain number or more, the amount by the proximity that the character components are arranged relatively densely, the size of the character component Quantities can be quantified from the amount of homogeneity, such as being almost the same.

  Next, there may be one horizontal row candidate and one vertical row candidate for each character component circumscribed rectangle. The direction flag of the character component is set horizontally or vertically. From this result, for each line candidate, the ratio of the character components belonging to that line candidate that have a direction flag that matches the line direction of that line candidate is obtained and used as the reliability of that line candidate.

  For example, as shown in FIGS. 8A to 8D, each character component is included only in the character component 70 in which the adjacent character component was not found in the processing of the line candidate extraction unit 12 and the vertical line candidate. There are a character component 71, a character component 72 included only in the horizontal line candidate, a character component 73 included in any of the horizontal lines and vertical lines, and the like. For each of these character components, the likelihood is quantitatively calculated using the measure of likelihood described above, and the “horizontal” or “vertical” direction flag is set. It should be noted that probable numerical values such as 0.5 vertical and 0.5 horizontal may be given to unclear ones rather than just flags. From this result, the ratio of direction coincidence is calculated and used as the row reliability. For example, as shown in FIG. 8E, the line reliability as a horizontal line is calculated by calculating the ratio of the character component in which the “horizontal” direction flag is set with respect to the horizontal line candidate of interest. Desired.

  The row extraction means 14 extracts a set of rows with high accuracy based on the row reliability (step S6). Here, a row candidate having a row reliability of a certain reliability or higher is output as a row with high accuracy.

  The inclination estimation means 15 estimates the inclination using the arrangement of the character components of the set of lines with high accuracy (step S7). Estimate the slope using the least squares method by fitting multiple straight lines from the coordinate values of the reference side (bottom side for horizontal rows and right side for vertical rows) of the character component circumscribed rectangle belonging to the set of rows with high accuracy .

  By using the document image inclination detection unit 1 as described above, it is possible to detect the inclination at high speed and with high accuracy even for a document in which horizontal lines and vertical lines are mixed. The inclination of the document image 2 is corrected based on the detected inclination angle 3, and the process proceeds to the next layout analysis process.

[3] Layout Analysis Unit FIG. 9 shows a basic configuration example of the layout analysis unit 5 according to the present invention. In the layout analysis unit 5 in the document image recognition apparatus 100 according to the present embodiment, the set 4 of basic elements of the corrected document image is input, line extraction and stage extraction are performed, and a layout analysis result 6 is output. For this reason, the layout analysis unit 5 has basic line extraction means 8 and row / stage mutual extraction means 9.

  Further, the basic row extraction means 8 includes a row element generation means 81 for generating row elements having horizontal and vertical row directions based on the association of the basic element sets, and reliability for each row element. And a row element matching means 83 for obtaining a line element based on the reliability and a row element matching means 83 for obtaining a consistent line and character set. The row / stage mutual extraction means 9 includes a stage element integration means 91 for integrating the stages associated with each other, a stage element matching means 92 for obtaining a consistent stage and row set, and consistency. And line element matching means 93 for obtaining a good line and character set.

(1) Basic Line Extraction Means FIG. 10 shows a processing outline of the basic line extraction means.

  The input of the basic line extraction means 8 is a set 4 of basic elements constituting the document, such as a black pixel connected component in the document image or an overlapping rectangle of circumscribed rectangles of the black pixel connected component in the document image. The output is a set of tree structures in which line elements, character elements, and basic elements are related to each other by inclusion relations.

  First, in step S10, when a basic element set 4 is input (step S10), the line element generating means 81 generates a character element that uniquely includes each basic element (step S11), and further, a line element by associating character elements. Is generated (step S12). Specifically, the row element generation unit 81 generates row elements having horizontal and vertical row directions based on the association between sets of basic elements. The basic element sets are associated with each other based on proximity or homogeneity.

  Subsequently, the row element selection means 82 estimates the reliability for each row element, and selects the row element based on the reliability (step S13).

  Further, the line element matching means 83 performs matching processing of the line element set to obtain a consistent line set and character set (step S14), and the line element, character element, and basic element (overlapping rectangular element) are determined. A set of tree structures associated with each other by an inclusion relation is output. Specifically, a row element set that satisfies the following conditions and an internal structure of the row element are obtained.

  1) All character components belong to a single line element.

  2) The row area of each row element does not overlap each other. Here, the line area is an area defined by a circumscribed rectangle of the character component belonging to each line element.

  3) The character elements belonging to the line elements determined in the line direction are ordered in the line direction, and are integrated with each other so that there is no character element overlapping the line and the vertical direction.

  Hereinafter, line extraction by the basic line extraction means 8 will be described in more detail according to a specific example.

  Assume that line / stage extraction of a document image as shown in FIG. 11 is performed. The document image shown in FIG. 11 is input by extracting the overlapping rectangle set as a set of basic elements by extracting the black pixel connected component by the labeling and overlapping rectangle integration processing (step S10 in FIG. 10). A set of character elements each having one of the input overlapping rectangle sets as a child is generated and set as a set of basic character elements (step S11).

  FIG. 12 shows an example of a set of basic character elements of the document image generated in this way. Each rectangle shown in FIG. 12 represents a character element.

  Next, a line element is generated by associating character elements (for example, proximity) (step S12). Here, each character element in the set of input basic character elements is at a distance not more than a threshold value times the size of the character element itself in the horizontal and vertical directions, and is closest to each other. A character element is searched for and linked to generate a row of character elements adjacent in the horizontal and vertical directions as row elements. There are both row elements having a horizontal row direction (horizontal element) and those having a vertical row direction (vertical element). Many character elements are children of both horizontal and vertical elements, and there is a lot of interference between elements.

  FIG. 13 shows an example of a document image displaying circumscribing rectangles of traversing elements generated by the proximity of character elements. FIG. 14 shows an example of a document image in which a circumscribed rectangle of a vertical element generated by the proximity of character elements is displayed.

  Since the document image with the inclination of almost zero can be assumed for the inclination correction by the document image inclination detection unit 1 performed as the pre-processing, the proximity condition does not have a spread due to the inclination. Also, the threshold value of the distance in the proximity condition is a value that is strictly smaller than that in the case of inclination correction by inclination detection, because rows that will belong to different stages are not integrated.

  Next, wrinkling of row elements using row reliability is performed (step S13). A method similar to the processing by the document image inclination detection unit 1 is used. First, the line reliability of the generated line element is calculated based on the average interval of the character elements to which it belongs, the number of character elements to which it belongs, and the aspect ratio of the line area by a circumscribed rectangle including all of the character elements to which it belongs. Next, for each character element, the direction of the associated line element is the character element direction. When there are two types of related row elements, a horizontal row element and a vertical row element, the direction of the row element having a high row reliability is set as the direction of the character element. Furthermore, the strength of the generated line element is defined as the ratio of character elements included in the line element that have the same direction as the line direction of the line element. , Leave those above the threshold as basic row elements with high reliability.

  FIG. 15 shows an example of a document image in which basic line elements with high reliability that have survived due to the bag are displayed in a circumscribed rectangle. In FIG. 15, it can be seen that part of the original horizontal row and vertical row survive as row elements having the correct row direction, and there is no erroneous integration of rows belonging to different stages.

  Next, row element set matching processing (step S14) by the row element matching unit 83 is performed. A processing flowchart of the row element matching unit 83 is shown in FIG.

  First, the area interference between row elements is eliminated (step S31). The circumscribed rectangles between the row elements may be in contact with each other. Here, the circumscribed rectangle of the line element means the smallest rectangle that includes the circumscribed rectangle of the character element included in the line element. If the circumscribing rectangles between the row elements are in contact with each other, it is regarded as interference between the row elements, so that it is eliminated.

  Next, an absorption hypothesis for the character element that has a region interference with the line element is generated (step S32). There may be character elements that touch the bounding rectangle of the line element but do not belong to the child of the line element. Therefore, such a character element is searched for each line element and absorbed as a child. Here, the line element that absorbs a character element as a child is called an absorption hypothesis.

  Subsequently, wrinkles are performed by verifying element interference in the absorption hypothesis of the character element (step S33). In the case of the processing in step S32 described above, since there is a high possibility that a character element that is in contact with a plurality of line elements is noise, it may be desirable to delete the character element without making it a child of the line element. I do.

  Next, the area interference between the row elements is resolved (step S34). As a result of the processing in steps S32 and S33, if there is a character element that is a new child of a certain row element, the circumscribed rectangle of the row element may expand. Therefore, in this case, the processes of steps S31 to S33 are repeated again for the row elements.

  Further, the character elements belonging to the line elements are arranged (step S35). If all the character elements contained in the circumscribed rectangle of each line element can be configured as children, then the character elements that overlap the line direction and the vertical direction are grouped together, It is desirable to limit the possibilities of character extraction candidates. At this time, a basic element set of overlapping rectangular elements that are children of a plurality of character elements that have been collected is integrated into one, and a new integrated character element child is obtained.

  The above row element matching processing will be described by paying attention to the image of one row in the image example shown in FIG.

  A rectangle in the image A in FIG. 17 is a basic column element with high reliability obtained after the row element. A rectangular column in the image B of FIG. 17 is a character element that is a child of the vertical element of the image A. It can be seen that there are two character elements that are not included in the outline rectangle of the line element. Character elements “ya” and “shi” in the image.

  Thereafter, the region interference between the row elements of the image is eliminated. The absorption hypothesis of the character element interfering with the row element is generated / deceived, and as shown in the image C in FIG. All character elements that are present are taken as children. Further, the result of resolving the area interference between the row elements and arranging the character elements belonging to the row elements is an image D in FIG. It can be seen from the image D that the character elements that overlap when projected in a direction perpendicular to the row direction (in this case, the horizontal direction) are integrated.

  Next, an image E in FIG. 17 is a result of the integration processing of the character elements adjacent to the line elements into the line elements. In this case, the whole is integrated as one line. Here, an image F in FIG. 17 shows a set of character elements that are children of the row elements after integration.

  The result of the above processing is shown in FIG. At this stage, all character elements become children of a line element, and a set of tree structures in which line elements, character elements, and basic elements are related to each other by inclusion relations can be constructed as an intermediate document structure.

  The set of row elements obtained as a result of using the above-described row extraction technique includes a plurality of row elements that are close to each other in the row direction when a stage is formed, as is apparent from the example of FIG. ing. In this sense, the row element obtained at this stage does not correspond to a complete row as a component of the stage. In order to construct a complete line, it is necessary to perform a process in conjunction with line extraction and stage extraction, in which line elements belonging to the same stage are further integrated through cooperation with stage extraction.

(2) Row / stage mutual extraction means The row / stage mutual extraction means 9 integrates the stages associated with each other, matches the stage element set to obtain a consistent stage and row set, and consistency. A line element set matching process is performed to obtain a line and a character set. Further, the row / stage mutual extraction means 9 repeatedly executes the integration of the stages, so that the row extraction results and the stage extraction results affect each other, so that the highly accurate row / stage extraction processing is performed gradually. To.

  FIG. 19 shows an outline of the processing flow of the row / stage mutual extraction means 9.

  The input of the row / stage mutual extraction means 9 is a set of tree structures whose vertices are the row elements obtained by the basic row extraction means 8 (a tree structure in which row elements, character elements, and basic elements are related to each other by inclusion relations). The output is a set of tree structures having a stage element as a vertex (a set of tree structures in which stage elements, line elements, character elements, and basic elements are associated with each other by inclusion relations).

  When a set of tree structures having row elements as vertices is input (step S40), a stage element that uniquely includes each row element is generated (step S41). The step elements are integrated by repeatedly executing the following steps S42 to S46 for the step elements. The association between step elements (step S42) and the integration of the associated step elements (step S43) are basically targeted at a set of character elements and row elements in the basic row extraction means 8 to generate row elements. This process is the same as that performed in step 1. This process is performed on a set of row elements and column elements.

  Next, a matching process for the stage element set is performed (step S44). This step element set matching process is a process aimed at obtaining a step element set and an internal structure of the step element that satisfy the following conditions.

  1) All row components belong to a single column element.

  2) The step regions of each step element do not overlap each other. Here, the step region is a region defined by a circumscribed rectangle of the row component belonging to each step element.

  3) The row elements belonging to the step elements determined in the row direction are ordered in the row direction, and are integrated with each other so that there are no row elements overlapping in the row direction and the vertical direction.

  After performing this stage element set matching processing, the row element set matching processing described in the basic row extraction means 8 is performed again (S45). Thus, the stage extraction result is fed back to the line extraction result, and the accuracy of row and stage extraction is improved. The processes in steps S42 to S45 are repeated until no new association occurs (step S46).

  In the above processing, the association between the stage elements is repeatedly executed while changing in the row direction and the row vertical direction, so that the association in the two-dimensional direction can be avoided and the amount of calculation such as matching can be reduced. In addition, the association between the column elements can be dynamically determined by the internal structure of the column element to be associated, thereby promoting high-precision line / column extraction according to the document structure. For example, in the case of a document structure having an organized level with a small number of large levels, it is sufficient to fix the parameters. However, in the case of a document structure with an irregular column with many small columns consisting of a small number of rows, or a tabular document structure in which white space acts as a separator, there is a method for generating hypotheses according to the situation. It is valid. The following three types can be considered as the association between the step elements according to the situation.

  1) The comparison with the standard size of the proximity determination distance is changed depending on the size of the steps to be associated (the number of rows included). As a result, it is possible to promote appropriate integration between the subdivided small stages or between the large and small stages while preventing over-integration between the large and stable stages.

  2) The proximity determination distance is set in proportion to the size of the row included in the stage to be associated. As a result, appropriate integration can be performed according to the size of the character elements to be included. Therefore, unintegrated stages with large line sizes, over-integrated stages with small line sizes, and over-integrated stages with greatly different line sizes. It can be greatly reduced.

  3) The ratio of the proximity determination distance to the standard size is changed according to the level of the expansion process. The row expansion process is repeated twice, but the first time the stage is still subdivided and it is necessary to promote integration, while the second time a rather stable stage can be generated. It is necessary to suppress integration. Therefore, the ratio of the standard size is set high at the first time to facilitate integration, and the ratio of the standard size is set small at the second time to make integration difficult.

  Specifically, the proximity determination distance D is given by kS. Here, S is a standard size that is adaptively determined in the above process 2), and k is a coefficient determined in the above processes 1) and 3).

  In order to realize the processes 1) and 3), for example, the integration of stages is repeated a total of four times in the row vertical direction, the row direction, the row vertical direction, and the row direction. If the coefficient k as shown is used, high-precision stage extraction is possible, and high-precision line extraction can be performed as a result of the stage set matching process. In FIG. 20, “small stage” is a stage that includes only one line, “middle stage” is a line that includes two lines, and the total number of characters included is within 100 characters, “large” A “column” is defined as a column that includes two rows and the total number of characters included is 101 characters or more, or three or more rows.

  Further, the step element sets can be associated with each other based on proximity or homogeneity. By using the homogeneity, it becomes possible to separate the stages with high precision even when the distance between the stages is narrower than the line spacing.

  In addition, by changing the element association parameters used for association between stage element sets according to the number of processes, the integration of rows and stages is advanced, and in the latter half, flexible processes such as suppressing over-integration of stages are performed. It is possible to further improve the accuracy of row / stage extraction.

  FIG. 21 shows the result of stage extraction of the document image example. FIG. 22 shows the row elements in the stage obtained by the row / stage mutual extraction means 9. As a result, it can be seen that for the rows belonging to the same stage, complete row elements that do not overlap each other can be extracted in the projection in the row direction.

  Hereinafter, an embodiment of the present invention will be described with reference to FIGS.

  As an example of a document image of a newspaper document that is a document in which horizontal and vertical rows are mixed, a binary document image obtained by capturing an image at 400 dpi from the newspaper as shown in FIG. 23 is targeted. Note that the document image of FIG. 23 is displayed in a reduced size. In the upper left part, a photo is printed, but here it is not recognized, so it is replaced with a blank.

  FIG. 24 shows an example of the result of the line extraction process for the document image of FIG. The extraction accuracy of the line extraction process for this document image was 100%. FIG. 25 shows an example of the result of the final stage extraction process. The extraction accuracy of the stage extraction process for this document image was also 100%.

It is a figure which shows the basic structural example of this invention. It is a figure which shows the hardware structural example to which this invention is applied. It is a figure which shows the basic structural example of a document image inclination detection part. It is a process flowchart of a document image inclination detection part. It is a figure explaining extraction of a character component. It is a figure explaining extraction of a line candidate. It is a figure explaining extraction of a line candidate. It is a figure explaining estimation of row reliability. It is a figure which shows the basic structural example of the layout analysis part which concerns on this invention. It is a figure which shows the process outline | summary of a basic line extraction means. It is a figure which shows the example of the document image of a process target. It is a figure which shows the example of the set of a basic character element. It is a figure which shows the example of the document image which displayed the circumscribed rectangle of a transverse element. It is a figure which shows the example of the document image which displayed the circumscribed rectangle of a longitudinal element. It is a figure which shows the example of the document image which displayed circumscribed rectangle the line element with the high reliability which survived by the bag. It is a processing flowchart of a row element matching means. It is a figure explaining the example of the image processed by the alignment process of a row element. It is a figure which shows the example of the result extracted by the basic line extraction means. It is a figure which shows the outline | summary of the flow of a process of a row | line | column mutual extraction means. It is a figure which shows the example of the coefficient k which changes the ratio with the standard size of proximity determination distance. It is a figure which shows the example of the stage element of the result extracted by the row / stage mutual extraction means. It is a figure which shows the example of the row element of the result extracted by the row and stage mutual extraction means. It is a figure for demonstrating evaluation of the Example of this invention. It is a figure for demonstrating evaluation of the Example of this invention. It is a figure for demonstrating evaluation of the Example of this invention. It is a figure for demonstrating the problem of a prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Document image recognition apparatus 1 Document image inclination detection part 11 Character component extraction means 12 Line candidate extraction means 13 Line reliability estimation means 14 Line extraction means 15 Inclination estimation means 2 Document image 3 Inclination angle 4 Set of basic elements 5 Layout analysis part 6 Layout analysis result 8 Basic line extraction means 81 Line element generation means 82 Line element selection means 83 Line element matching means 9 Row / stage mutual extraction means 91 Stage element integration means 92 Stage element matching means 93 Line element matching means

Claims (16)

In recognition of an electronic document image, in a document image recognition device that analyzes the layout of a document image,
A basic line extraction means for extracting a set of lines determined in a line direction from a basic element set constituting a document;
A document image recognition apparatus comprising: a line / stage mutual extraction unit that extracts a line and a stage by mutually executing a stage extraction based on association between lines and a line extraction with a stage as a constraint. The document image recognition apparatus according to claim 1, wherein an overlapping rectangle of a circumscribed rectangle of a black pixel connection component in a document image or a black pixel connection component in a document image is used as the basic element set. The basic row extraction means includes:
Row element generation means for generating row elements having horizontal and vertical row directions based on the association of the basic element sets;
A row element input means for estimating the reliability for each row element and inputting the row element based on the reliability;
The document image recognition apparatus according to claim 1, further comprising: line element matching means for obtaining a matched line and character set. The document image according to claim 3, wherein the association of the basic element sets is performed based on proximity such as a distance between elements, or homogeneity such that element sizes or element intervals are substantially equal. Recognition device. The row element selection means is:
Each row element is calculated quantitatively for each row element, and the direction of the row reliability of the horizontal and vertical row elements to which each character element belongs is set as the direction of the character element. The ratio of the character elements belonging to the line candidate to those having the character direction that matches the line direction of the line element is defined as the reliability of the line element. Document image recognition device. The reliability of the line element is an amount based on the length of the line that the number of character elements included in the line element is a certain number or more, an amount based on the proximity that the character elements are closely arranged, or The document image recognition according to any one of claims 3 to 5, wherein the character image is quantified by using a quantity based on homogeneity such as the size or spacing of character elements being approximately the same. apparatus. The row / stage mutual extraction means is:
Stage element integration means for integrating the stages associated with each other;
A stage element matching means to obtain a consistent stage and row set;
The document image recognition apparatus according to any one of claims 1 to 6, further comprising a line element matching unit for obtaining a line and a character set having consistency. The document image recognition apparatus according to any one of claims 1 to 7, wherein the row / stage mutual extraction unit repeatedly performs integration of stages. The document according to any one of claims 1 to 8, wherein the association between the stages in the row / stage mutual extracting means is repeatedly executed while changing the row direction and the row vertical direction. Image recognition device. The condition for associating stages with each other in the row / stage mutual extracting means is dynamically changed according to the internal structure of the stage element to be associated. The document image recognition apparatus described in 1. The row-stage mutual extraction means performs the association between stages based on proximity such as a distance between elements, or homogeneity such as element sizes or element intervals being substantially equal. The document image recognition apparatus according to any one of claims 1 to 10. The document image recognition apparatus according to any one of claims 1 to 11, wherein a parameter used for associating stages with each other in the row / stage mutual extraction unit is changed according to the number of processing times. As the internal structure of the stage element to be associated in the row / stage mutual extraction means, the size of the circumscribed rectangle of the stage, the number of lines or characters included in the stage, or the position of the circumscribed rectangle of the stage is used. The document image recognition apparatus according to any one of claims 1 to 12. The line element matching means in the basic line extracting means or the line element matching means in the row / stage mutual extracting means is such that the standard character width, height, spacing, etc. of the line are determined from the set of character elements belonging to the line element. 8. The document image recognition apparatus according to claim 3 or 7, wherein information serving as row attributes is calculated and stored, and used for association between elements. The stage element matching means in the row / stage mutual extraction means is:
Calculate and maintain information that is the attribute of the column, such as the standard row width, height, and spacing, from the set of row elements belonging to the column element, and use it for associating the elements. The document image recognition apparatus according to claim 7. A computer-readable storage medium storing a program used in a document image recognition apparatus for analyzing a layout of a document image when recognizing an electronic document image,
A basic line extraction process for extracting a fixed line set from the basic element set constituting the document;
A program is stored that allows the computer to execute row extraction and row mutual extraction processing to extract rows and steps by mutually executing row extraction based on associations between rows and row extraction with row constraints. A storage medium for a document image recognition program.

Images