JP4847592B2 - Method and system for correcting distorted document images - Google Patents

Description

  The present invention relates to a method and system for correcting a distorted document image, and more particularly to a method and system for correcting a distorted document image obtained from a book or a book bound by a digital camera.

  Digital cameras are becoming increasingly popular not only as consumer consumables but also as useful tools in the business and technical fields. For the OCR (optical character recognition) field, digital cameras offer the potential to replace scanners, which are document imaging devices. However, current OCR technology is designed primarily for digitally scanned images of flat documents and cannot process general document images taken by a camera.

  The image captured by the scanner has a viewing angle of approximately 0 degrees when viewed directly from the front, and is generally flat, so there is little geometric distortion caused by fluoroscopy or warping problems. However, when capturing an image of a book or other type of bound document with a digital camera, it has some non-zero viewing angle, and the book or bound document has a specific curvature that depends on their thickness. Have. As a result, document images taken with a digital camera are adversely affected by geometric distortions caused by perspective and warping problems. FIG. 8 shows an example of a document image taken by a digital camera including obvious distortion caused by the problems of perspective and warpage. If such a distorted document image is used directly for OCR operation, the recognition accuracy will be low.

  Many methods for correcting the distortion of the distorted document image have been proposed. One category of methods uses special 3D scanning equipment to correct the distortion. According to the method in NPL 1, a laser projector is used to project a 2D optical network onto a 3D surface of a document and a mesh is created to represent the 3D surface of the document and is directly flattened into a deployable mesh. Or converted.

  Alternatively, the 3D shape of the surface is estimated from the document image. There are parametric methods for estimating 3D shapes and non-parametric methods for avoiding shape estimation processing.

  Non-Patent Document 2 introduced a method of using a combination of a cylinder and a plane for simulating a 3D surface model of a book, a method for estimating parameters related to this model, and a model for correcting distortion Problems with how to do remain unresolved. Furthermore, this method is expensive because special equipment needs to be used. This method can also be applied to an image scanned by a scanner.

  According to the method introduced in Non-Patent Document 3, the depth of the document at each point of the document image is acquired by a specific stereo vision method so as to create a depth image, and the document image is planarized according to the depth image. Correct it. Although any type of image distortion is believed to be corrected, the method of mapping a point on the noisy rough surface of the document defined by the depth image to a point on the plane remains a problem.

  In Non-Patent Document 4, a scanned image of a book that has been bound is corrected by a character segmentation process. The shadowed characters (with rounded surfaces) are segmented, their orientation and original location are estimated, and the characters are adjusted accordingly.

  A method of using a model for correcting an image of a bound document captured by a camera was proposed in Non-Patent Document 5. The surface of the document is represented by a general column surface. Obviously, other warp types such as folds cannot be handled.

  In general, if the document image has only distortions due to perspective problems, the information about the direction of the edge of the page is used to approximate the direction of the characters. However, when distortion occurs due not only to the perspective problem but also to the warp problem, the character of one page has different distortions in different directions, so the above approximation method is not effective.

  Therefore, there is a need for a technique that is easy and effective to cope with and realize the distortion in the image due to the problem of warping and fluoroscopy necessarily associated with the document image taken by the camera.

"Former Books Digital Processing: Image Warping" by A. Doncescu, A. Bouju, V. Quillet, Proc. Workshop of Document Image Analysis, pages 5-9, 1997 "Global and Local Document Degradation Models" by T. Kanungo, R. Haralick, and I. Philips in Proc. 2nd International Conference on Document Analysis and Recognition, 1993 "Document Restoration Using 3D Shape: a General Deskewing Algorithm for Arbitrarily Warped Documents" by M. S. Brown, W. B. Seales, Proc. International Conference on Computer vision, July 2001 "Restoration of Images Scanned from Thick Bound Documents" by Z. Zhang, C. L. Tan in Proc. 6th International Conference on Document Analysis and Recognition, 2001 `` Rectifying the Bound Document Image Captured by the Camera: A Model Based Approach '' Proceedings of the Seventh International Conference on Document Analysis and Recognition (ICDAR 2003) by Huaigu Cao, Xiaoqing Ding, and Changsong Liu

  In view of the technical problems present in the prior art described above, a new method for correcting distorted document images is provided. The present invention extends a curved document page that is always present in a naturally opened book or a naturally opened stack of paper, extending perpendicularly to the direction of the text line or parallel to the direction of binding, and arranged in the direction of the text line. Based on the basic concept of being approximated by a group of planar stripes. That is, these image stripes are obtained by dividing a distorted document image including at least one vertical row. In each stripe, warping is ignored and distortion due to fluoroscopy becomes the main distortion. Thus, the complex nonlinear problem is converted into a plurality of simpler local linear problems. The present invention is further based on two important text features which are the local orientation of the text line and the vertical character stroke. Those features are used to identify local linear distortions.

  The present invention does not require any auxiliary devices and can address a wide variety of distortion types, such as those caused by bookbinding, folding and fluoroscopy. The present invention is mainly intended to correct distortion of a document image taken by a digital camera, but the present invention is input from another device such as a document image of a thick book scanned by a flatbed scanner. This is applicable when correcting a document image to be processed.

According to one aspect of the present invention, a method for correcting geometric deformation in a distorted document image of a document, comprising:
A vertical vanishing point detecting step of detecting a vertical vanishing point of the distorted document image, which is a vertical vanishing point perpendicular to the text line of the document;
An image dividing step of dividing the entire region of the distorted document image into a plurality of image stripes using a plurality of vertical lines derived from the detected vertical vanishing points;
By obtaining the vanishing point of the horizontal direction parallel to the text lines of the document in each of said plurality of image stripes, and the horizontal vanishing point detecting step of detecting a horizontal vanishing point for each of the plurality of image stripes,
Wherein a plurality of vertical lines derived from the vertical vanishing point, a plurality of mesh model formed based on a horizontal line derived from each of the horizontal vanishing point detected for each of the plurality of image stripes, A distortion model generation step for establishing a distortion model describing a mapping relationship between the distortion document image and the corrected document image;
A correction step of generating a corrected document image based on the distortion model;
There is provided a method characterized by comprising:

According to another aspect of the present invention, a system for correcting geometric deformation in a distorted document image of a document, comprising:
Vertical vanishing point detecting means for detecting a vertical vanishing point of the distorted document image, which is a vertical vanishing point perpendicular to the text line of the original;
Image dividing means for dividing the entire region of the distorted document image into a plurality of image stripes using a plurality of vertical lines derived from the detected vertical vanishing point;
By obtaining the vanishing point of the horizontal direction parallel to the text lines of the document in each of said plurality of image stripes, and the horizontal vanishing point detecting means for detecting a horizontal vanishing point for each of the plurality of image stripes,
Wherein a plurality of vertical lines derived from the vertical vanishing point, a plurality of mesh model formed based on a horizontal line derived from each of the horizontal vanishing point detected for each of the plurality of image stripes, A distortion model generating means for establishing a distortion model describing a mapping relationship between the distortion document image and the corrected document image;
Correction means for generating a corrected document image based on the distortion model;
There is provided a system characterized by comprising:

  Further characteristic features and advantages of the present invention will become apparent upon reading the following description with reference to the drawings.

  The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, explain the principles of the invention.

It is a block diagram which shows the structure of the arithmetic unit for implement | achieving the system which correct | amends a distorted document image according to this invention. It is a block diagram which shows the general structure of the system which correct | amends the distortion document image comprised from each module means according to one Embodiment of this invention. 6 is a flowchart illustrating a general process for realizing a method for correcting a distorted document image according to the present invention. 6 is a flowchart illustrating an exemplary process for detecting a vertical vanishing point according to an embodiment of the present invention. It is a figure which shows the intersection angle prescribed | regulated by the line which connects the line segment and the midpoint of the line segment, and an intersection, in order to show the method of calculating a vanishing point. 6 is a flowchart illustrating an exemplary process for identifying the position of a curve in a text line in accordance with the present invention. FIG. 6 is a schematic diagram illustrating how a distorted document image is mapped to a corrected document image of a mesh model grid. It is a figure which shows an example of the general document image image | photographed from one page of the book with the camera. It is a figure which shows the example edge image derived | led-out from the document image shown in FIG. (A) is a figure which shows the example edge image acquired by performing a rotation, compression, and a run length smoothing algorithm with respect to the edge image shown in FIG. 9, (b) is (a). It is a figure which shows the example image comprised by the point of the intermediate | middle height extracted from the image shown to. FIG. 10 is a diagram illustrating an exemplary edge image composed of vertical strokes acquired by removing horizontal stroke edges from the edge image shown in FIG. 9. It is a figure which shows the example image acquired by finding the connection component of the vertical stroke extracted from the edge image shown in FIG. FIG. 10 is a diagram illustrating the document image of FIG. 9 together with a mesh configured by a method according to an embodiment of the present invention. It is a figure which shows the example correction | amendment document image by the distortion correction method which concerns on this invention. It is a figure used in order to explain the method by which a mesh is constituted.

  Embodiments of the present invention will be described in detail below with reference to the drawings.

  In the description and claims of the present invention, particularly when used in distorted document images, the term “horizontal” or “x direction” means substantially horizontal and the term “vertical” or “y direction” is approximately Means vertical. In particular, the term “horizontal” used in “distorted document image” means in the direction parallel to the corresponding corrected document image or text line of the document. For example, the term “horizontal line” means that the line of the correction plane image (or document) corresponding to the line is horizontal with the text line of the correction plane image (or document). Similarly, the term “vertical” in a distorted document image means perpendicular to the corresponding corrected document image or text line of the document. For example, the term “vertical stroke” means that the stroke is perpendicular to the text line of the corrected planar image (or document).

  In the description of the present invention, the terms “left” and “right” refer to the left side and the right side when browsing a page in the usual manner similar to when reading a book or document.

  In the description of the present invention, unless otherwise indicated, all sizes (length or width, etc.) are in “pixel” units. For example, L <5 means that L is less than 5 pixels.

  FIG. 1 is a block diagram showing the configuration of an arithmetic unit for realizing a system for correcting a distorted document image according to the present invention. For simplicity, the system is shown as being built into a single computing device. However, the system is effective regardless of whether the system is incorporated in a single computing device or a plurality of computing devices as a network system.

  As shown in FIG. 1, the arithmetic device 100 is used to realize processing for correcting a distorted document image. The arithmetic device 100 may include a CPU 101, a chip set 102, a RAM 103, a storage device controller 104, a display controller 105, a hard disk drive 106, a CD-ROM drive 107, and a display 108. The arithmetic device 100 includes a signal line 111 connected between the CPU 101 and the chip set 102, a signal line 112 connected between the chip set 102 and the RAM 103, and a connection between the chip set 102 and various peripheral devices. Peripheral device bus 113, signal line 114 connected between storage device controller 104 and hard disk drive 106, signal line 115 connected between storage device controller 104 and CD-ROM drive 107, and display controller 105 And a signal line 116 connected between the display 108 and the display 108.

  The client 120 may be directly connected to the arithmetic device 100 or may be connected via the network 130. The client 120 may send a correction task to the arithmetic device 100, and the arithmetic device 100 may return the correction result to the client 120.

  FIG. 2 is a block diagram showing a general configuration of a system for correcting a distorted document image constituted by each module means.

  As shown in FIG. 2, the distortion correction system 200 uses the vertical vanishing point detecting means 201 for detecting the vertical vanishing point of the distorted document image and the distorted document image using a vertical line starting from the detected vertical vanishing point. An image dividing unit 203 for dividing the entire area into a plurality of image stripes, a horizontal vanishing point detecting unit 205 for detecting a horizontal vanishing point for each image stripe, a vertical vanishing point, and a horizontal vanishing point of the image stripe. Further, a distortion model generation unit 207 that establishes a distortion model that describes a mapping relationship between the distortion document image and the corrected document image, and a correction unit 209 that generates a correction document image based on the model may be included. The vertical vanishing point detection unit 201 includes a vertical stroke extraction unit 2011 that extracts a plurality of vertical strokes of a character from a distorted document image, a vertical straight line segment fitting unit 2013 that fits the vertical stroke by a plurality of vertical straight line segments, It is preferable to include vertical optimum convergence point calculation means 2015 for calculating a vertical vanishing point from the vertical straight line segment by searching for an optimal convergence point of the vertical straight line segment. The vertical optimum convergence point calculation means 2015 includes an intersection calculation means 2015-1 for calculating an intersection between any two of the vertical straight line segments, and an optimum convergence point that minimizes the sum of the sine squares of the intersection angles. It is preferable to include optimum point selection means 2015-2 for selecting one of the intersections. The horizontal vanishing point detecting means 205 extracts a text line curve position specifying means 2051 for specifying the position of the curve of the text line along the direction of the text line from the distorted document image, and a text line curve fragment existing in the image stripe. A segment extracting unit 2052 for performing horizontal segment line fitting unit 2053 for fitting the segment of the text line curve by a horizontal line segment, and searching for an optimal convergence point of the horizontal line segment by horizontally searching the horizontal line segment. It is preferable to include a horizontal optimum convergence point calculating means 2054 for calculating the vanishing point. The horizontal optimal convergence point calculation means 2054 is an intersection calculation means 2054-1 for calculating an intersection between any two of the horizontal straight line segments, and an optimal convergence point that minimizes the sum of the squares of the sine of the intersection angles. And an optimum point selecting means 2054-2 for selecting one of the intersections. The text line curve position specifying means 2051 uses the intermediate height point extracting means 2051-1 for extracting the intermediate height point with respect to the pixel of the character of the distorted document image, and the intermediate height point to use the text. It is preferable to include text line curve position specifying means 2051-2 for specifying the position of the text line curve over the middle height of the characters of the line.

  The above means is an exemplary suitable module for realizing the processing described below. The modules that implement the various steps are not exhaustively described above. However, when there is a step for executing a specific process, there is a corresponding functional module or means for realizing the same process.

  FIG. 3 is a flowchart showing a process for realizing a method for correcting a distorted document image according to the present invention. FIG. 8 shows an example of a general document image taken from a book page by a camera. As shown in FIG. 8, there is a clear distortion due to warpage and perspective problems.

  In step S310, the direction of the vertical character stroke is used to detect the vertical vanishing point. The meaning of the vanishing point will be described below. All straight lines in planes that are parallel to each other do not, in theory, intersect each other regardless of how they extend. On the other hand, if the plane is placed in a three-dimensional space with a non-zero viewing angle, the lines that were initially parallel to each other will no longer appear to be parallel, and theoretically the expanded line of all those lines is Intersect at one point. The point where lines parallel to each other in the plane intersect each other in a three-dimensional space with a non-zero viewing angle is called a vanishing point. Therefore, the point where planar vertical lines intersect each other in a three-dimensional space with a non-zero viewing angle is called a vertical vanishing point. Similarly, the point where planar horizontal lines intersect each other in a three-dimensional space with a non-zero viewing angle is called a horizontal vanishing point. The terms “horizontal” and “vertical” are defined above. In most cases, the natural warping of a bound book extends in a direction parallel to the binding line or perpendicular to the text line. Therefore, there is only one vertical vanishing point for one captured image. “Interpreting Perspective Images” by Barnard, ST, Artificial Intelligence, vol. 21, pp. 435-462, the method of using Gaussian spheres disclosed in 1983 and “Vanishing Point Detection by Virginio Cantoni, Luca Lombardi, Marco Porta, Nicolas Sicard. There are many well-known methods for detecting vanishing points such as the method based on Hough transform accumulation in the polar space disclosed in “Representation Analysis and New Approaches”, Proceedings of the 11th International Conference on Image Analysis & Processing.

  After the vertical vanishing point is determined in step S310, in step S320, a group of vertical lines generated from the vertical vanishing point and extending over the image area is derived, and the entire image area is arranged along the horizontal direction. Divide into

  In step S330, a horizontal vanishing point for the image stripe is detected.

  In step S340, a model describing the overall distortion of the distorted document image is constructed from the image stripes and the corresponding horizontal and vertical vanishing points. After the vertical vanishing point and the horizontal vanishing point are determined, both the perspective characteristic and the warp characteristic are determined. Thus, various document features such as text lines, character strokes and page edges are used to construct such models.

  In the final step S350, a corrected document image is generated using the constructed model.

  The following is an exemplary embodiment according to the present invention for correcting distortion of a distorted document image.

  In the first step S310, the orientation of the vertical character stroke is used to detect the vertical vanishing point. Detailed steps for executing step S310 will be described below with reference to FIG.

  FIG. 4 is a flowchart showing the process of detecting the vertical vanishing point in step S310.

  In step S410, the edge of the foreground object in the image is detected. In order to detect edges, various general contour detection techniques such as Sobel operator, Canny operator, etc. are applied herein. The output is an edge image and an edge angle image. The edge angle image has information regarding the tilt angle for each detected edge pixel. The inclination angle of the pixel is an angle indicating the change direction of the gradation value of the pixel. That is, the inclination angle of a pixel indicates a direction in which the gradation is changed from the adjacent pixel to the pixel. FIG. 9 shows an exemplary edge image of the document image shown in FIG. It can be seen that the edge of each character is extracted.

  In step S420, the edge image is processed in the following manner to select a vertical character stroke. Documents photographed by a digital camera may not be properly placed, so it is very likely that a specific skew will be introduced. In order to detect the skew angle, for example, “A Nearest-Neighbour Chain Based Approach to Skew Estimation in Document Images” Pattern Recognition letters 24 (2003), pages 231-2323 by Yue Lu and Chew Lim Tan. Various existing methods for detecting the skew angle in a binary image, such as the nearest neighbor method proposed in US Pat. Since all that is necessary is to obtain the rough direction of the text line from the skew angle, the detected skew angle does not necessarily have to be very accurate.

  Edge pixels that do not belong to a vertical stroke are removed by comparing their tilt direction with the skew angle. θi is the inclination angle of the pixel located at the edge (xi, yi), and θ is the skew angle of the document. If | θi−θ | is greater than a predetermined threshold, the pixel should be removed. Note that (xi, yi) are coordinates in the Cartesian coordinate system established for the distorted document image. FIG. 11 shows an exemplary edge image obtained by removing pixels at the edge of the horizontal stroke from the edge image shown in FIG.

  In step S430, vertical stroke candidates are found by searching for connected components in the edge image acquired in step S420. A connected component means a set of pixels, and each pixel is combined with at least another pixel in the set. Two pixels are considered “connected” if the pixel is in a particular neighborhood of another pixel (eg, a neighborhood within 3 pixels horizontally and 4 pixels vertically from the other pixel). There are many algorithms known in the prior art for searching for connected components from pixels. For example, in the search strategy, a start point is first selected from the lower side of the image, and black pixels are searched for upward in the vertical direction. The next foreground pixel is always searched from the black pixel within the neighborhood described above. The neighborhood size can be arbitrarily selected by those skilled in the art according to actual requirements. For the purpose of vanishing point calculation, it is preferable to consider only the main connected components whose length L is in a certain desired range (eg 12 <L150). That is, connected components having a length not within this range are preferably removed or ignored. The numbers in the above range are merely examples, and those skilled in the art may arbitrarily select this range according to the design requirements or the actual document state. FIG. 12 shows an exemplary image obtained by finding the connected components of the extracted vertical strokes from the edge image shown in FIG.

  For each connected component found, the line is fitted and parameterized by the distance ρ from the origin and the angle θ.

  Where x and y are the x and y coordinates of a point on a line in a Cartesian coordinate system, and θ and ρ are two parameters determined during fitting.

Each acquired connected component is a group of edge pixels (x _i , y _i ) having the same inclination direction, and (x _i , y _i ) is the coordinate of the i-th pixel of the connected component in the Cartesian coordinate system. And i = 1, 2, 3,... The line parameters are determined directly from the eigenvalues λ ₁ and λ _{2 of} the matrix D associated with the edge pixels and the eigenvectors v ₁ and v ₂ . The matrix D is defined as follows.

Since processing for evaluating eigenvalues and eigenvectors of a matrix is well known in the prior art, detailed processing for acquiring eigenvalues λ ₁ and λ ₂ and eigenvectors v ₁ and v ₂ of the matrix D is omitted in this specification. For an ideal line, one of the eigenvalues should be zero. The quality of the line fitting is characterized by the ratio of the two eigenvalues of the matrix D, ie v = λ ₁ / λ ₂ . Line parameters are determined from the eigenvectors v ₁ and v ₂ , where v ₁ is the eigenvector associated with the largest eigenvalue. The line parameters are calculated as follows:

v ₁ (1) is a first dimension of _{v 1, v} 1 (2) is a second dimension of _{v 1.} According to the above equation, the line parameters θ and ρ are obtained, thereby obtaining the vertical straight line segments fitting each connected component to the vertical stroke.

In step S440, the vertical vanishing point is acquired by searching for the optimum convergence point of the vertical line segments. There are various existing techniques that can be used in estimating the optimal convergence point of a plurality of lines. The following is an exemplary process for estimating the optimal convergence point of these vertical straight line segments. First, the intersection between any two of the line segments is calculated as a candidate set of vertical vanishing points (x ₀ ^j , y ₀ ^j ), j = 1, 2, 3. A statistical method is then used to select the optimal convergence point from the group of intersections as the resulting vertical vanishing point. The vertical vanishing point may be, for example, one of the intersection points (x ₀ ^j , y ₀ ^j ) that minimizes the function F (j).

See Figure 5 for intuitive expression. It should be noted that the above description for step S440 is merely exemplary and is not intended to limit the scope of the present invention. As described above, there are many known techniques for acquiring the vanishing points of a plurality of lines, and the method for acquiring the vanishing points is not limited to the above-described method.

After the vertical vanishing point is determined in step S310 (eg, by using the above-described substeps of S410 to S440), in step S320, a group of vertical lines that originate from the vertical vanishing point and span the image area is derived. The entire image area is divided into a plurality of planar stripes arranged along the horizontal direction. For example, this division is basically based on one or more of the following exemplary criteria.
(1) The length of a single text line curve in each sprite is within a specific range [L1, L2]. L1 and L2 are values determined according to the average character size in the document image, for example.
(2) If the image being processed spans two pages, the stapling or binding line is one of those vertical lines.
(3) When an image includes two or more columns, a separation line such as the vertical line exists between adjacent columns.
(4) The stripe is narrower at the center of the page, and wider at the left and right sides of the page.
(5) Each stripe is considered to be substantially flat. That is, one stripe has only one horizontal vanishing point. The criteria for flatness may depend on actual requirements and expected OCR accuracy.

  The above criteria ensure that the split stripe is wide enough to calculate an accurate horizontal vanishing point, while at the same time ensuring that the stripe is sufficiently flat.

  The above criteria are merely examples, and are not intended to limit the protection scope of the present invention. One skilled in the art may employ one or more of the above criteria, or can design other criteria for segmenting the image area. The criteria for the image region segmentation method depends on the actual requirements and the expected OCR accuracy.

  In step S330, a horizontal vanishing point for each image stripe is detected. In general, since the paper of a book fluctuates along the horizontal direction, the horizontal vanishing point of each image stripe is different. That is, each image stripe has its own horizontal vanishing point.

  The following is an exemplary process for obtaining a horizontal vanishing point for each image stripe.

  Initially, a curve over the middle height of each text line is detected from the edge image as extracted in step S410. Specifically, the mid-height points of the character are extracted, and the position of the curve in the text line is determined from the mid-height points by using connected component analysis. The text line curve is sufficiently accurate to specify the distortion information along the horizontal direction.

  FIG. 6 shows a detailed flowchart for specifying the position of the curve in the text line. Note that the process of FIG. 6 is merely an illustrative and preferable example. Those skilled in the art are familiar with various methods for locating various text line curves from an edge image.

  In step S610, the binary image that is the edge image extracted in step S410 is converted by, for example, affine transformation, and a new binary image I1 is generated. In the affine transformation operation, the binary image is rotated by the above skew angle so that the binary image looks almost vertical, and the image rotated by the “OR” method in which the horizontal compression rate N is larger than the vertical compression rate M is compressed. Is equivalent to The “OR” method is such that for an N × M image block on an uncompressed image corresponding to a pixel on the compressed image, the corresponding pixel on the compressed image is present when at least one black pixel is present in the image block. It means that it is set as black. The purpose of the rotation is to make the text lines sufficiently horizontal, and the main purpose of compression is to blur the character spacing and get a “solid” text block for each text line. It is much easier to search for mid-height points of characters in a “solid” text block.

  In order to achieve an appropriate “solid” effect and at the same time not to mix adjacent text lines, the vertical compression ratio M is such that the height of the original image divided by M is a predetermined value (eg 512). Specified as the smallest positive integer that satisfies not being greater. Furthermore, the vertical compression rate M can be assigned according to the average height of the detected characters. For example, if the average height of the detected characters is H, M may be assigned as H / 8. The horizontal compression rate N can be specified as 3 * M.

  In step S620, to obtain a more appropriate “solid” effect on the text line, the binary image I1 performs a run length smoothing algorithm (RLSA) in the horizontal direction and then performs RLSA in the vertical direction. Is smoothed. A run means a part of consecutive N pixels that do not have a space (white pixel) between them. The run-length smoothing algorithm is characterized by a parameter characterized by length (number of pixels), and when the distance between the two pixels is less than that length, the pixels between the two pixels are all rendered “black” Or in other words, two pixels are considered “contiguous” according to a run-length smoothing algorithm. This parameter can be selected as a value between 2 and 4. The run-length smoothing algorithm has the effect of communicating pixels that have a short distance together to identify “almost” continuous lines or curves. FIG. 10A shows an exemplary edge image obtained by performing the rotation, compression and run-length smoothing algorithms on the edge image of FIG. 9 as described above.

  In the next step S630, a black run is found along the vertical direction on the binary image I1. Black runs that do not belong to the letter are discarded by removing runs that are too short or too long. For example, H1 and H2 are respectively designated as the text height that may be the minimum and the text height that may be the maximum in the document after the affine transformation. If the length of the black run is shorter than H1 or longer than H2, the black run is discarded. After the discard step, most of the retained black runs belong to the characters.

  In step S640, the intermediate height point of the black run to be held is extracted as the intermediate height point of the character. Thereafter, the binary image I2 is generated with the same size as the binary image I1. In the binary image I2, the pixel is set to black at a position corresponding to an intermediate height point. FIG. 10B shows an exemplary image composed of intermediate height points extracted from the image of FIG.

  In step S650, after a binary image I2 containing a medium height point is acquired, a curve is found by using a connected component search method. Similar to the method described in step S430, the search strategy is to first select a starting point from the left side of the image, for example, and search for black pixels in the horizontal direction toward the right side. In an implementation of the method, the next foreground pixel is always searched from black pixels, for example within 4 pixels in the horizontal direction and 3 pixels in the vertical direction.

  In many cases, the curve obtained in this way is not smooth due to variations in character strokes as shown in FIG. Therefore, in step S660, run length information is preferably used to smooth the curve. For example, the average run length H of each curve is calculated, and points whose corresponding run length exceeds the range [a * H, b * H] are removed. Here, a <1 and b> 1. Other smoothing methods can be used in step S660 as long as the curve is smoothed. If the computational complexity is limited or the accuracy requirements allow, step S660 can be further omitted.

  In step S670, the coordinates of the curve in the text line are converted to the original edge image. This conversion operation is an inverse operation of the above-described affine conversion operation.

Then, for each image stripe, the horizontal vanishing point is calculated by the following steps.
a) extracting a curve segment of a text line located in an image stripe;
b) fitting each piece of the curve of the text line by a horizontal straight line;
c) calculating a horizontal vanishing point from the horizontal and vertical line segments by selecting an optimal convergence point of the horizontal straight line segment.

  The process of selecting the optimum convergence point when calculating the horizontal vanishing point may be executed by the same process as that for calculating the vertical vanishing point in step S440, for example. However, the process of selecting the optimum convergence point when calculating the horizontal vanishing point may be executed in a process different from that for calculating the vertical vanishing point in step S440 as long as the vanishing point can be calculated.

  Returning now to FIG.

  In step S340, a model describing the distorted document image from the image stripes and the corresponding horizontal and vertical vanishing points is configured to describe the mapping between the distorted document image and the corrected document image. In this example, the model is a mesh. FIG. 15 illustrates a method for constructing a mesh. As shown in FIG. 15, the document image Pa-Pb-Pc-Pd is drawn with a solid curve, and is divided into three stripes, STRIP1, STRIPE2 and STRIPE3 in order from the left side to the right side. One vertical vanishing point VVP and three horizontal vanishing points HVP1, HVP2 and HVP3 are found according to the method described above. Horizontal vanishing points HVP1, HVP2, and HVP3 are horizontal vanishing points for STRIP1, STRIPE2, and STRIPE3, respectively. Thus, there are two vertical lines Pe-Pf and Pg-Ph starting from the vertical vanishing point VVP that divides the image into the three stripes. Here, the horizontal curve of the mesh is considered. Two horizontal curves are used in the example illustrated in FIG. However, the number of horizontal curves may be determined depending on expected OCR accuracy and requirements, as well as conditions such as processing speed and computing power. For example, two points on the left edge of this image are selected as point P011 and point P012. These points are preferably selected to equally divide the left edge. However, this is not a strict requirement. Thereafter, one line is drawn starting from the horizontal vanishing point HVP1 toward the point P011, extended to have an intersection P121 with the line Pe-Pf, and one line starting from the horizontal vanishing point HVP1. Drawn toward the point P012 and extended to have an intersection P122 with the line Pe-Pf. Next, one line is drawn starting from the horizontal vanishing point HVP2 toward the point P121, and has an intersection P231 with the line Pg-Ph, and one line starts from the horizontal vanishing point HVP2. It is drawn toward P122 and has an intersection P232 with the line Pg-Ph. Finally, one line is drawn starting from the horizontal vanishing point HVP3 toward the point P231 and has an intersection P301 with the right edge of the image, and one line starting from the horizontal vanishing point HVP3. It is drawn towards P232 and has an intersection P302 with the right edge of the image. As a result, eight points P011, P012, P121, P122, P231, P232, P301 and P302 are obtained. The two horizontal curves may be fitted by using two groups of points: a group of points P011, P121, P231 and P301 and a group of points P012, P122, P232 and P302. That is, generally a horizontal curve is determined by calculating the intersection of each horizontal vanishing point and a vertical line. Note that in the example shown above, the method starts from the left horizontal vanishing point, but any one horizontal vanishing point as long as the direction of the horizontal curve within a particular stripe is determined by the horizontal vanishing point for that stripe. A point can be used as the starting horizontal vanishing point.

  FIG. 13 shows the document image of FIG. 9 together with the mesh constructed by the method described above. As shown in FIG. 13, the entire image area is divided into eight planar stripes derived by seven vertical lines derived from the vertical vanishing point. The group of nine horizontal curves is determined by calculating the intersection between the horizontal vanishing point and the vertical line as described above.

  After the mesh is established, a mapping between points on the distorted document image and points on the corrected document image is generated by referring to the mesh, and then the corrected document image is obtained by referring to the mapping. .

  The mapping between points on the distorted document image and points on the corrected document image is determined based on boundary interpolation. One of the boundary interpolation methods is “Identification of Text-Only Areas in Mixed-type Documents” by C. Strouthopoulos, N. Papamarkos and C. Chamzas, Engng Applic. Artif. Intell., Elsevier Science Ltd, Great Britain, Vol. 10, No. 4, pages 387-401, explained in 1997.

In one example, natural cubic splines are used to fit curves that connect their intersections and surround the grid of meshes as boundary curves C _i (i = 1, 2, 3, 4), as shown in FIG. Is done. The lower left part of FIG. 7 shows one grid of a mesh surrounded by four boundary curves C _i (i = 1, 2, 3, 4) as described above. These boundary curves are part of the horizontal curve and the vertical line acquired by the vertical vanishing point and the horizontal vanishing point as described above. The lower right part of FIG. 7 shows the corrected document image defined over the parameter spaces u and v in the corrected document image grid corresponding to the distorted document image grid. Here, uε [0,1] and vε [0,1]. The horizontal boundary curves c ₁ and c ₃ are represented by x coordinates c _ix (u) and y coordinates c _iy (u) (i = 1, 3), and the vertical boundary lines c ₂ and c ₄ are represented by a x-coordinate _c ix (v) and _c iy is y-coordinate (v) (i = 2,4) . That is, for each u in the corrected document image, the points on the horizontal boundary curves c ₁ and c ₃ in the distorted document image are (c _ix (u), c _iy (u)) (i = 1, 3 ), And for each v of the corrected document image, each point on the vertical boundary lines c ₂ and c ₄ in the distorted document image is (c _ix (v), c _iy (v)) (i = 2) , 4).

Each boundary curve ci (i = 1, 2, 3, 4) maps to a corresponding side of an image consisting of straight lines in uv space. For example, the u axis of the corrected document image corresponds to the curve c ₁ in the distorted document image, and the v axis of the corrected document image corresponds to the curve c ₄ in the distorted document image. In this case, an arbitrary point (u, v) in the corrected document image is surrounded by a boundary curve ci (i = 1, 2, 3, 4) in the distorted document image (c _x (u , V), c _y (u, v)), a 2D function that describes how to map is provided using a bilinear mixed Coons patch, for example:

  These equations are formed by linear interpolation of two opposing boundary curves (first and second terms of the equation), and the correction function is based on the corner points of the boundary (third term of the equation). More details on such expressions can be found in “Correcting document image warping based on regression of curved text lines” proceedings of the Seventh International Conference on Document Analysis and Recongnition (ICDAR '03) by Zheng Zhang, Chew Lim Tan. .

To get the mapping relationship in any grid of the mesh, the two related horizontal curves surrounding the grid are selected as c ₁ and c ₃ and the two related vertical lines surrounding the grid are selected as c ₂ and c ₄ Is done.

  According to the above processing, pixel mapping between the distorted document image and the corrected document image is established by establishing mapping for each point in each grid of the mesh.

  Note that the natural cubic spline method is used to fit curves connecting their intersections, but various other curves such as quadratic curves can also be used and the corresponding interpolation method is used for mapping. May be. Furthermore, a method of simply connecting the intersections with straight lines can be used. In this case, each grid of meshes may be approximated by a square, and each point in the square is interpolated using well-known linear techniques.

  In the final step S350, the corrected document image is acquired by mapping. Specifically, when a mapped image is acquired corresponding to a pixel of a distorted document image, the mapped pixel is rendered with the same color as the corresponding pixel of the distorted document image. FIG. 14 shows an exemplary corrected document image corrected from the distorted image shown in FIG. 8 by the distortion correcting method according to the present invention. It can be seen that the corrected document image looks very flat without distortion due to perspective and warping problems. After correction, the OCR recognition accuracy using the corrected document image is greatly improved compared to the distorted document image.

  The method and system of the present invention can be implemented in many ways. For example, the methods and systems of the present invention can be implemented by software, hardware, firmware, or any combination thereof. The above-described order of steps for the method is intended to be exemplary only, and the steps of the method of the invention are not particularly limited to the order described above unless otherwise indicated. Further, in some embodiments, the present invention may be implemented as a program recorded on a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Therefore, the present invention also includes a recording medium for storing a program for realizing the method according to the present invention.

  While several specific embodiments of the present invention have been demonstrated in detail using examples, those skilled in the art will appreciate that the above examples are intended to be illustrative only and are not intended to limit the scope of the invention. Should be understood. It should be understood by those skilled in the art that the above embodiments can be modified without departing from the scope of the spirit of the present invention. The scope of the present invention is defined by the appended claims.

Claims (9)

A method for correcting geometric deformation in a distorted document image of an original,
A vertical vanishing point detecting step of detecting a vertical vanishing point of the distorted document image, which is a vertical vanishing point perpendicular to the text line of the document;
An image dividing step of dividing the entire region of the distorted document image into a plurality of image stripes using a plurality of vertical lines derived from the detected vertical vanishing points;
By obtaining the vanishing point of the horizontal direction parallel to the text lines of the document in each of said plurality of image stripes, and the horizontal vanishing point detecting step of detecting a horizontal vanishing point for each of the plurality of image stripes,
Wherein a plurality of vertical lines derived from the vertical vanishing point, a plurality of mesh model formed based on a horizontal line derived from each of the horizontal vanishing point detected for each of the plurality of image stripes, A distortion model generation step for establishing a distortion model describing a mapping relationship between the distortion document image and the corrected document image;
A correction step of generating a corrected document image based on the distortion model;
A method characterized by comprising: The vertical vanishing point detecting step includes
Extracting a plurality of vertical strokes of characters from the distorted document image;
A sub-step of fitting the vertical stroke by a plurality of vertical straight line segments;
Calculating the vertical vanishing point from the vertical line segment by searching for an optimal convergence point of the vertical line segment ; and
The method of claim 1, comprising: The horizontal vanishing point detecting step includes:
A sub-step of locating the curve of the text line along the direction of the text line from the distorted document image;
For each image stripe, a sub-step of extracting a curve segment of the text line located in the image stripe;
Fitting the fragment of the curve of the text line by a horizontal straight line segment for each image stripe ;
For each of the image stripe, by searching an optimal convergence point of the horizontal line segments, the sub-steps of calculating the horizontal vanishing point for each of the plurality of images stripes from the horizontal line segments,
The method according to claim 1 or 2 , characterized by comprising: The sub-step of locating the curve of the text line along the direction of the text line from the distorted document image comprises:
An intermediate height point extracting step for extracting an intermediate height point for the pixel of the character of the distorted document image;
A text line curve location step that uses the intermediate height points to locate the text line curve across the intermediate height of the characters of the text line ;
4. The method of claim 3 , comprising: A system for correcting geometric deformation in a distorted document image of a document,
Vertical vanishing point detecting means for detecting a vertical vanishing point of the distorted document image, which is a vertical vanishing point perpendicular to the text line of the original;
Image dividing means for dividing the entire region of the distorted document image into a plurality of image stripes using a plurality of vertical lines derived from the detected vertical vanishing point;
By obtaining the vanishing point of the horizontal direction parallel to the text lines of the document in each of said plurality of image stripes, and the horizontal vanishing point detecting means for detecting a horizontal vanishing point for each of the plurality of image stripes,
Wherein a plurality of vertical lines derived from the vertical vanishing point, a plurality of mesh model formed based on a horizontal line derived from each of the horizontal vanishing point detected for each of the plurality of image stripes, A distortion model generating means for establishing a distortion model describing a mapping relationship between the distortion document image and the corrected document image;
Correction means for generating a corrected document image based on the distortion model;
The system characterized by having. The vertical vanishing point detecting means includes
Means for extracting a plurality of vertical strokes of characters from the distorted document image;
Means for fitting the vertical stroke by a plurality of vertical straight line segments;
Means for calculating the vertical vanishing point from the vertical line segment by searching for an optimal convergence point of the vertical line segment ;
The system of claim 5 further comprising: The horizontal vanishing point detecting means includes
Means for identifying the position of the curve of the text line along the direction of the text line from the distorted document image;
Means for extracting , for each image stripe, a piece of a curve of the text line located in the image stripe;
Means for fitting the fragment of the curve of the text line by a horizontal straight line segment for each image stripe ;
For each of the image stripe, by searching an optimal convergence point of the horizontal line segments, and means for calculating the horizontal vanishing point for each of the plurality of images stripes from the horizontal line segments,
The system according to claim 5 or 6 , further comprising: The means for identifying the position of the curve of the text line along the direction of the text line from the distorted document image comprises:
An intermediate height point extracting means for extracting an intermediate height point for the pixel of the character of the distorted document image;
Text line curve position specifying means for determining the position of the text line curve over the intermediate height of the characters of the text line using the intermediate height point ;
8. The system of claim 7, comprising: The program for making a computer perform each step of the method of any one of Claims 1 thru | or 4.