RU2820743C1 - Method of producing rectified images of documents folded in half - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to processing of digitized documents. Upper half of the image comprises a larger portion of the upper half of the document, and the lower half of the image comprises a larger portion of the lower half of the document, wherein the height is greater than the width, wherein the original image I is divided in half into an upper part I^t and a lower part I^b, a rough approximation of the halves of the document is constructed by two quadrangles, for this purpose, a quadrangle localization algorithm based on detection of boundaries and lines is applied to both halves, output of which is a set of quadrangles sorted by their contour estimate, thus obtaining a plurality of quadrangles Q^t located on I^t, and a plurality of quadrangles Q^b located on I^b, wherein each pair of quadrangles is used to make a hexagon, which would represent the outer boundaries of the folded document on the image, a correction algorithm is applied to each formed and refined hexagon, after which the hexagons are sorted by their contour estimate.

EFFECT: high quality of scanning folded documents with resistance to occlusion of the edge of the document.

1 cl, 27 dwg

Description

The claimed invention relates to tools for working with documents. In everyday life, there is often a need to digitize documents. Previously, this problem was solved using scanners and recognition systems. With the rise of mobile phones, camera images have become more popular than scanned images. Recognition systems running on such images may experience difficulties. For example, the image of a document (and therefore the text on the document) can be projectively distorted, so one cannot count on high quality recognition of systems designed for scanners. Therefore, before recognition, an additional step is required - obtaining a rectified image - an image of the document, as if it had been scanned on a flatbed scanner. When receiving a rectified image, it is necessary that the contents of the document are not distorted in any way (there are no image breaks, horizontal lines are preserved, etc.). It often happens that business documents are stored folded. Therefore, when the need arises to digitize them, you need to solve the problem of rectifying a document folded in half.

Various methods are known in the prior art for obtaining rectified images of physically warped documents (see Michael S Brown and W Brent Seales, Image restoration of arbitrarily warped documents. IEEE Transactions on pattern analysis and machine intelligence, 26(10):1295-1306, 2004 (1) Shaodi You, Yasuyuki Matsushita, Sudipta Sinha, Yusuke Bou, and Katsushi Ikeuchi. Multiview rectification of folded documents. Hao Feng, Yuechen Wang, Wengang Zhou, Jiajun Deng, and Houqiang Li. Doctr: Document image transformer for geometric unwarping and illumination correction. arXiv preprint arXiv: 2110.12942, 2021 (3); Gaofeng Meng, Chunhong Pan, Shiming Xiang, Jiangyong Duan , and Nanning Zheng. Metric rectification of curved document images. IEEE transactions on pattern analysis and machine intelligence, 34(4):707-722, 2011 (4)), however, they are not designed to solve the problem of unfolding documents folded in half from one image. , obtained from a mobile phone camera: in (1) it is proposed to use external additional equipment, in work (2) it is necessary to have several images at the input, the neural network approach (3) does not take into account projective distortions that arise when shooting with a camera, in work (4) it is superimposed a significant limitation on the document model - it must be represented by a smooth cylindrical surface with guides parallel to the text lines. From reference (5) (see Sagnik Das, Gaurav Mishra, Akshay Sudharshana, and Roy Shilkrot. The common fold: utilizing the four-fold to dewarp printed documents from a single image. In Proceedings of the 2017 ACM Symposium on Document Engineering , pages 125-128, 2017) there is a known method for unfolding documents folded several times - this is the article. However, in (5), firstly, the projective distortions that arise when shooting with a camera are not taken into account, and secondly, when each part of the document is detected separately and then stitched together, a gap in the content occurs at the splice sites.

The proposed method allows you to solve the problem of unfolding folded documents using one image obtained from a mobile phone camera and has the following advantages:

- does not require additional equipment, unlike (1).

- takes one image as input, unlike (2).

- takes into account projective distortions in contrast to (3), (5).

- guarantees that there is no tear on the bend, unlike (5).

In addition, important details are that the proposed method is suitable for running on a mobile phone, and is also resistant to partial occlusion of the edge of the document - for example, if part of it is hidden by the hand.

The objective of the claimed invention is to eliminate the disadvantages of the prior art. The technical result consists in providing a method for obtaining rectified images of documents folded in half, which makes it possible to improve the quality of scanning folded documents using one image obtained from a mobile phone camera, and the presented method does not require additional equipment, takes one image as input, and takes into account projective distortions , ensures no tearing on the fold, is suitable for running on a mobile phone, and is also resistant to partial occlusion of the edge of the document - for example, if part of it is hidden by the hand.

The stated problem is solved, and the stated technical result is achieved through the claimed method of obtaining rectified images of documents folded in half.

The claimed method for obtaining rectified images of documents folded in half is to input a document presentation model, with the upper half of the image containing most of the upper half of the document, and the lower half of the image containing most of the lower half of the document, with the height being greater than the width, and the original image I is divided in half into the upper part I ^t and the lower part I ^b , a rough approximation of the halves of the document by two quadrangles is constructed, for this purpose a quadrilateral localization algorithm based on detection of boundaries and lines is applied to both halves, the output of which is a set of quadrangles sorted by their contour score , thus obtaining a set of quadrilaterals Q ^t located on I ^t and a set of quadrilaterals Q ^b located on I ^b , with each pair of quadrilaterals being used to construct a hexagon that would represent the outer boundaries of the folded document in the image, to each formed and a correction algorithm is applied to the refined hexagon, after that the hexagons are sorted by their contour score and the one with the highest contour score is the output from the localization stage, the last stage is to display the input image determined by two projective transformations.

The figures show:

Fig. 1: Flowchart of the algorithm used in the proposed method.

Fig. 2: Block diagram of the quadrilateral set detection algorithm

Fig. 3: (a) Horizontal (green) and vertical (red) boundaries on the “general” boundary map, (b) detected straight lines (red and green) and a blue line dividing the image in half, (c) a pair of quadrangles (the top one is shown in blue, the bottom one is shown in yellow), the true fold line (green) and the fold line obtained by intersecting vertical segments (red), (d) broken ABCD approximating the boundary break on the track, highlighted by the hatched rectangle in (a), (e) localized hexagon, ( f) boundaries lying along the detected hexagon, (g) rectified image

Fig. 4: Flowchart of hexagon formation algorithm

Fig. 5: Examples of continuous (H1, H2) and discontinuous (H3, H2) transformations

Fig. 6: Examples of unsatisfactory and satisfactory rectification: (a) input image with a detected hexagon (blue) and its correction (red), which is considered invalid, (b) rectification using a red hexagon, (c) input image with a detected hexagon (blue) and its correction (green), which is recognized as valid, (d) rectification using the green hexagon

Fig. 7: Example of the algorithm: (a) Original image, (b) detected boundaries, (c) detected straight lines, (d) found quadrilaterals, (e) constructed and corrected hexagon, (f) rectified image

Fig. 8: Example of the algorithm: (a) Original image, (b) detected boundaries, (c) detected straight lines, (d) found quadrilaterals, (e) constructed and corrected hexagon, (f) rectified image

To use the proposed method you must:

- a camera for receiving the input image, for example, a smartphone camera;

- a computing device with any modern ARM or x86 processor, such as a desktop computer or smartphone.

Let there be an image containing an A4 document, which was captured by a camera. This sheet is folded in half so that the fold line is parallel to the short side of this document (hereinafter referred to as the folded document). You need to restore the image of this document as if it had been scanned on a flatbed scanner. Below we will call this image rectified, and the task itself will be called the rectification task.

As a method for solving the problem of rectification of a folded document, it is proposed to detect the hexagon of its outer boundary and projectively transform each of its halves separately. To implement this method, it is proposed to use the following algorithm.

Consider the circuit in Fig. 1. It is assumed that the document presentation model is such that the top half of the image contains most of the top half of the document, and the bottom half of the image contains most of the bottom half of the document, with the height being greater than the width. Therefore, first the original image I is divided in half into the upper part I ^t and the lower part I ^b , and a rough approximation of the document halves is constructed using two quadrangles. To do this, a quadrilateral localization algorithm (Section 1), based on edge and line detection, is applied to both halves, the output of which is a set of quadrilaterals sorted by their contour score. Thus, we obtain a set of quadrangles Q ^t located on I ^t and a set of quadrangles Q ^b located on I ^b .

Each pair of quads is used to construct a hexagon that represents the outer boundaries of the folded document in the image. In order to create such a hexagon, special clarifying techniques are used (section 2). A correction algorithm (Section 3) is applied to each generated and refined hexagon to ensure that it meets the continuity criterion. After this, the hexagons are sorted by their contour score and the one with the highest contour score is the output from the localization stage (Section 4).

The last step of the algorithm is to display the input image, defined by two projective transformations. The image thus obtained is the output of the proposed algorithm. Let us consider in detail all the blocks of the circuit in Fig. 1, highlighted with a dotted line.

1. Finding quadrilaterals

In FIG. Figure 2 shows a block diagram of the quadrilateral set detection algorithm.

Let's consider its stages.

1.1. Border selection

Here and further we will say that the vector has a horizontal direction if the angle between it and the direction vector of the horizontal direction in the image lies in the set , otherwise - that it has a predominantly vertical direction. Finding boundaries in an image begins with performing morphological operations. The image is enlarged by δ _m (=5) in each direction by repeating the boundary pixels (Here and below, all parameters of the proposed algorithm, which are manually adjusted, are denoted by Greek letters, their values are presented in parentheses. The units of measurement of the metric distance coincide with the distance generated by a grid of 1 pixel size .). Then the morphological closure operation is carried out with a window size of ω (=3) pixels. After this, blur is performed using a Gaussian filter with vertical dispersion σ _e (=0.83) and zero horizontal dispersion. In a blurred image, the derivative along the vertical direction is calculated by calculating the absolute values of the difference between neighboring pixels. The above operation is performed channel by channel, after which the results of the work are averaged and a gray image of the boundaries is obtained. It performs suppression of non-maxima - those pixels next to which there are brighter ones or whose value is less than θ (=2) are zeroed, and the rest are filled with a fixed value. Non-zero pixels are then combined into 8-connected components containing at most one pixel in each image column. Such connected components will be called tracks below. Tracks are filtered by the number of pixels: if its length is less than ρ _e (=10%) of the length of the longest track, then all pixels in it are also zeroed. Let us denote the resulting map of horizontal boundaries as E _h . Vertical boundaries are detected similarly and the corresponding boundary map E _v is introduced. The boundary maps E _v and E _h are superimposed on the input image in FIG. 3. a.

1.2. Detection of straight lines

After calculating the boundary maps E _h and E _v , the search for straight lines in the horizontal and vertical directions, respectively, occurs. For the vertical direction, the boundary map is divided into two (horizontally in half): . Images blurred using a Gaussian filter with dispersion σ _h (=1) and blurred boundary maps are obtained . To images the fast Hough transform (6) is applied. For vertical boundaries, the global maximum M is found on the half-image for both images, after which on each of them there are ν _max (=15) local maxima, the values of which are at least ρ ₁ (=20%) of M, and in circle of radius δ ¹ (=10) from which there are no brighter local maxima. For horizontal boundaries, ν _max local maxima in the image are similarly found . The found local maxima are converted into straight lines coordinate-wise using inverse FPC. Three sets of straight lines - two sets of vertical ones and one set of horizontal straight lines L _h - are the result of the work of this stage. In fig. 3.b shows the result of this stage.

1.3. Collection and filtering of quadrilaterals

Let us denote the line dividing the image I in half by I _s . Many quadrangles are formed in each half . Each quadrilateral q ^t is formed by the intersection of four lines: one horizontal line I _h ∈ L _h , a line I _s and two vertical lines . To form the set Q ^t , all such possible quadrilaterals that lie entirely in I ^t are formed. Each quadrilateral is assigned its contour estimate s(q ^t ), proposed in (7) (its analogue will be considered in Section 4), after which all quadrangles are sorted in descending order of this estimate.

The set Q ^b of quadrangles completely lying in I ^b is formed in a similar way.

2. Construction of hexagons

In FIG. Figure 4 shows a block diagram of the algorithm for forming hexagons from two quadrangles. Let's consider its stages.

Here and below, by fold points we mean the points at which the fold line intersects the outer border of the document (point F in Fig. 3.a)

Consider a pair of quadrilaterals q^t∈ Q^t, q^b∈Q^b. To form a hexagon, all segments of these quadrilaterals are used, except those that lie on line I_s. One might think that to obtain a hexagon it is sufficient to intersect the corresponding vertical segments q^t and q^b. However, since in general the document halves are not flat and their outlines are not straight lines, these intersections may not correspond to fold points (Fig. 3c). Because of this, we analyze contours rather than straight lines. Consider the boundary map E_v. Each vertical segment of quadrilaterals q^t, q^b the track from the border map is matched and considered further.

Namely, for each track e _v the number of its pixels at a distance of no more than (=3) from a given segment s. After this, the track with the largest such number is selected and is considered to correspond to the segment s:

If this number is for is greater than ρ _min (=50%) of length s, then such a track is considered valid.

After valid tracks are associated with vertical segments (if found), it is checked whether there is a pair of vertical segments s ^t ∈ q ^t , s ^b ∈ q ^b such that . We will denote this and we will say that it corresponds to two vertical segments. If such a pair of segments exists on both the left and right sides of the image, up to four hexagons are created - variants of the position of the document - based on the pair of quadrangles in question (q ^t , q ^b ). To form them, we use two primitives: an angle that approximates the edge break caused by folding the paper (Section 2.1) and a fold line (Section 2.2). Options are defined by these primitives as follows: h ₁ is defined by two corners, h ₂ - by a left corner and a fold line, h ₃ - by a right corner and a fold line, h ₄ - by one fold line.

2.1. Finding bend points

Let's consider common for two vertical segments s ^t , s ^b (for example, the red track inside the area marked with a dotted line in Fig. 3.a). Our goal is to find the bend point (F in the same figure) as the point with the greatest break in the boundary.

To find the bend point, we construct a three-link polyline ABCD with a central segment BC and external segments AB and CD on the track. Let's number the points from 0 to N from top to bottom and denote the index of an arbitrary point p of the track as i(p). Starting from an arbitrary point , let's construct an optimal broken line as follows. Consider an arbitrary point as point B for the broken line ABCD. First considering that , let's construct the shortest segment , such that a linear regression built on the points between them will have a square error exceeding ε _c (=0.2) and after that we take point C ^* as the point preceding

Here dist _A denotes the algerbaic distance between a point and a line.

After this, assuming that i(B) > i(A), a segment A ^* B of maximum length is formed in such a way that a similar regression will have a squared error of no more than ε _o (=1). Similarly, assuming that i(C ^* ) < i(D), the segment C ^* D ^* is constructed (Fig. 3.d). The angle corresponding to this broken line is constructed along the segments A ^* B and C ^* D ^* , as the largest angle between the lines containing these segments. The vertex of the angle is the intersection point of the segments A ^* B and C ^* D ^* . If these lines do not intersect, the vertex of the angle is considered to be the middle of the segment BC ^* .

After for each point such a broken line was built, on the track there is

the point with the smallest angle constructed for it. If this angle exceeds (=170 degrees), then it is considered that the desired break on the track was not found. Otherwise, the corner vertex is the output of the algorithm for finding bend points on the track.

2.2. Straight bend detection

Alternative h ₂ uses the left fold point and fold line, alternative h ₃ uses the right fold point and fold line, alternative h ₄ uses only the fold line. First, for each option, we will describe the algorithm for determining the fold line, and then the algorithm for determining the coordinates of the missing vertices.

The fold line for h ₂ and h ₃ is selected from the set L _h , which contains all horizontal lines. The desired fold line is selected as the brightest (in terms of the BPH image) straight line among those straight lines that are located at a distance of no more than (=15) from the bend point. If there are no straight lines in the vicinity of the point under consideration, then the options h ₂ , h ₃ are not considered further.

For h ₄ the fold line is found as follows. First, the intersection points are calculated straight lines containing horizontal segments q ^t and q ^b , with the left and right borders of the image. Then, for each line from the set of horizontal lines, intersections with the same image boundaries are found. After that, those lines are selected for which the intersection with the left boundary lies between at least at a distance (=10) from each, and on the right - between with similar minimum distances from them.

From the selected straight lines, the one that corresponds to the largest value on the BPC image is selected, and it will be the desired fold line. After this, we consider tracks (there may be more than one) corresponding to fold lines whose length is at least (=40%) image width. Determining which track corresponds to the fold line is carried out in the same way as in section 2. After this, for each such track a check is made to ensure that at least (=90%) of its length lies inside . If this is not the case for any of the found tracks, then it is no longer included in the consideration.

We then augment the remaining tracks by adding β(=3) pixels to the left in the same image row as its leftmost point and β pixels to the right in the same image row as its rightmost point. After this they intersect with If a track common to a pair of vertical segments is on both the left and the right, then an intersection with the left track is sought. The intersection point (if found) is now treated as the vertex of a hexagon.

For options h ₂ , h ₃ , h ₄ five vertices are known (all but one vertex on the fold line) and the fold line. Without loss of generality, assume that the left vertex is known. To complete the right vertex, the fold line intersects with the right vertical segments q ^t and q ^b . After this, the middle of the segment between these intersections is taken, and it is considered the right internal vertex.

3. Correction of the hexagon to meet the continuity criterion

When a mapping defined by two projective transformations is used to rectify a hexagon corresponding to the outer contour of a document - the top half is projectively mapped to the top half of the rectified image, the bottom half is projectively mapped to the bottom half of the rectified image - there is a possibility that the text will be cut in place connecting the halves (red part of Fig. 5).

It turns out that the following criterion holds for the absence of a discontinuity. Let two quadrilaterals be given on the plane: A ₁ B ₁ CD and A ₂ B ₂ CD, adjacent on the side CD and lying in different half-planes relative to it, and rectangles A ₁ 'B ₁ 'C'D' and A ₂ 'B ₂ 'C'D' with adjacent side C'D' in different half-planes relative to it. Let H ₁ be a projective transformation from A ₁ B ₁ CD to A ₁ 'B ₁ 'C'D', and H ₂ be a projective transformation from A ₂ B ₂ CD to A ₂ 'B ₂ 'C'D'. Let H be a transformation equal to H ₁ on the half-plane in which A ₁ B ₁ CD lies and equal to H ₂ on the half-plane in which A ₂ B ₂ CD lies. Then H is correctly defined if and only if the lines A ₁ B ₁ , A ₂ B ₂ and CD intersect at one point (blue part of Fig. 3).

We transform each of the document layout options in accordance with the criterion. For three horizontal segments of a hexagon, algorithm (8) is used, which calculates three straight lines passing through one point and approximating these segments. Once these lines have been obtained, the vertices of the hexagon are recalculated: the resulting top and bottom lines intersect with the corresponding vertical lines. The resulting fold line intersects with the top vertical segments. Let us denote the operator of transforming a hexagon to a model of a single horizontal vanishing point V. Thus, from a hexagon h a new one is obtained, h'=V(h).

4. Filtering and ranking hexagons

For each hexagon from the set {h}, its contour score is calculated, which is a quantitative measure of the brightness of the hexagon. For this, boundary maps blurred by a Gaussian filter with dispersion σ(=1.83). When calculating the contour estimate, seven segments are considered: the sides of the hexagons and the segment on the fold line. For each of them, the following values are calculated: the sum of pixels p _m along the segment and the number of zero pixels r _m along the segment. The number of zero pixels along each segment is summed and divided by the length of all segments l, resulting in the ratio We also calculate the value of the penalty q proposed in (7). Consider the vertex of the hexagon _vn , which is not located on the fold. To q is added the sum β _p (=10) of the pixels closest to v _n lying outside h along the sides intersecting at v _n . In the case of a vertex lying on the fold line v _c , the sum β _p (=10) of the pixels closest to v _c lying outside h along the fold line is calculated (Fig. 3. f). Then the contour estimate of the hexagon is the number

From the set of hexagons {h'}, select h'* such that

Where (h') - hexagon h, from which h' was obtained under the action of the operator V, R _t (h') and R _b (h') ^_ aspect ratios (9) of the upper and lower half of the hexagon h ^* , calculated in the camera model -pinholes with focal length λ(=0.705) and the principal point in the center of the original image, R ₀ - true aspect ratio equal to for A4 paper.

After this, the distances between the corresponding vertices h' ^* and h ^* = (h' ^* ), as well as the angles between the old and new segments. If any of these angles exceeds (=2.56 degrees), or whichever of these distances is greater than (=1%) of the image height, then in this case we believe that correcting the hexagon when bringing it to the model of a single vanishing point can damage the horizontality of the text, such a hexagon is considered invalid and the output of the localization algorithm is empty.

The need for such verification is dictated by the following. Consider Fig. 6. It shows the results of the algorithm on two different images. In both cases, the found hexagons (shown in blue in Fig. 6. a, c) correspond to the true locations of the document. However, the document in the first image does not correspond to the two-plane model, so that the hexagon h ^* changes greatly under the action of the operator V and the rectification using the adjusted hexagon h' ^* (shown in red in Fig. 6. a) is unsatisfactory (Fig. 6. b). The document in the second image corresponds to the two-plane model, so the hexagon h ^* (shown in green in Fig. 6. c) is considered valid and the rectification is satisfactory (Fig. 6. d).

5. Projective transformation

If the selected hexagon h' ^* is invalid, the original image is returned. Otherwise, each half of the hexagon is projectively transformed (using bilinear interpolation (10)) to the corresponding half of the rectified image (2100 × 2970). This image is the output of the algorithm.

Examples of implementation of the method are presented in Fig. 7 and 8.

The described algorithm can be extended and adapted for localization and projective normalization of once-folded documents with a straight fold, non-parallel to the short sides of the document, intersecting both long sides of the document.

In this case, the document in the image can be represented by a hexagon, and to detect it, it is proposed to use the algorithm from stages 1 and 2.

Let a hexagon h be detected. In order to projectively normalize its upper and lower quadrangles, it is necessary to determine in what respects the fold line divides the sides of the sheet of paper in the rectified image. These ratios can be calculated using algorithm (11) for restoring the image of a rectangle in the image, applying it to one of the parts of the detected hexagon as an image of a rectangle with aspect ratio R ₀ (see section 4).

The criterion for the absence of a discontinuity when concatenating the images of the upper and lower quadrilaterals can be reformulated as follows.

Let there be two quadrilaterals on the plane: A ₁ B ₁ CD and A ₂ B ₂ CD, adjacent on the side CD and lying in different half-planes relative to it, point V ₁ = A ₁ B ₁ CD, point V ₂ = A ₂ B ₂ CD and rectangle A ₁ 'B ₁ 'B ₂ 'A ₂ ', divided by line C'D' in such a way that point C' lies on side A ₁ 'A ₂ ' and divides it in the ratio x ₁ , and point D' lies on side B ₁ 'B ₂ and divides it in the ratio x _r . Let H ₁ be a projective transformation from A ₁ B ₁ CD to A ₁ 'B ₁ 'C'D', and H ₂ be a projective transformation from A ₂ B ₂ CD to A ₂ 'B ₂ 'C'D'. Let H be a transformation equal to H ₁ on the half-plane in which A ₁ B ₁ CD lies and equal to H ₂ on the half-plane in which A ₂ B ₂ CD lies. Then H is correctly defined if and only if the relation

This relationship, similar to the condition for the existence of a single vanishing point in paragraph 3, imposes a restriction on the relative position of the segments And . Having fixed the position of the segment CD, we can find the following And , the total square deviation of which from the corresponding segments is minimal and at the same time the constraint L is observed (existence is ensured by the fact that the family of required parameters is two-parameter, and the number of imposed conditions is equal to three).

After bringing the segments A ₁ B ₁ , A ₂ B ₂ to correspond to the criterion of the absence of a gap in the concatenation of projectively normalized quadrilaterals, it is proposed to apply stage 4 without taking into account the check of the aspect ratio in choosing the best hexagon, and then projectively correct the best hexagon by mapping its quadrilaterals to the corresponding quadrilaterals of the rectified Images.

Thus, the claimed method allows to achieve the following technical results:

- obtaining images of documents as if they were scanned on a flatbed scanner, without using the latter, using only a camera and a computing device;

- providing the ability to run recognition systems designed for images obtained from a flatbed scanner on rectified images of documents folded in half. Such systems are less resource-intensive and lightweight than recognition systems designed for photographs;

- providing the ability to obtain rectified images on one device equipped with a camera, for example, a smartphone, which allows you to avoid wasting time transferring data between devices. At the same time, it does not require a powerful computer (video card) for fast operation on end user devices;

- providing the ability to store and use physical documents intended for digitization by recognition systems, folded in half;

- providing the ability to cut off cases when the document in the image does not satisfy the presentation model (two flat halves of an A4 page);

- providing the ability to find and rectify documents partially hidden by hand. The ability to hold a document in your hands while shooting significantly improves the user experience;

- the algorithm does not require information about the filling of the document for its geometric normalization.

Bibliography

(1) Michael S Brown and Brent Seales. Image restoration of arbitrarily warped documents. IEEE Transactions on pattern analysis and machine intelligence, 26(10):1295–1306, 2004.

(2) Shaodi You, Yasuyuki Matsushita, Sudipta Sinha, Yusuke Bou, and Katsushi Ikeuchi. Multiview rectification of folded documents. IEEE transactions on pattern analysis and machine intelligence, 40(2):505–511, 2017.

(3) Hao Feng, Yuechen Wang, Wengang Zhou, Jiajun Deng, and Houqiang Li. Doctr: Document image transformer for geometric unwarping and illumination correction. arXiv preprint arXiv:2110.12942, 2021.

(4) Gaofeng Meng, Chunhong Pan, Shiming Xiang, Jiangyong Duan, and Nanning Zheng. Metric rectification of curved document images. IEEE transactions on pattern analysis and machine intelligence, 34(4):707–722, 2011.

(5) Sagnik Das, Gaurav Mishra, Akshay Sudharshana, and Roy Shilkrot. The common fold: utilizing the four-fold to dewarp printed documents from a single image. In Proceedings of the 2017 ACM Symposium on Document Engineering, pages 125-128, 2017

(6) Martin L Brady. A fast discrete approximation algorithm for the radon transform. SIAM Journal on Computing, 27(1):107-119, 1998.

(7) Natalya Skoryukina, Dmitry P Nikolaev, Alexander Sheshkus, and Dmitry Polevoy. Real time rectangular document detection on mobile devices. In Seventh International Conference on Machine Vision (ICMV 2014), volume 9445, pages 458-463. SPIE, 2015.

(8) Julia Shemiakina, Ivan Konovalenko, Daniil Tropin, and Igor Faradjev. Fast projective image rectification for planar objects with manhattan structure. In Twelfth International Conference on Machine Vision (ICMV 2019), volume 11433, pages 450-458. SPIE, 2020.

(9) Zhengyou Zhang and Li-Wei He. Whiteboard scanning and image enhancement. Digital signal processing, 17(2):414-132, 2007.

(10) Anton Trusov and Elena Limonova. The analysis of projective transformation algorithms for image recognition on mobile devices. In Twelfth International Conference on Machine Vision (ICMV 2019), volume 11433, pages 250-257. SPIE, 2020.

(11) Daniil V. Tropin, Ivan A. Konovalenko, Natalya S. Skoryukina, Dmitry P. Nikolaev and Vladimir V. Arlazarov. Improved algorithm of ID card detection by a priori knowledge of the document aspect ratio. In Thirteenth International Conference on Machine Vision (ICMV 2020), volume 11605, pages 407-415. SPIE, 2021.

Claims (1)

A method for obtaining rectified images of documents folded in half, which consists of inputting a document presentation model, wherein the upper half of the image contains most of the upper half of the document, and the lower half of the image contains most of the lower half of the document, the height being greater than the width, characterized in that the original image I is divided in half into the upper part I ^t and the lower part I ^b , a rough approximation of the halves of the document by two quadrangles is constructed, for this, a quadrilateral localization algorithm based on detection of boundaries and lines is applied to both halves, the output of which is a set of quadrangles sorted by their contour estimation, thus obtaining a set of quadrilaterals Q ^t located on I ^t and a set of quadrilaterals Q ^b located on I ^b , with each pair of quadrilaterals being used to construct a hexagon that would represent the outer boundaries of the folded document in the image, a correction algorithm is applied to each generated and refined hexagon, after which the hexagons are sorted by their contour score and the one with the highest contour score is the output of the localization stage, the last stage being the display of the input image defined by two projective transformations.