WO2015141009A1

WO2015141009A1 - Original document distortion correction apparatus, original document distortion correction method, and program

Info

Publication number: WO2015141009A1
Application number: PCT/JP2014/057912
Authority: WO
Inventors: 健李; 貴彦深澤; 夕貴松田
Original assignee: 株式会社Pfu
Priority date: 2014-03-20
Filing date: 2014-03-20
Publication date: 2015-09-24
Also published as: JPWO2015141009A1; JP6194407B2

Abstract

According to the present embodiment, in a case of dividing three-dimensional information into a rectangular mesh, the three-dimensional information is adaptively finely mesh-divided in accordance with a depth, and the three-dimensional information as divided is then extended into a plane.

Description

Document distortion correction apparatus, document distortion correction method, and program

The present invention relates to a document distortion correction device, a document distortion correction method, and a program.

Conventionally, when an image of a binding medium having a thickness such as a book is acquired by an image reading apparatus such as an overhead scanner, the image is distorted due to a change in a three-dimensional shape such as a paper fold or a natural paper float. For this reason, a technique for correcting distortion has been developed.

For example, a distortion component based on a page outline is converted into a height component, and vertical and horizontal mesh lines are generated for a three-dimensional shape. The coordinates of mesh intersections where mesh lines intersect are stored in a mesh table, and based on a mesh model A method for flattening an image has been developed (see Patent Document 1).

In addition, a developable surface having a non-flat distortion is imaged, a first point set representing a three-dimensional profile with respect to the reference surface is generated from the captured image, and a second point set representing a developable mesh is defined as the first point set. A method for texture mapping of an image has been developed in order to adjust the second point set for distortion correction in conformity with the point set (see Patent Document 2).

JP 2013-26830 A Japanese Patent No. 4623898

However, in the conventional three-dimensional distortion correction method, the image is flattened by dividing the mesh at equal intervals, and not only the distortion of the original cannot be accurately reproduced, but also the processing load is large. Had.

In particular, if the conventional method is applied to an overhead scanner, the distance from the sensor to the object to be read is longer than that of a flatbed or ADF (auto document feeder) scanner. The problem is that the correction result is not good when the mesh is easily distorted and there is a significant change point such as a crease between the meshes.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-described problems. An original distortion correction apparatus, an original distortion correction method, and an original distortion correction apparatus that can accurately correct original distortion even when the original is lifted or folded. The purpose is to provide a program.

In order to achieve such an object, the document distortion correction apparatus according to the present invention includes a mesh dividing unit that adaptively finely divides a three-dimensional information into rectangular meshes according to a depth, and the mesh division. Stretching means for stretching the three-dimensional information divided by the means into a plane.

In the document distortion correction method according to the present invention, when three-dimensional information is divided into rectangular meshes, a mesh dividing step for adaptively finely dividing the mesh according to the depth, and the 3 divided by the mesh dividing step. Stretching the dimensional information to a plane.

In addition, when dividing the three-dimensional information into rectangular meshes, the program according to the present invention includes the mesh division step for adaptively finely dividing the mesh according to the depth, and the three-dimensional information divided in the mesh division step. And causing the computer to perform a stretching step of stretching to a plane.

The computer-readable recording medium according to the present invention records the program according to the present invention described above.

According to the present invention, it is possible to accurately correct document distortion even when the document is lifted or folded.

FIG. 1 is a hardware configuration diagram illustrating an example of the information processing apparatus 100. FIG. 2 is a functional block diagram illustrating an example of the information processing apparatus 100. FIG. 3 is a flowchart illustrating an example of overall processing in the information processing apparatus 100 when the positional relationship from the image reading apparatus 12 to the document table is fixed. FIG. 4 is a flowchart illustrating an example of overall processing in the information processing apparatus 100 when the positional relationship from the image reading apparatus 12 to the document table is variable. FIG. 5 is a diagram schematically showing the relationship between a document and a document table on which the document is placed. FIG. 6 is a flowchart illustrating an example of the cropping process in the information processing apparatus 100 according to the present embodiment. FIG. 7 is an output conceptual diagram of the cropping process in FIG. FIG. 8 is a flowchart illustrating another example of the cropping process in the information processing apparatus 100 according to the present embodiment. FIG. 9 is an output conceptual diagram of the cropping process in FIG. FIG. 10 is a flowchart illustrating an example of document table depth estimation processing in the information processing apparatus 100 when the positional relationship from the image reading device 12 to the document table is unknown. FIG. 11 is a diagram schematically showing a document placed on a document table. FIG. 12 is a flowchart illustrating an example of mesh division processing in the information processing apparatus 100. FIG. 13 is a diagram schematically showing how the document area is divided into meshes. FIG. 14 is a diagram schematically showing how the document area is divided into meshes. FIG. 15 is a diagram schematically illustrating the result of the final mesh division. FIG. 16 is a diagram schematically showing a spring model applied in the present embodiment. FIG. 17 is a flowchart illustrating an example of decompression processing in the information processing apparatus 100. FIG. 18 is a diagram showing an old mesh set before expansion. FIG. 19 is a diagram showing a new mesh set after expansion. FIG. 20 is a flowchart illustrating an example of mapping processing in the information processing apparatus 100. FIG. 21 is a diagram schematically illustrating a mapping process between a mesh set and an RGB image.

Embodiments of a document distortion correction apparatus, a document distortion correction method, and a program according to the present invention will be described below in detail with reference to the drawings. In addition, this invention is not limited by this embodiment. In particular, in this embodiment, the reading target may be described as a manuscript such as a magazine. However, the reading target is not limited to this. Newspaper, a medium bound with staples, a stack of single sheets, and the like are read. Also good.

[1. Configuration of this embodiment]
The configuration of the information processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 1 is a hardware configuration diagram illustrating an example of the information processing apparatus 100.

As shown in FIG. 1, the present embodiment includes an information processing apparatus 100 that executes a document distortion correction method, and an image reading device 12 that acquires a document image. The information processing apparatus 100 includes a storage unit 106 and a control unit 102, and the image reading apparatus 12 includes a pattern light source 121 and an image reading unit 122. These units are communicably connected via an arbitrary communication path. The image reading unit 122 may be either a linear sensor or an area sensor, or may include two types of sensors. In the present embodiment, an example of acquiring three-dimensional information by irradiating the original with the pattern light source 121 is described. However, the present invention is not limited to this, and the TOF (Time Of Flight) method is used without using the pattern light source. Three-dimensional information may be acquired.

Here, FIG. 2 is a functional block diagram illustrating an example of the information processing apparatus 100. The storage unit 106 stores various databases, tables, files, and the like. The storage unit 106 is a storage unit, and for example, a memory device such as a RAM / ROM, a fixed disk device such as a hard disk, a flexible disk, an optical disk, or the like can be used. The storage unit 106 stores computer programs for giving instructions to a CPU (Central Processing Unit) and performing various processes. Here, FIG. 2 is a block diagram illustrating an example of the storage unit 106 of the information processing apparatus 100.

As shown in FIG. 2, the storage unit 106 includes a temporary image data file 106a, a three-dimensional file 106b, and a processed image file 106c, as shown.

Among these, the image data temporary file 106a temporarily stores the image data read by the reading control unit 102a.

Also, the three-dimensional file 106b stores three-dimensional information generated based on the image data stored in the image data temporary file 106a.

Further, the processed image file 106c stores processed image data processed or edited from the image data stored in the image data temporary file 106a and / or the three-dimensional information stored in the three-dimensional file 106b.

The input / output interface unit 108 connects the input unit 112 and the output unit 114 to the control unit 102. As the input unit 112, a mouse, a keyboard, or the like can be used in addition to an image reading unit such as the image reading device 12. As the output unit 114, a display unit such as a monitor or a display, or an audio output unit such as a microphone is used. Can be used.

The control unit 102 includes a CPU that controls the information processing apparatus 100 in an integrated manner. The control unit 102 has an internal memory for storing a control program, a program defining various processing procedures, and necessary data, and performs information processing for executing various processes based on these programs.

As shown in FIG. 2, the control unit 102 is roughly divided as shown in the figure, and includes a reading control unit 102a, a three-dimensional information acquisition unit 102b, a crop processing unit 102c, a mesh division unit 102e, and a decompression processing unit 102f. A color mapping unit 102g.

The reading control unit 102a controls the image reading device 12 as the input unit 112, acquires an image including a document, and stores it in the image data temporary file 106a. For example, the reading control unit 102 a may control the pattern light source 121 to emit pattern light and acquire an image of the document irradiated with the pattern light via the image reading unit 122. More specifically, the reading control unit 102a controls the pattern light source 121 to irradiate the target with a phase pattern, and performs image reading so as to synchronize with the pattern light source. When a plurality of images irradiated with this phase pattern are acquired, a three-dimensional information acquisition 102b described later restores the three-dimensional shape. In addition, the reading control unit 102a may acquire color information (RGB information or the like) of the document without irradiating the pattern light. That is, depth information is acquired by the former, and color information is acquired by the latter.

The three-dimensional information acquisition unit 102b acquires three-dimensional information including depth information based on the image data including the original irradiated with the pattern light stored in the image data temporary file 106a, and stores the three-dimensional information in the three-dimensional file 106b. To do. For example, the three-dimensional information acquisition unit 102b may acquire three-dimensional information based on a known pattern light projection method. Note that the three-dimensional information acquisition unit 102b may acquire three-dimensional information based on a known TOF method.

The crop processing unit 102c performs crop processing on the document area in the three-dimensional information. For example, the crop processing unit 102c may determine the document area based on the color and depth in the three-dimensional information stored in the three-dimensional file 106b, and perform the crop process on the document area. Note that the crop processing unit 102c may acquire the depth information of the background region outside the determined document region for post-processing by the extension processing unit 102f or the like. Note that the crop processing unit 102c stores the three-dimensional information of the document area subjected to the crop processing in the three-dimensional file 106b.

Here, as shown in FIG. 2, in the present embodiment, the crop processing unit 102c determines the color edge from the color information stored in the image data temporary file 106a and the depth from the depth information stored in the three-dimensional file 106b. An edge extraction unit 102d for extracting an edge is provided. Then, the crop processing unit 102c may determine the document area based on the color edge and the depth edge extracted by the edge extracting unit 102d. Here, the crop processing unit 102c may determine the document area by giving priority to the color edge out of the color edge and the depth edge. More specifically, when the distance between the corresponding points of the color edge and the depth edge is short, the crop processing unit 102c prioritizes the color edge to determine the document area in order to perform cropping with an apparent cut rather than the depth. May be.

Here, when the distance between the corresponding points of the color edge and the depth edge extracted by the edge extraction unit 102d exceeds the threshold value, the crop processing unit 102c may determine the document area by giving priority to a point far from the image center. Good. In other words, when the distance between the corresponding points of the color edge and the depth edge exceeds the threshold value, it is considered that no false detection occurs outside the document, and in order to prevent missing documents and increase the reliability of cropping. The crop processing unit 102c prioritizes a point far from the image center among the color edge and the depth edge, and determines the document area.

When the original is a bound medium, the crop processing unit 102c may prioritize the depth edge obtained by scanning in the binding direction and determine the original area. That is, in the case of a thick binding medium, a three-dimensional shape change occurs due to natural paper floating, and therefore a depth edge tends to appear in the binding direction. Therefore, the crop processing unit 102c determines the document area by giving priority to the depth edge obtained by scanning in the binding direction over the color edge. Conversely, the crop processing unit 102c may determine the document area by giving priority to the color edge obtained by scanning in the direction perpendicular to the binding direction over the depth edge.

The mesh division unit 102e divides the three-dimensional information into rectangular meshes. In the present embodiment, the mesh division unit 102e adaptively finely divides the mesh according to the depth when dividing the three-dimensional information into rectangular meshes. More specifically, the mesh division unit 102e repeats the process of further dividing the rectangular mesh into a plurality of rectangles when the error of the approximate plane obtained by dividing the mesh is greater than or equal to a threshold for the three-dimensional information. This makes it possible to finely set the mesh according to paper floats and creases, especially when there are significant change points such as creases between the meshes. The accuracy of correction is improved.

Here, the mesh division unit 102e is not limited to dividing the three-dimensional information including the document region and the background region acquired by the three-dimensional information acquisition unit 102b into rectangular meshes, and the document subjected to the crop processing by the crop processing unit 102c. The three-dimensional information of the area may be divided into rectangular meshes. Thereby, it is possible to handle only three-dimensional information of the document area, and it is possible to reduce the calculation load and remove adverse effects from the background area by using a spring model described later. In the present embodiment, an example in which the three-dimensional information of the background area is removed by the crop processing of the document area will be described. However, the present invention is not limited to this, and a transparent document table is used when the reading control unit 102a performs reading. The three-dimensional information of only the document area can also be obtained by placing the document on the document.

The extension processing unit 102f extends the three-dimensional information divided by the mesh dividing unit 102e into a plane. More specifically, the extension processing unit 102f applies a spring model between the three-dimensional vertices of each mesh-divided region, and then extends the three-dimensional vertices to the reference plane. Here, in the 3D information acquired by the 3D information acquisition unit 102b, the expansion processing unit 102f uses the depth of the background area outside the area determined as the document area by the crop processing unit 102c as the reference plane depth. You may extend to a plane. When the positional relationship between the image reading device 12 and the document table is fixed in advance, the extension processing unit 102f may extend the predetermined depth plane that is set in advance.

The color mapping unit 102g maps the color information stored in the image data temporary file 106a onto the plane data expanded by the expansion processing unit 102f. More specifically, the color mapping unit 102g reflects the color information (RGB information and the like) stored in the image data temporary file 106a from the rectangular mesh shape before expansion to the rectangular mesh shape after expansion. As described above, mapping is performed on the plane data.

[2. Processing of this embodiment]
An example of processing executed by the information processing apparatus 100 configured as described above will be described with reference to FIGS. In the processing of the following embodiment, an example in which not only the original distortion correction method but also the cropping method is implemented will be described. However, the present invention is not limited to this processing example, and the original distortion correction method and the like. Only part of the processing in the description may be the subject of the present invention.

[2-1. Overall processing (part 1)]
An example of overall processing in the information processing apparatus 100 according to the present embodiment will be described with reference to FIG. FIG. 3 is a flowchart illustrating an example of overall processing in the information processing apparatus 100 when the positional relationship from the image reading apparatus 12 to the document table is fixed.

As shown in FIG. 3, first, the three-dimensional information acquisition unit 102b controls the image reading device 12 by the processing of the reading control unit 102a to acquire an image of a document table on which a document is not placed. Then, the depth information of the document table is acquired (step SA-1). This process is performed only for the first time, and since the positional relationship from the image reading device 12 to the document table is fixed, the same document table depth is used after the second time.

Subsequently, the three-dimensional information acquisition unit 102b controls the image reading device 12 by the processing of the reading control unit 102a to acquire the image of the document table on which the document is placed, thereby obtaining the color information of the document and the 3D information. Dimension information (depth information etc.) is acquired (step SA-2). This three-dimensional information includes three-dimensional information of the document area and the background area.

Then, the crop processing unit 102c determines the document area based on the color and depth in the three-dimensional information, and performs the crop process on the document area (step SA-3). Thereby, three-dimensional information of only the document area is obtained.

Then, the mesh dividing unit 102e divides the three-dimensional information into rectangular meshes by adaptively finely dividing the mesh according to the depth (step SA-4). More specifically, the mesh dividing unit 102e repeats the process of further dividing the rectangular mesh into a plurality of rectangles when the error of the approximate plane obtained by dividing the mesh is greater than or equal to a threshold for the three-dimensional information. Then, mesh division is adaptively finely performed according to the distortion.

Then, the extension processing unit 102f extends the three-dimensional information divided by the mesh dividing unit 102e to the plane having the fixed depth obtained in Step SA-1 (Step SA-5). More specifically, the extension processing unit 102f applies a spring model between the three-dimensional vertices of each mesh-divided region, and then extends the three-dimensional vertices to a fixed depth reference plane.

The color mapping unit 102g maps the color information stored in the image data temporary file 106a onto the plane data expanded by the expansion processing unit 102f (step SA-6). More specifically, the color mapping unit 102g reflects the color information (RGB information and the like) stored in the image data temporary file 106a from the rectangular mesh shape before expansion to the rectangular mesh shape after expansion. As described above, mapping is performed on the plane data.

The above is an example of the overall processing in the information processing apparatus 100 of the present embodiment.

[2-2. Overall processing (2)]
Here, another example of the entire process described above will be described with reference to FIG. FIG. 4 is a flowchart illustrating an example of overall processing in the information processing apparatus 100 when the positional relationship from the image reading apparatus 12 to the document table is variable.

As shown in FIG. 4, first, the three-dimensional information acquisition unit 102b controls the image reading device 12 by processing of the reading control unit 102a to acquire an image of a document table on which a document is placed. Document color information and three-dimensional information (depth information, etc.) are acquired (step SB-1). This three-dimensional information includes three-dimensional information of the document area and the background area.

Then, the crop processing unit 102c determines the document area based on the color and depth in the three-dimensional information, and performs the crop process on the document area (step SB-2). Thereby, three-dimensional information of only the document area is obtained.

Then, the crop processing unit 102c estimates and acquires the depth information of the background area, which is the outer area determined as the document area, as the depth information of the plane of the document table (step SB-3). Here, FIG. 5 is a diagram schematically showing the relationship between the document and the document table on which the document is placed. As shown in FIG. 5, since the background area outside the document area represents the depth of the document table, the depth information of the background region can be estimated as the depth of the document table.

Then, the mesh division unit 102e divides the three-dimensional information into rectangular meshes by adaptively finely dividing the mesh according to the depth (step SB-4). More specifically, the mesh dividing unit 102e repeats the process of further dividing the rectangular mesh into a plurality of rectangles when the error of the approximate plane obtained by dividing the mesh is greater than or equal to a threshold for the three-dimensional information. Then, mesh division is adaptively finely performed according to the distortion.

Then, the extension processing unit 102f extends the three-dimensional information divided by the mesh dividing unit 102e to the plane of the document table depth estimated in step SB-3 (step SB-5). More specifically, the extension processing unit 102f applies a spring model between the three-dimensional vertices of each mesh-divided region, and then extends the three-dimensional vertices to the reference plane of the estimated document table depth.

The color mapping unit 102g maps the color information stored in the image data temporary file 106a onto the plane data expanded by the expansion processing unit 102f (step SB-6). More specifically, the color mapping unit 102g reflects the color information (RGB information and the like) stored in the image data temporary file 106a from the rectangular mesh shape before expansion to the rectangular mesh shape after expansion. As described above, mapping is performed on the plane data.

[2-3. Cropping process (1)]
Here, a more specific example of the cropping process in the overall process described above will be described with reference to FIGS. FIG. 6 is a flowchart illustrating an example of the cropping process in the information processing apparatus 100 according to the present embodiment. FIG. 7 is an output conceptual diagram of the cropping process in FIG.

As illustrated in FIG. 6, first, the crop processing unit 102 c performs processing from the color information stored in the image data temporary file 106 a and the depth information stored in the three-dimensional file 106 b by the processing of the edge extraction unit 102 d, respectively. Color edges and depth edges are extracted (step SC-1). Here, MA-1 in FIG. 7 is a diagram showing a document binding medium before extraction, MA-2 is an example of color edge extraction, and MA-3 is an example of depth edge extraction. The broken line represents a portion where the edge could not be extracted for some reason. The white line represents the color edge, and the alternate long and short dash line represents the depth edge.

Referring back to FIG. 6, the crop processing unit 102c detects a contour for each of the extracted color edge and depth edge (step SC-2). Here, the crop processing unit 102c uses the color and the depth to improve the reliability of the cropping as follows.

That is, the following processes of SC-3 to SC-8 are repeatedly executed for the entire image.

In the repetitive processing, the crop processing unit 102c detects corresponding points for the outermost contours of color and depth (step SC-4).

Then, the crop processing unit 102c determines whether or not the distance between corresponding points of color and depth is equal to or less than a threshold value (step SC-5).

When the distance between the corresponding points of color and depth is equal to or smaller than the threshold value (step SC-5, Yes), the crop processing unit 102c determines the document area to be cropped using the point of the color edge as the cropping target point (step SC). SC-6). Here, MA-4 in FIG. 7 is a diagram in which the color edge and the depth edge are superimposed. As shown in FIG. 7, both the color edge and the depth edge are detected in the horizontal direction, and in this example, the color edge is selected as the cropping target point because the distance between them is equal to or less than the threshold value. Thereby, it is possible to perform cropping at an apparent break rather than the depth.

On the other hand, when the distance between the corresponding points of color and depth exceeds the threshold (step SC-5, No), the crop processing unit 102c determines the document area to be cropped using a point far from the center of the image as the cropping target point. (Step SC-7). Since it is considered that erroneous detection does not occur outside the original, it is possible to prevent missing of the original and improve cropping reliability.

When the above iterative processing (steps SC-3 to SC-8) is executed for the entire image, the crop processing unit 102c determines that the determined cropping target point group is a cropping target document area ( Step SC-9).

The above is an example of the cropping process in the information processing apparatus 100 of the present embodiment.

[2-4. Cropping process (2)]
Here, another example of the cropping process in the overall process described above will be described with reference to FIGS. FIG. 8 is a flowchart illustrating another example of the cropping process in the information processing apparatus 100 according to the present embodiment. 9 is an output conceptual diagram of the cropping process in FIG.

As shown in FIG. 8, first, the crop processing unit 102c extracts the color edge in the horizontal direction from the color information stored in the temporary image data file 106a by the processing of the edge extraction unit 102d, and stores it in the three-dimensional file 106b. A vertical depth edge is extracted from the obtained depth information (step SD-1). Here, MB-1 in FIG. 9 is a diagram showing the original binding medium before edge extraction, MB-2 is an example of horizontal color edge extraction, and MB-3 is a depth edge in the vertical direction. An example of extraction is shown. The broken line represents a portion where the edge could not be extracted for some reason.

In FIG. 9, white lines represent horizontal color edges, and alternate long and short dash lines represent vertical depth edges. In the cropping process (part 2) of the present embodiment, when the binding medium is a document, a depth edge is likely to appear in the binding direction, and thus a vertical depth edge is detected. On the other hand, in the horizontal direction perpendicular to the binding direction, it is difficult to detect the depth edge, so the color edge is given priority.

That is, as shown in FIG. 8 and FIG. 9 <MB-3>, the crop processing unit 102c starts from the extracted vertical depth edge and continues to the top and bottom continuous edges T (Top), B ( Bottom) is detected (SD-2).

Then, as shown in FIG. 8 and FIG. 9 <MB-2>, the crop processing unit 102c starts from the extracted horizontal color edge and continues to the left and right continuous edges L (Left), R in the image. (Right) is detected (SD-3).

Then, the crop processing unit 102c determines whether or not the shortest distance between the end points of the edges T and B and the edges L and R is equal to or less than a threshold value (step SD-4).

When the shortest distance is equal to or smaller than the threshold (step SD-4, Yes), the crop processing unit 102c integrates the edges T and B and the edges L and R as shown in FIG. The target document area is determined (step SD-5). As a result, in the horizontal direction where the depth edge is difficult to appear, it is possible to perform cropping at an apparent cut.

On the other hand, when the shortest distance exceeds the threshold (No at Step SD-4), the crop processing unit 102c does not use the edges L and R as shown in FIG. The document area to be cropped is determined by connecting both end points of B (step SD-6). Thereby, it is possible to perform cropping at an apparent break rather than the depth. As a result, it is possible to prevent missing originals and perform highly reliable cropping.

The above is another example of the cropping process in the information processing apparatus 100 of the present embodiment.

[2-5. Document table depth estimation process]
An example of document table depth estimation processing in the information processing apparatus 100 according to the present embodiment will be described with reference to FIGS. 10 and 11. FIG. 10 is a flowchart illustrating an example of document table depth estimation processing in the information processing apparatus 100 when the positional relationship from the image reading device 12 to the document table is unknown. FIG. 11 is a diagram schematically showing a document placed on the document table.

As shown in FIG. 10, first, the three-dimensional information acquisition unit 102b refers to the three-dimensional file 106b to determine whether or not the plane information of the document table is acquired in advance (step SE-1).

When the plane information of the document table is acquired in advance (Yes at Step SE-1), the document table depth estimation process is finished. On the other hand, when the plane information of the document table is not acquired yet (Step SE-1, No) The three-dimensional information acquisition unit 102b acquires the plane information of the document table based on the background area outside the document area (step SE-2). More specifically, as shown in FIG. 11, the three-dimensional information acquisition unit 102b obtains a plane ax + by + cz + d = 0 from the three-dimensional information of the background region outside the document region determined by the crop processing unit 102c. Plane approximation is obtained by performing plane approximation from the three-dimensional information of the background area, which is the area of the document table.

The above is an example of document table depth estimation processing in the information processing apparatus 100 of the present embodiment.

[2-6. Mesh division processing]
An example of mesh division processing in the information processing apparatus 100 according to the present embodiment will be described with reference to FIGS. FIG. 12 is a flowchart illustrating an example of mesh division processing in the information processing apparatus 100. FIGS. 13 and 14 are diagrams schematically showing how the document area is divided into meshes. FIG. 15 is a diagram schematically showing the result of the final mesh division.

As shown in FIG. 12, first, the mesh dividing unit 102e forms a rough rectangular mesh on a plane perpendicular to the platen plane with respect to the medium curved surface based on the three-dimensional information of the document area cropped by the crop processing unit 102c. Divide (step SF-1). That is, as shown in FIG. 13, the mesh division unit 102e may equally divide the three-dimensional information of the document area into meshes of a predetermined size.

Then, the mesh dividing unit 102e approximates the three-dimensional point group in each area obtained by dividing the mesh to a plane (step SF-2).

Then, the mesh dividing unit 102e calculates the distance (error) between the three-dimensional point in each area obtained by dividing the mesh and the approximate plane (step SF-3).

When the distance to the approximate plane is equal to or larger than the threshold (step SF-4, Yes), the mesh dividing unit 102e detects the point having the largest distance in the mesh-divided region, and uses the vertical plane passing through this point. Further, it is divided into fine rectangular meshes and a new area is added (step SF-5). As shown in FIG. 14, in the first 6-divided mesh, when the amount of distortion of the upper middle mesh is large and the error is greater than or equal to the threshold value, the mesh dividing unit 102e Divide into rectangular meshes.

On the other hand, if the distance from the approximate plane is less than the threshold value (step SF-4, No), the process is not performed and the process proceeds to the next step.

Then, the mesh dividing unit 102e determines whether or not the check has been completed for all divided mesh regions (step SF-6). If there is a mesh area that has not been determined yet (step SF-6, No), the process returns to step SF-2 for the next area and the above-described processing is repeated. In other words, if there is a difference in the distance between the approximate plane and the actual three-dimensional information, the mesh area is reduced and the same processing is repeated. That is, the region where the depth changes sharply is finely meshed, and the loose portion is meshed in a wide range.

On the other hand, when all the mesh regions have been checked (step SF-6, Yes), the divided mesh set is recorded and the process ends (step SF-7). As shown in FIG. 15, the finally divided mesh region is adaptively finely divided according to the amount of distortion of the document. This makes it possible to finely set the mesh according to paper floats and creases, especially when there are significant change points such as creases between the meshes. The accuracy of correction is improved.

The above is an example of mesh division processing in the information processing apparatus 100 of the present embodiment.

[2-7. Decompression processing]
An example of decompression processing in the information processing apparatus 100 according to the present embodiment will be described with reference to FIGS. FIG. 16 is a diagram schematically showing a spring model applied in the present embodiment. FIG. 17 is a flowchart illustrating an example of decompression processing in the information processing apparatus 100. FIG. 18 is a diagram showing an old mesh set before expansion, and FIG. 19 is a diagram showing a new mesh set after expansion.

In performing the expansion process, a spring model is applied in the present embodiment. As shown in FIG. 16, when the three-dimensional vertex is V (X, Y, Z), the two three-dimensional vertices a and b are contracted and expanded as if there is a spring having a spring coefficient K_d. (See MS Brown's "Document restoration using 3D shape: a general deschewing algorithm for arbitrarily warped documents, Computer Vision, 2001").

As specific decompression processing, as shown in FIG. 17, first, the decompression processing unit 102f demodels the three-dimensional information divided by the mesh division unit 102e and extracts a three-dimensional vertex (step SG-1). .

Then, the extension processing unit 102f calculates the force acting on each vertex of the three-dimensional vertex group (step SG-2). More specifically, the extension processing unit 102f calculates the force between the two vertices a and b based on the following formula.

Then, the extension processing unit 102f updates the speed and the movement amount based on the calculated force acting between the three-dimensional vertices (step SG-3). More specifically, the extension processing unit 102f calculates a position vector and a movement vector from the force between the two vertices a and b based on the following formula.

Then, the extension processing unit 102f determines whether or not all vertex groups have reached the document table plane (predetermined depth) (step SG-4).

If all the vertex groups have not reached the platen plane (SG-4, No), the decompression processing unit 102f returns the process to step SG-2 and repeats the above-described process.

On the other hand, when all the vertex groups have reached the document table plane (SG-4, Yes), the expansion processing unit 102f applies the current expanded length and the expansion to all mesh edges on the document table plane. The positions of the vertices are adjusted by comparing the previous original lengths (step SG-5). That is, the extension processing unit 102f compares the original length before extension shown in FIG. 18 with the length after extension shown in FIG. 19, and adjusts the vertex position of the mesh region.

Then, the decompression processing unit 102f determines whether or not a predetermined convergence condition is satisfied (step SG-6). If the predetermined convergence condition is not satisfied (step SG-6, No), the decompression processing unit 102f returns the process to step SG-5 and performs readjustment.

When the predetermined convergence condition is satisfied (step SG-6, Yes), the extension processing unit 102f finishes the adjustment, acquires new coordinates of each vertex after extension as a new mesh set, and ends the process (step SG-7).

The above is an example of decompression processing in the information processing apparatus 100 of this embodiment.

[2-8. Mapping process]
An example of the mapping process in the information processing apparatus 100 according to the present embodiment will be described with reference to FIGS. FIG. 20 is a flowchart illustrating an example of mapping processing in the information processing apparatus 100. FIG. 21 is a diagram schematically illustrating a mapping process between a mesh set and an RGB image.

As shown in FIG. 20, the color mapping unit 102g acquires the original position (X, Y, Z) of the mesh square vertex before expansion by the expansion processing unit 102f stored in the three-dimensional file 106b (step SH- 1, FIG. 21 <MC-1>).

Then, the color mapping unit 102g acquires the color information (RGB image) stored in the image data temporary file 106a, and acquires the position (u, v) in the corresponding RGB image (step SH-2, FIG. 21 <MC-2>).

Then, the color mapping unit 102g acquires the position (X ′, Y ′, Z ′) of the new mesh square vertex that has been expanded by the expansion processing unit 102f (step SH-3, FIG. 21 <MC-3>). ).

Then, the color mapping unit 102g acquires the position (u ′, v ′) in the corresponding RGB image (step SH-4, FIG. 21 <MC-4>).

The color mapping unit 102g obtains a perspective transformation matrix by using the four vertex RGB image positions (u, v) and (u ′, v ′) (step SH-5).

Then, the color mapping unit 102g obtains new RGB image coordinates of each pixel in the original square using the perspective transformation matrix and designates RGB information (step SH-6).

Through the above processing, the color mapping unit 102g acquires the two-dimensional RGB image subjected to distortion correction, and stores it in the processed image file 106c as processed image data.

The above is an example of the color mapping process of the information processing apparatus 100 of the present embodiment.

[3. Summary of this embodiment and other embodiments]
As described above, according to the present embodiment, when the three-dimensional information is divided into rectangular meshes, the information processing apparatus 100 adaptively finely divides the mesh according to the depth, and expands the divided three-dimensional information on a plane. As a result, even when the original is lifted or folded, the original distortion can be accurately corrected.

In addition, according to the present embodiment, when the error of the approximate plane obtained by mesh division with respect to the three-dimensional information is greater than or equal to the threshold, the process of dividing the rectangular mesh into a plurality of rectangles is repeated, so that the depth is steep. The region that changes to be finely meshed, and the loose part can be meshed in a wide range.

In addition, according to the present embodiment, the 3D information of the document is acquired, the document area is cropped in the 3D information, and mesh division is performed using the cropped 3D information. Information can be handled, and the adverse effect can be removed from the background area by reducing the calculation load and using a spring model described later.

In addition, according to the present embodiment, in the three-dimensional information acquired by the three-dimensional information acquisition unit, by obtaining the depth of the background area of the determined document area, a reference plane for extending to the plane is obtained. Even when the distance to the document table is unknown, the depth to the document table can be estimated.

In addition, according to the present embodiment, since the three-dimensional information of the document placed on the transparent document table is acquired and mesh division is performed using the three-dimensional information, it is possible to handle the three-dimensional information of only the document region. It is possible to reduce the calculation load and remove adverse effects from the background area by using a spring model described later.

Further, according to the present embodiment, color information is mapped onto the expanded plane data, so that an image with corrected document distortion can be acquired.

Furthermore, the present invention may be implemented in various different embodiments other than the above-described embodiments within the scope of the technical idea described in the claims. For example, the image reading unit 122 may detect light in a wavelength region other than the infrared region. Moreover, although the case where the information processing apparatus 100 performs processing in a stand-alone form has been described as an example, processing is performed in response to a request from a client terminal in a separate casing from the information processing apparatus 100, and the processing result is You may make it return to a client terminal. In addition, among the processes described in the embodiment, all or a part of the processes described as being automatically performed can be manually performed, or all of the processes described as being manually performed can be performed. Alternatively, a part can be automatically performed by a known method. In addition, the processing procedures, control procedures, specific names, information including registration data for each processing, screen examples, and database configuration shown in the above documents and drawings may be arbitrarily changed unless otherwise specified. Can do.

Further, regarding the information processing apparatus 100, each illustrated component is functionally conceptual and does not necessarily need to be physically configured as illustrated. For example, the processing functions provided in each device of the information processing apparatus 100, in particular, the processing functions performed by the control unit 102, all or any part thereof are interpreted and executed by a CPU (Central Processing Unit) and the CPU. It may be realized by a program to be executed, or may be realized as hardware by wired logic. The program is recorded on a recording medium to be described later, and is mechanically read by the information processing apparatus 100 as necessary. That is, a computer program for performing various processes is recorded in the storage unit 106 such as a ROM or an HDD. This computer program is executed by being loaded into the RAM, and constitutes a control unit in cooperation with the CPU. The computer program may be stored in an application program server connected to the information processing apparatus 100 via an arbitrary network, and may be downloaded in whole or in part as necessary. is there.

Further, the program according to the present invention may be stored in a computer-readable recording medium, or may be configured as a program product. Here, the “recording medium” includes a memory card, USB memory, SD card, flexible disk, magneto-optical disk, ROM, EPROM, EEPROM, CD-ROM, MO, DVD, and Blu-ray (registered trademark). It includes any “portable physical medium” such as Disc. The “program” is a data processing method described in an arbitrary language or description method, and may be in any format such as source code or binary code. Note that the “program” is not necessarily limited to a single configuration, but is distributed in the form of a plurality of modules and libraries, or in cooperation with a separate program typified by an OS (Operating System). Including those that achieve the function. In addition, a well-known structure and procedure can be used about the specific structure for reading a recording medium in each apparatus shown in embodiment, a reading procedure, or the installation procedure after reading.

Various databases and the like (image data temporary file 106a, three-dimensional file 106b, and processed image file 106c) stored in the storage unit 106 are a memory device such as a RAM and a ROM, a fixed disk device such as a hard disk, a flexible disk, Storage means such as an optical disk, which stores various programs, tables, databases, and the like used for various processes.

Further, the information processing apparatus 100 may be configured as an information processing apparatus such as a known personal computer or workstation, or may be configured by connecting an arbitrary peripheral device to the information processing apparatus. The information processing apparatus 100 may be realized by installing software (including programs, data, and the like) that causes the information processing apparatus to implement the method of the present invention. Furthermore, the specific form of distribution / integration of the devices is not limited to that shown in the figure, and all or a part of them may be functional or physical in arbitrary units according to various additions or according to functional loads. Can be distributed and integrated. That is, the above-described embodiments may be arbitrarily combined and may be selectively implemented.

As described above, the document distortion correction apparatus, the document distortion correction method, and the program according to the present invention can be implemented in many industrial fields, particularly in the image processing field that handles images read by a scanner. Useful.

DESCRIPTION OF SYMBOLS 12 Image reading apparatus 121 Pattern light source 122 Image reading part 100 Information processing apparatus 102 Control part 102a Reading control part 102b Three-dimensional information acquisition part 102c Crop processing part 102d Edge extraction part 102e Mesh division part 102f Decompression processing part 102g Color mapping part 106 Storage Unit 106a temporary image data file 106b three-dimensional file 106c processed image file 108 input / output interface unit 112 input unit 114 output unit

Claims

When dividing the three-dimensional information into rectangular meshes, mesh dividing means for adaptively finely dividing the mesh according to the depth;
Stretching means for stretching the three-dimensional information divided by the mesh dividing means into a plane;
A document distortion correction apparatus comprising:
The document distortion correcting device according to claim 1,
The mesh dividing means is
An original document distortion correction apparatus, characterized by repeating the process of dividing the rectangular mesh into a plurality of rectangles when the error of the approximate plane obtained by dividing the mesh with respect to the three-dimensional information exceeds a threshold value.
In the document distortion correction device according to claim 1 or 2,
3D information acquisition means for acquiring the 3D information of the original;
Crop processing means for cropping the document area in the three-dimensional information;
Further comprising
The mesh dividing means is
An original document distortion correction apparatus that performs mesh division using the cropped three-dimensional information.
The document distortion correction device according to claim 3,
The expansion means is
In the three-dimensional information acquired by the three-dimensional information acquisition means, a reference plane for expansion to a plane is obtained by obtaining the depth of the background area of the document area determined by the crop processing means. Document distortion correction device.
The document distortion correcting device according to claim 2,
3D information acquisition means for acquiring the 3D information of a document placed on a transparent document table;
Further comprising
The document distortion correcting apparatus, wherein the mesh dividing means performs mesh division using the three-dimensional information.
The document distortion correction device according to any one of claims 1 to 5,
A document distortion correction apparatus, further comprising mapping means for mapping color information onto the plane data expanded by the expansion means.
In the case of dividing the three-dimensional information into rectangular meshes, a mesh division step for adaptively finely dividing the mesh according to the depth;
An extension step of extending the three-dimensional information divided in the mesh division step into a plane;
A document distortion correction method comprising:
In the case of dividing the three-dimensional information into rectangular meshes, a mesh division step for adaptively finely dividing the mesh according to the depth;
An extension step of extending the three-dimensional information divided in the mesh division step into a plane;
A program that causes a computer to execute.