JP2002077578A - Apparatus and method for correcting distortion as well as distortion correction program recording computer readable recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus for correcting distortion capable of more accurately correcting the distortion of a photographed image generated due to that a height of the surface to be photographed is not uniform without depending upon photographing conditions. SOLUTION: A method for correcting the distortion comprises the steps of conducting a predetermined preprocessing input image data such as an original in a two-page spread or the like (S301), and executing a process of forming an edge image (S303). The method further comprises the steps of detecting the upper end and the lower end of the original from the formed edge image (S305). The method also includes a step of detecting a relative position of an imaging unit and the original from information of the upper and lower ends of the detected original (S307). The method also includes a step of selecting the original end at a position further than the imaging unit of the upper and lower ends of the original (S308). The method also includes a step of calculating height information at each point on the original by using the selected original end information, a known distance from the imaging unit to the original and the relative position of the imaging unit (S309). The method also includes a step of finally correcting a geometrical deformation using the height information for input image data (S311).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a distortion correction apparatus and method, and a computer-readable recording medium on which a distortion correction program is recorded. TECHNICAL FIELD The present invention relates to a distortion correction device and method capable of obtaining an image as if it were a computer, and a computer-readable recording medium that stores a distortion correction program.

[0002]

2. Description of the Related Art When a double-page spread document such as a bound book is photographed from above, a distorted image is obtained because the height from the platen to the surface of the document (surface to be photographed) is not uniform. FIG.
FIG. 2 shows an example of a distorted image obtained when a facing book is photographed from above.

Referring to FIG. 1, a photographed image of spread book 2 has a character string that is distorted and includes a different rotation for each position, as indicated by a circled area a. Furthermore, as in a region b surrounded by a circle, the size of the character is different for each position of the image,
In addition, character deformation occurs. Therefore, it has been difficult to perform character recognition from such a distorted image. It should be noted that such a problem of image distortion is not limited to a character region, but also occurs in a photograph region, a region surrounded by ruled lines, and the like.

The manner in which an image is distorted will be briefly described below with reference to FIGS. 23 and 24 separately for a vertical direction and a horizontal direction of an original image.

First, a description will be given of the distortion generated in the vertical direction. FIG. 23 shows the relationship between the position of a document image in the horizontal direction.
FIG. 7 is a diagram illustrating a state in which the size of an image changes. Referring to this drawing, here, a diagram of the facing book 2 viewed from the side is schematically shown. The imaging unit is located at a point P1 substantially above the center of the facing book 2 (point P0) and above. Facing book 2
The points A and B on the photographing surface have different heights from the platen. For this reason, the vertical distance from a certain point P1 of the imaging unit is also different.

[0006] When the vertical distance from the image pickup unit is different, the size of the image changes accordingly. Specifically, the smaller the vertical distance, the larger the size of the image, and the longer the vertical distance, the smaller the size of the image. Therefore, as shown in the lower diagram, the image size changes according to the horizontal position of the document, that is, according to the height from the platen,
A change in resolution occurs.

Next, the distortion occurring in the horizontal direction will be described. FIG. 24 is a diagram illustrating a state in which the image expands and contracts in the horizontal direction of the document image. The photographing conditions are the same as those in FIG. 23, and also here, a diagram in which the facing book 2 is viewed from the side is schematically shown. The shooting surface of the facing book 2 is a curved surface. For this reason, the image at the position of the equally-spaced point on the facing book 2 is converted to a position with a different space on the plane as shown in the lower diagram. Therefore, the obtained captured image expands and contracts in the horizontal direction.

As a result, as shown in FIG. 25, the original plane image (a) is photographed as an image (b) distorted in both the horizontal and vertical directions.

Conventionally, such image distortion (hereinafter, referred to as “distortion”)
Various methods have been proposed to correct "book distortion". For example, JP-A-5-292314 and JP-A-6-97658 disclose techniques for correcting book distortion by detecting the surface shape of an original using the irradiated spot light and slit light, respectively. Japanese Patent Application Laid-Open No. Hei 9-102854 discloses a technique of detecting a change in height from the side of a document by using a mirror placed beside the document and correcting book distortion.
Further, Japanese Patent Application Laid-Open No. 2000-20682 discloses a technique in which a change in the height direction of a document is calculated based on a distorted state of a document edge, and correction is performed using the calculated change.

[0010]

However, the above-described correction method has the following problems.

That is, in the technique disclosed in Japanese Patent Application Laid-Open No. 5-29231, it is necessary to measure a very large number of positions in order to detect the surface shape of the document, that is, to increase the height detection accuracy. In addition, a sufficient amount of spot light must be secured for accurate measurement. For this reason, there has been a problem that the apparatus becomes large and the processing time becomes long. Further, in order to obtain a final document image, a correction process for eliminating the influence of spot light from the image is also required. Similarly, the technique disclosed in Japanese Patent Application Laid-Open No. 6-97658 also has a problem regarding the amount of slit light and the necessity of a correction process for erasing.

Further, in the technique disclosed in Japanese Patent Application Laid-Open No. Hei 9-102854, since a change in height is detected by using an installed mirror, there is a problem that the positions of a photographing device and a subject are limited.

Further, in the technique disclosed in Japanese Patent Application Laid-Open No. 2000-20682, it is assumed that a contact type scanner in which the position of the original and the image pickup device is fixed is used. It was difficult to apply to the case where is uncertain.

That is, if the relative position is not fixed, the shape of the document edge when the document whose height changes drastically is viewed from almost right above,
The document cannot be distinguished from the document edge shape when a document having a gentle change in height is viewed at an angle. For this reason, there has been a problem that the change in the height direction of the document cannot be specified, and accurate correction cannot be performed.

The present invention has been conceived in view of the above circumstances, and an object of the present invention is to reduce distortion of a photographed image caused by uneven height of a photographing target surface with higher accuracy regardless of photographing conditions. It is an object of the present invention to provide a computer-readable recording medium having a distortion correction device and method capable of correcting the distortion, and a distortion correction program recorded thereon.

[0016]

In order to achieve the above object, according to one aspect of the present invention, a distortion correction apparatus comprises: an image data acquisition unit for acquiring image data of an object photographed by an imaging unit; Detecting means for detecting a desired linear area from the obtained image data, and the height of the photographing surface of the object based on the detected desired linear area and information on the relative position between the imaging unit and the object. Calculating means for calculating the distribution information, and converting means for geometrically converting the acquired image data based on the calculated height distribution information. A desired linear area detected by the detecting means is imaged. It is a line-shaped area located farther from the part.

According to the present invention, the height distribution information of the photographing surface of the object is calculated based on the information on the linear region located farther from the image pickup unit and the relative position between the image pickup unit and the object. You.

Therefore, not only is a case where a double-page spread document whose height from the platen is not uniform is photographed by a stationary image pickup device having a fixed (known) relative positional relationship between the image pickup section and the document, but also by an image pickup device Even when the image is taken by a hand-held imaging device in which the imaging position changes, the height distribution information can be accurately calculated by using the information on the relative position between the imaging unit and the object.

Further, the linear region used for calculating the height distribution is a linear region located farther from the image pickup unit. That is, for example, when the upper end and the lower end of a document can be detected as a line-shaped region, the document end having the larger distortion is used for the calculation. Therefore, it is possible to calculate the height distribution information of the document from the shape of the document edge with higher accuracy.

The image data is geometrically transformed based on the calculated height distribution information. Since the geometric transformation is performed based on the height information calculated with high accuracy, high-precision correction is performed even for the distortion of the captured image.

Therefore, the distortion of the photographed image caused by the uneven height of the photographing target surface can be changed regardless of the photographing conditions.
It is possible to provide a distortion correction device capable of correcting with higher accuracy.

Preferably, the desired line-shaped area includes an end of the object or a continuous line such as a ruled line, a character string, or the like described on the photographing surface of the object.

According to this, the detecting means detects a continuous line described on the photographing surface of the object, such as an edge of the object or a ruled line or a character string, from the acquired image data. When the edge of the target object such as the edge of a two-page spread document is detected, a linear region located farther from the imaging unit can be obtained. When a ruled line or the like is detected, line data can be appropriately obtained as an alternative even when the edge of the object is not included in the captured image.

[0024] Preferably, the distortion correction apparatus further includes position information acquiring means for acquiring information on a relative position between the imaging section and the object, and the information on the relative position relates to a vertical distance between the imaging section and the object. Contains information and horizontal distance information.

According to this, even when photographing is performed with a hand-held image pickup device in which the photographing position of the image pickup device changes,
Information on the vertical distance and the horizontal distance between the imaging unit and the object is acquired by the position information acquiring unit. Therefore, the positional relationship between the imaging unit and the object becomes clear, and it is possible to easily and accurately calculate the height distribution information of the imaging surface of the object.

Here, the information related to the vertical distance means the length of the foot of the perpendicular line dropped from the imaging unit to the object to be photographed (rough distance information).
This means the position information of the perpendicular foot lowered from the imaging unit to the imaging target with respect to the imaging target.

Preferably, the detecting means includes an extracting means for extracting two linear regions from the acquired image data;
Selecting means for selecting a linear area at a position farther from the image pickup unit among the two extracted linear areas, wherein the selected linear area is set as a desired linear area.

According to this, two linear areas are extracted from the acquired image data, and a linear area farther from the image pickup unit is selected. Then, the selected linear area is used for calculating the height distribution information of the target object. As in the case of a hand-held image pickup device, two linear regions are extracted, and when information on the horizontal distance between the imaging unit and the object is obtained based on the extracted two linear regions, the two linear regions are extracted. By using the linear region located farther from the imaging unit, highly accurate calculation of height distribution information can be easily performed.

[0029] More preferably, the selection means includes inclination detection means for detecting the inclination of each of the two extracted linear areas, and based on the detected inclination, the linear area located farther from the imaging unit. Is selected.

According to this, the inclination of each of the two extracted linear regions is detected, and based on the inclination, a line region located farther from the image pickup unit is selected.
Therefore, a desired linear area can be easily selected by comparing the inclination of the linear area.

According to another curved surface of the present invention, the distortion correction method includes an image data acquiring step of acquiring image data of an object photographed by an imaging section, and detecting a desired line-shaped area from the acquired image data. And a calculating step of calculating height distribution information of the imaging surface of the object based on the detected desired linear area and information on the relative position between the imaging unit and the object. A transformation step of geometrically transforming the acquired image data based on the height distribution information obtained, and the desired linear area detected by the detection step is a linear area located farther from the imaging unit. It is characterized by the following.

According to the present invention, the height distribution information of the photographing surface of the object is calculated based on the linear region located farther from the image pickup unit and the information on the relative position between the image pickup unit and the object. You.

Therefore, not only is a case where a double-page spread document whose height from the platen is not uniform is photographed by a stationary type image pickup device in which the relative positional relationship between the image pickup section and the document is constant (known), but also by the image pickup device Even when the image is taken by a hand-held imaging device in which the imaging position changes, the height distribution information can be accurately calculated by using the information on the relative position between the imaging unit and the object.

Moreover, the linear region used for calculating the height distribution is a linear region located farther from the image pickup unit. Therefore, it is possible to calculate the height distribution information of the document from the shape of the document edge with higher accuracy.

Then, based on the calculated height distribution information, the image data is geometrically transformed. Since the geometric transformation is performed based on the height information calculated with high accuracy, high-precision correction is performed even for the distortion of the captured image.

Therefore, the distortion of the photographed image caused by the uneven height of the photographing target surface can be reduced regardless of the photographing conditions.
It is possible to provide a distortion correction method capable of correcting with higher accuracy.

Preferably, the desired line area includes an end of the object or a continuous line such as a ruled line, a character string, etc., which is described on the photographing surface of the object.

According to this, in the detecting step, a continuous line described on the photographing surface of the object, such as an edge of the object or a ruled line or a character string, is detected from the acquired image data. When the edge of the target object such as the edge of a two-page spread document is detected, a linear region located farther from the imaging unit can be obtained. When a ruled line or the like is detected, line data can be appropriately obtained as an alternative even when the edge of the object is not included in the captured image.

Preferably, the distortion correction method further includes a position information obtaining step of obtaining information on a relative position between the imaging unit and the object, wherein the information on the relative position includes information on a vertical distance between the imaging unit and the object. Contains information about horizontal distance.

According to this, even when shooting with a hand-held imaging device in which the imaging position of the imaging device changes,
In the position information acquisition step, information on the vertical distance and the horizontal distance between the imaging unit and the object is acquired. Therefore, the positional relationship between the imaging unit and the object becomes clear, and it is possible to easily and accurately calculate the height distribution information of the imaging surface of the object.

Preferably, the detecting step includes an extracting step of extracting two linear areas from the acquired image data, and a linear area located farther from the imaging unit among the extracted two linear areas. And selecting the selected linear area as a desired linear area.

According to this, two linear regions are extracted from the acquired image data, and a linear region at a position farther from the imaging unit is selected. Then, the selected linear area is used for calculating the height distribution information of the target object. As in the case of a hand-held image pickup device, two linear regions are extracted, and when information on the horizontal distance between the imaging unit and the object is obtained based on the extracted two linear regions, the two linear regions are extracted. By using the linear region located farther from the imaging unit, highly accurate calculation of height distribution information can be easily performed.

More preferably, the selecting step includes a tilt detecting step of detecting a tilt of each of the two extracted linear regions, and based on the detected tilt, a linear region located farther from the imaging unit. Is selected.

According to this, the inclination of each of the two extracted linear areas is detected, and a linear area farther from the imaging unit is selected based on the inclination.
Therefore, a desired linear area can be easily selected by comparing the inclination of the linear area.

According to still another aspect of the present invention, a computer-readable recording medium records a distortion correction program for causing a computer to execute any of the distortion correction methods described above.

According to the present invention, the computer is caused to execute a distortion correction method capable of correcting the distortion of a photographed image caused by the uneven height of the photographing target surface, regardless of the photographing conditions, with higher accuracy. It is possible to provide a computer-readable recording medium on which a distortion correction program for recording is recorded.

[0047]

Next, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows a hand-held image pickup device 1 provided with a distortion correction device according to an embodiment of the present invention,
FIG. 3 is a diagram showing a state in which a photograph is taken. The distortion correction device is provided inside a hand-held image pickup device 1 as shown in FIG. Then, a double-page spread document 2 such as a book is captured by the image pickup device 1.
Is photographed from above, distortion correction processing of the photographed image is performed.

FIG. 2 is a diagram showing a schematic configuration of the entire distortion correction apparatus 200 according to the embodiment of the present invention. With reference to this figure, distortion correction device 200 includes an input unit 201 for inputting image data of a document photographed by image pickup device 1,
A memory 203 that stores a distortion correction processing program executed by the CPU 209 and stores input image data, processing data, and the like; a distance sensor 205 that measures a distance from the image pickup device 1 to a document surface to be photographed; , CPU
An output unit 207 that outputs image data after distortion correction has been performed by the CPU 209, and a CPU 209 that controls these and performs distortion correction processing.

The image data taken by the image pickup device 1 is
Digital data is input from the input unit 201 into the distortion correction device 200. The input image data is temporarily stored in a predetermined area in the memory 203. The stored image data is appropriately read out by the CPU 209 and subjected to various image processing. At this time, the approximate distance data to the document surface measured by the distance sensor 205 is used as necessary. The final image data after the image processing is performed is output from the output unit 207 to the outside of the distortion correction device 200.

Hereinafter, the processing performed by the distortion correction apparatus 200 will be described in detail. FIG. 3 is a flowchart showing the overall flow of the book distortion correction process in the distortion correction device 200. First, before performing the processing of step S301, the distortion correction apparatus 200 obtains information about the approximate distance from the handheld image pickup device 1 to the document surface to be photographed by the distance sensor 205. At this time, the distance sensor 20
5 may calculate the distance using any of the contrast method, triangulation, and the phase difference method. If the document size is known, the approximate distance information may be calculated based on the document size in the image.

Thereafter, a process for correcting the book distortion is performed. Here, for simplification of description, it is assumed that the document is known to be in the left-right spread state. Whether the spread direction of the document is left or right, up or down, or whether the document is rotating can be detected based on the positional relationship between the document edge or the corner of the document. If the image or the coordinate axis is rotated so that the original is in the left-right spread state, the same correction can be performed. Therefore, the detailed description is omitted here, and the left-right spread state is assumed.

When the approximate distance information to the document surface is obtained,
In step S301, preprocessing is performed on image data of a two-page spread document captured by the image pickup device 1.
That is, the input image data is subjected to reduction processing, sharpening processing, smoothing processing, white pixel enlargement processing, and the like.

FIG. 4 is a view showing an image obtained by reducing, processing, sharpening processing, white pixel enlarging processing, and smoothing processing of a two-page spread document shown in FIG. By performing such preprocessing, it is possible to reliably create an edge image to be performed later.

When the pre-processing is completed, in step S303, a process of creating an edge image is performed. For the generation of the edge image, for example, a second derivative process is used. FIG.
FIG. 4 shows a diagram in which the absolute value of the second derivative is calculated based on the image subjected to the preprocessing in FIG. 4 to create a second derivative image. By creating such an edge image, it is possible to reliably detect a document edge to be performed later.

When the creation of the edge image is completed, in step S305, a document edge detection process is performed. In this step, both the upper end and the lower end of the document are detected because they are necessary for the relative position detection processing of the image pickup device 1 to be performed later. For details of the detection method,
It will be described later.

When the end of the document is detected, step S307 is performed.
At the upper and lower edge information of the detected document,
A detection process is performed on the relative position between the document and the document.
Here, the relative position is position information indicating where the foot of the perpendicular line is located on the photographed document image when the perpendicular line is dropped from the image pickup device 1 on the plane on which the document is placed. is there. If the relative position and the distance information between the document and the image pickup device 1 are not known, the shape of the detected document edge is
Accurate height information cannot be obtained.

FIGS. 8 to 10 are diagrams showing a case where the same document edge shape is obtained when the document edges are detected for documents having different heights. In any case, consider the case where the image pickup device 1 exists at the point P1 and the upper end of the document is detected.
In addition, the point P0 is a leg of a perpendicular line lowered from the imaging device 1. 3A is a perspective view showing a photographing positional relationship between the image pickup device 1 and the document, and FIG. 3B is a plan view of the document viewed from above.

For example, as shown in FIG. 8, a document edge shape 1 obtained when a thin double-page spread document is viewed from the image pickup device 1 at a considerably acute angle, and a thick document as shown in FIG. FIG. 10 shows a document end shape l ′ obtained when viewed.
As shown in FIG. 7, when a thin document is viewed from almost right above and the document size is large, the document end shape l ″ is obtained.
Are similar in shape. Therefore, in these cases, if the relative position information (position information about the point P0) is not known, a change in height from the upper end cannot be accurately detected.

When the detection of the relative position of the image pickup device 1 is completed,
Next, in step S308, a document edge selection process for calculating document height information is performed. That is,
Among the upper and lower ends of the document extracted in step S305, the image pickup device 1 to be used for calculating the next height information
Of the original located farther from the original is selected.

When the end of the original is selected, step S309 is performed.
, A process of calculating height information is performed based on the shape of the selected document edge. That is, the height information at each point on the document is calculated using the approximate distance from the image pickup device 1 to the document, the relative position of the image pickup device 1, and the document edge information.

It is assumed that a document image having a spread in the left-right direction has a small change in height in the vertical direction. Therefore, here, the height information in the left-right direction is calculated using only the upper end of the document edge.

After the height information is calculated, finally, in step S311, a geometric deformation correction process using the height information is performed on the document image data. That is,
A geometric transformation of the image is performed on the photographed document image in such a manner that a curved surface is extended in a horizontal direction (x-axis direction). In the vertical direction (y-axis direction),
The magnification of an image having a different image size and undergoing expansion / contraction is changed.

When the geometric deformation correction processing is completed, the correction processing for book distortion ends. By performing the above-described processing, distortion of an image generated when a double-page spread document is photographed from above is corrected, and the image can be reproduced as if it were a flat document.

Subsequently, the document edge detection processing (S305), the relative position detection processing of the image pickup device (S307), the height information calculation processing (S309), and the geometric deformation correction processing (S311) in the book distortion correction processing of FIG. ) Will be described in detail.

First, the document edge detection process (S
305) will be described. With reference to this figure, a rectangular frame indicates an extension of the entire input image, and a curve L indicates a detected document edge. Further, dotted lines Lv0, Lv1, and L
v2 is a line when the entire input image is divided into four equal parts in the horizontal direction.

Normally, it is considered that the original image always exists near the center of the input image. For this reason, the uppermost continuous edge existing between the dotted line Lv1 and the dotted line Lv2 is recognized as the upper end of the document.

Therefore, the edge pixel existing at the top on the dotted line Lv1 is detected first. Specifically, this is performed by adjusting whether or not a pixel at a position on the dotted line Lv1 of the secondary differential image is larger than a certain threshold value Th1 in order from the top. When a pixel larger than the threshold value Th1 is detected, the pixel is recognized at that time as the uppermost edge pixel.

When an edge pixel on Lv1 is detected, the right edge pixel is sequentially detected. FIG. 7 is an enlarged view of a pixel near A in FIG. 6 for explaining detection of a right edge pixel. Referring to this figure, assuming that the detected edge pixel is a target pixel P, the next edge pixel is detected from five pixels (01 to 05) on the right side of the target pixel position.

First, a pixel having the largest pixel value of the second derivative image is selected from the five right pixels. Next, the pixel value of this pixel is compared with a threshold value Th2 (<Th1) to determine whether or not it is larger. As a result, if larger, this pixel is recognized as an edge pixel. Then, the pixels recognized as the edge pixels are set as the target pixels, and edge pixels are similarly extracted for the five right pixels. In this way, edge tracking is performed sequentially.

By the time the dotted line Lv2 is reached, the threshold T
If a pixel larger than h2 does not exist and the edge is interrupted, the line is not an edge, and a new edge is extracted. That is, first, a pixel below the previously extracted pixel is selected as an edge pixel on the dotted line Lv1, and the edge is similarly traced to the right. By repeating this process as appropriate, the dotted line L
The uppermost continuous edge extending between v1 and the dotted line Lv2 is extracted as the document upper end.

When a continuous edge is extracted, next, a dotted line L
Similar edge tracking is performed on the left side from v1 and on the right side from the dotted line L2. Then, finally, the upper end of the document is detected in the second derivative image.

Similarly, the lower end of the document is detected.
That is, by performing the processing in FIG. 6 on the upper and lower sides, the lower end of the document can be detected. Note that the same operation is performed when it is necessary to detect the left end or the right end.

Since the detected upper edge of the original is detected for the reduced image, it is necessary to correspond to the size of the original image. Therefore, original edge image information is restored by an existing interpolation method such as linear interpolation or cubic convolution.

Next, the process of detecting the relative position between the image pickup device 1 and the document (S307) will be described. FIG. 11 is a diagram for describing a specific example of calculating a relative position from document edge information. Referring to this figure, here, the x axis is taken in the horizontal direction and the y axis is taken in the vertical direction. Then, the relative position is calculated based on the local inclination of the document edge.

First, the inclination angle θ at the upper end point (X, Ya) of the taken two-page spread original 2 and the inclination angle φ at the lower end point (X, Yb) of the same x coordinate value are determined. As a result, when θ and φ are the same and | θ |> | φ |, the position (relative position coordinate) Yc of the image pickup device 1 in the y direction is
Yc = Yb + k × (Yb−Ya) × | φ | If θ and φ have the same sign and | θ | ≦ | φ |, then Yc = Ya−k × (Yb−Ya) × | θ |, and if θ and φ have different signs, Yc = ｛(Ya +
Yb) / 2｝ + ｛(Yb−Ya) / 2｝ × (| φ | − |
θ |) / (| θ | + | φ |).

Here, the coefficient k is a value that is uniquely set each time in accordance with the approximate distance between the image pickup device 1 and the document, the standard size of the subject (or the angle of view), and the like. In the figure, the Y coordinate position indicated by the character string S is the relative position of the imaging device 1. Note that, since the tilt angle is positive in the counterclockwise direction, in this figure, θ is positive but φ is negative.

Instead of calculating the relative position using the local inclination angle of the document edge, the calculation may be similarly performed using the inclination of a character string, rectangle, ruled line, or the like in the document. This is because both the upper and lower edges of the document are not necessarily captured in the image. The extraction of the inclination angle is performed by a known method such as measurement after the black pixel is enlarged by reducing the image, enlarging the black pixel, or smoothing.

In the case where the inclination is extracted using the document edge, the character string, or the like, if the inclination is not detected and a local region where the inclination angle is 0 is detected, the position may be determined as a relative position. Good. For example, in FIG. 11, a region including a character string S having a tilt angle of 0 is determined as a relative position. Further, the relative position of the image pickup device 1 may be calculated based on the inclination angles obtained for a plurality of regions such as a region surrounded by a dotted line in FIG. 12 in order to improve the accuracy of the relative position.

The information on the inclination angles θ and φ shown in FIG. 11 is also used in the process of selecting the edge of the document used for calculating the height change information (S308). That is, by comparing the detected inclination angles, it is possible to determine which document end is far from the image pickup device 1. In the example of FIG. 11, since θ> φ,
It is determined that the upper end of the document is located farther from the imaging device. Therefore, the upper end is selected as the document end used for calculating the height information.

Next, the process of calculating the height information will be described with reference to FIG. 13 and FIG. FIG. 13 is a perspective view showing the positional relationship between the imaged document end L and the image pickup device 1. FIG. 14 is a diagram for explaining the positional relationship when FIG. 13 is viewed from the direction of the arrow.

Referring to FIGS. 13 and 14, point P1 is the position of image pickup device 1, and point P0 is the point P on the plane on which the original is placed.
Point A ', the foot of the perpendicular line lowered from 1, the highest position on the document edge, and point B', the position on any document edge. A curve L indicates the shape of the edge of the captured document.

For this reason, for example, the height information of the point A ′ is obtained by using the similarity relation of a triangle, A′A = P1P0 · A
It can be calculated as 2A / A2P0. Similarly,
The height information of the point B ′ is also B′B = P1P0 · B2B / B2
It can be calculated as P0.

Here, P1P0 is the approximate distance from the image pickup device 1 to the document. Further, since P0P is obtained as a relative position of the image pickup device 1 with respect to the document, A2P0 and B2P
P0 is obtained from P0P and each coordinate value.

Finally, the geometric deformation correction processing using the height information (step S311) will be described. FIG.
FIG. 3 is a diagram conceptually showing a state of being geometrically deformed in the horizontal direction. FIG. 15A shows the shape of the original surface before deformation, and FIG. 15B shows the shape of the original surface after deformation. Referring to this drawing, the geometric deformation is divided into M0, M1, M2,... In order to equally divide the plane after the deformation, and each of the grid points (M0 to M9) on the curved surface before the transformation is performed. This is done by finding the position.

The manner in which an image is converted by such geometric deformation will be described in more detail with reference to FIG. FIG.
Referring to FIG. 6, the data of the image before conversion to be placed at the grid point M1 after conversion is M0M1 = N0N on the document edge.
This is the data at the position N1 that becomes 1. This N1 point is obtained based on the slope near N0N1 (where M
0 = N0). In addition, as the inclination near N0N1, the inclination on the curved surface at the position near M1 may be calculated from the height information and used.

When the N1 point is obtained, then, M1M2 = N
The position N2 that becomes 1N2 is similarly obtained. Then, the data at the point N2 is converted into the lattice point M2 on the plane. Similarly, a position N3 that satisfies M2M3 = N2N3 is obtained, and the data at N3 points is converted to M3 points. By repeating this process, geometric transformation in the horizontal direction is performed.

FIG. 17 is a diagram conceptually showing how book distortion is corrected by geometric deformation correction based on height information. With reference to this figure, the distorted image shown in FIG. 17A is expanded in the horizontal direction as shown in FIG. 17B by the above-described geometric transformation in the horizontal direction.
The image is converted into an image having a uniform resolution.

Then, the image corrected in the horizontal direction shown in FIG. 17B is subjected to a magnification conversion process in the vertical direction, and finally, as shown in FIG. A planar image in which the change in size is also corrected can be obtained.

The magnification conversion process in the vertical direction is performed using the height information calculated from the document edge shape and the shooting distance (approximate distance) of the image pickup device 1. That is, as described with reference to FIG. 23, the difference in image size in the vertical direction depending on the horizontal position is due to a change in height from the document to the image pickup device. Therefore, the height from each point of the document to the image pickup device can be obtained from the height information and the approximate distance, and the magnification can be converted in the vertical direction using this. For example, geometric transformation is performed for each vertical line based on the highest position or the lowest position on the document.

In FIG. 17, magnification conversion in the vertical direction is performed after geometric conversion in the horizontal direction, but the processing order may be reversed. In order to increase the speed, only the coordinate calculation may be performed in advance, and both the horizontal and vertical conversion processes may be performed collectively.

In the above description, a case has been considered in which the distortion correction apparatus 200 according to the present embodiment is used for the hand-held image pickup device 1. However, the distortion correction device 200 can also be used for a stationary image pickup device. FIG. 18 shows a stationary image pickup device 18 provided with a distortion correction device 200.
Is a view showing a state in which a double-page spread document 2 is photographed.

As shown in this figure, a stationary image pickup device 1
In 8, the imaging unit is fixed to the column. Therefore,
Unlike the handheld image pickup device 1, the positional relationship between the document 2 and the image pickup device 18 is known.

For this reason, when the distortion correction device 200 is used for the stationary image pickup device 18, the flow of the distortion correction process is the same as the flow of the correction process in the hand-held image pickup device 1 (FIG. 3).
Is slightly different from the above. That is, in the document edge detection process in step S305 in FIG. 3, not only the upper edge and the lower edge of the document are detected, but only the edge of the document farther than is known in advance.

Since the positional relationship between the document 2 and the image pickup device 18 is known, the relative position detecting process in step S307 is not performed. Further, the document edge selecting process in step S308 is not required. In the height information calculation processing in step S309, the height information is calculated using the far end of the document detected in step S305.

FIG. 19 is a diagram showing an image of the original 2 photographed by the stationary image pickup device 18. As shown in FIG. The photographed image of the original 2 is greatly distorted because the upper end l of the original is located farther from the image pickup device than the lower end l ′. In the stationary image pickup device 18, the upper end l of this document is used to calculate height information.

As described above, even when the distortion correction apparatus 200 is used for the stationary image pickup device 18, height information can be calculated with higher accuracy by using the farther document edge. In particular, since the stationary image pickup device 18 has a known positional relationship between the image pickup device 18 and the document 2, it is possible to immediately detect the far end of the document. Since there is no need to acquire relative position information between the two, the number of processing steps can be reduced. For this reason, it is possible to easily and accurately perform the original distortion correction processing.

In the above description, the case where the photographed double-page spread original has no skew or tilt is considered. Hereinafter, a process in a case where there is a skew or a tilt in the entire original will be described with reference to FIGS.

As shown in FIG. 20A, when there is a skew or a tilt in a two-page spread document, each end point of A to D is obtained based on the detection result of the document end detection process (S305) in FIG. Then, based on these end points, FIG.
These are corrected in advance as shown in FIG. Then, each processing of steps S307, S309, and S311 of FIG. 3 is performed.

If the spatial positional relationship between the two-page spread document and the image pickup device 1 is known, the skew and the tilt may be corrected using the positional relationship information. In this case, the book distortion correction (steps S301 to S311) in FIG. 3 is performed from the beginning using the image data corrected for tilt and the like.

FIG. 21 is a diagram for explaining a method of calculating height information from a photographed document image when the height change is different between the upper end and the lower end of the document due to slack or tilt. Here, the straight line AM, the straight line ST, and the straight line DN are straight lines (horizontal) parallel to the x-axis. Also,
The straight line QR corresponds to a vertical straight line on the document, and is a straight line that is not parallel to the y-axis on the photographed document image due to tilt.

Referring to this figure, first, on a straight line AM (y =
The height information of an arbitrary point on ya) is calculated from the shape of the upper end of the document by the above-described method. Also, on the straight line DN (y
Similarly, the height information of the arbitrary point (= yd above) is calculated from the shape of the lower end of the document. Therefore, the height information Hr of the point Q on the straight line AM and the height information Hr of the point R on the straight line DN can be calculated if the values xq and xr of the respective x coordinates are known.

Here, xq = ｛(xm−xa) (xp−
xs) / (xt−xs)｝ + xa xr = ｛(xn−xd) (xp−xs) / (xt−x
s)｝ + xd. Note that xi (i = a, d, m, n, p, t)
Is the value of the x coordinate at point i. Therefore, x
By using q and xr, the height information Hp of an arbitrary point P on the straight line ST can be calculated by the following equation.

Hp = ｛(yd-yp) Hq + (yp-y
a) Hr｝ / (yd-ya) In this manner, the height information of an arbitrary point on the document is calculated using the distance from the top and bottom edges of the document and the height information of the top and bottom edges of the document. .

In the geometric deformation processing based on the height information, coordinate position conversion is performed in the horizontal direction based on a height distribution on a horizontal line (straight line ST) passing an arbitrary point P, and in the vertical direction, Coordinate position conversion is performed based on height information at an arbitrary point P (or an average value of height information of all points on the OR).

As described above, by performing book distortion correction using two pieces of height information, more accurate distortion correction can be performed.

As described above, according to the distortion correction apparatus 200 of the present embodiment, first, the edges of the original are detected from the image data obtained when the double-page spread originals having different heights are photographed from above. The document edge detected at this time is the document edge located farther from the image pickup device. Next, the height information of the document is calculated by using the detected document edge, the approximate distance (vertical distance) between the image pickup device and the document, and the relative distance (horizontal distance). Then, geometric transformation is performed on the original image using the calculated height information.

As a result, an image of a two-page spread original having a distorted original surface is converted into an image as if it were a flat original. In particular, since the document edge used for calculating the height information is the document edge located farther from the image pickup device, it is possible to obtain more accurate height information. Therefore, the accuracy of the geometric transformation performed based on the height information is also improved. As a result, in the corrected image, the document can be easily identified, and the document image processing can be easily performed.

Furthermore, since relative distance and height information can be obtained based on the document edge shape (edge shape) detected from the photographed image information, an illumination device and a complicated sensor device are not required. In addition, the size of the apparatus can be simplified and reduced.

[0110] The distortion correction processing described here can be realized by causing a computer to execute a distortion correction program for causing the above-described series of processing operations to function. The distortion correction program may be installed in a hard disk in a computer in advance, or may be recorded on a removable recording medium such as a CD-ROM or a magnetic tape. In any case, the distortion correction program is recorded on a computer-readable recording medium.

The computer-readable recording medium includes a tape system such as a magnetic tape and a cassette tape, a magnetic disk (flexible disk, hard disk device, etc.) and an optical disk (CD-ROM / MO / MD).
/ DVD etc., IC card (including memory card) and optical card etc., mask ROM, EPROM, EEPROM, flash RO
A medium that fixedly holds the program, such as a semiconductor memory such as M, may be used.

Further, the medium may carry the program in a fluid manner so that the program is downloaded from the network.

The content stored in the recording medium is not limited to a program but may be data.

In this embodiment, FIG.
As described above, the document edge shape (edge shape) is obtained by detecting the document edge. However, not limited to the edge of the original, an original line-shaped area such as a character string or a ruled line on the original may be detected as an edge. These linear regions are also bent and photographed in the same manner as the document edge due to a change in the height and direction of the document. Therefore, height information can be calculated based on the shape of these linear regions and the like. Note that these shapes can be detected by using the same detection method as the above-described inclination detection method for character strings, rectangles, ruled lines, and the like when calculating the relative position.

Also, the document edge detection processing of FIG. 3 (S305)
Has been described with reference to FIGS. 6 and 7 where the edge of the original is detected. However, there is no limitation to such a detection method, and the number of pixels to be compared at the time of edge tracking may be increased or decreased, or may be detected by another method.

The embodiment disclosed this time is an example in all respects, and should not be considered as limiting. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a state in which a hand-held image pickup device 1 including a distortion correction device according to an embodiment of the present invention captures a two-page spread document 2;

FIG. 2 shows a distortion correction device 2 according to an embodiment of the present invention.
FIG. 1 is a diagram showing a schematic configuration of the whole 00.

FIG. 3 is a flowchart showing an overall flow of a book distortion correction process in the distortion correction device 200.

FIG. 4 is a diagram showing an image obtained when a sharpening process, a white pixel enlarging process, and a smoothing process are sequentially performed after reducing a photographed image of a two-page spread document illustrated in FIG. 22;

FIG. 5 is a diagram in which the absolute value of the second derivative is calculated based on the image subjected to the preprocessing in FIG. 4 to create a second derivative image.

FIG. 6 is a diagram for explaining a document edge detection process (S305).

FIG. 7 illustrates the detection of a right edge pixel.
FIG. 7 is an enlarged view of a pixel near A in FIG. 6.

FIG. 8 is a diagram for explaining a document edge shape 1 obtained when a thin double-page spread document is viewed from the image pickup device 1 at a considerably acute angle.

FIG. 9 is a diagram for explaining a document end shape l ′ obtained when a thick document is viewed from almost right above.

FIG. 10 shows a document edge shape l ″ obtained when a thin document is viewed from almost right above and the document size is large.
FIG.

FIG. 11 is a diagram for describing a specific example of calculating a relative position from document edge information.

FIG. 12 is a diagram showing a case where a relative position of the image pickup device 1 is calculated based on a plurality of inclination angles.

FIG. 13 is a perspective view showing a positional relationship between a photographed document end L and an image pickup device 1.

FIG. 14 is a diagram for explaining a positional relationship when FIG. 13 is viewed from the direction of the arrow.

FIG. 15 is a diagram conceptually showing a state of being geometrically deformed in the horizontal direction.

FIG. 16 is a diagram for explaining in more detail how an image is converted by geometric deformation.

FIG. 17 is a diagram conceptually showing how book distortion is corrected by geometric deformation correction based on height information.

FIG. 18 is a diagram illustrating a state in which the stationary image pickup device 18 provided with the distortion correction device 200 captures a two-page spread original 2;

FIG. 19 is a diagram showing an image of the document 2 photographed by the stationary image pickup device 18.

FIG. 20 is a diagram for describing processing when there is a skew or a tilt in the entire document.

FIG. 21 is a diagram for explaining a method of calculating height information from a captured document image when a change in height is different between the upper end and the lower end of the document due to slack or tilt.

FIG. 22 is a diagram illustrating an example of a distorted image obtained when a facing book is photographed from above.

FIG. 23 is a diagram illustrating a state in which the size of an image changes depending on the position of the document image in the horizontal direction.

FIG. 24 is a diagram illustrating a state in which the image expands and contracts in the horizontal direction of the document image.

FIG. 25 is a diagram illustrating a state in which a planar image is captured as an image distorted in each of a horizontal direction and a vertical direction.

[Explanation of symbols]

1 imager (hand-held type), 18 stationary imager,
200 distortion correction device, 201 input unit, 203 memory, 205 distance sensor, 207 output unit, 209 C
PU.

Continued on the front page F-term (reference)

Claims (11)

[Claims] An image data acquisition unit that acquires image data of an object photographed by an imaging unit; a detection unit that detects a desired linear region from the acquired image data; Line-shaped area, and, based on information about the relative position between the imaging unit and the object,
Calculating means for calculating height distribution information of the imaging surface of the object; andconversion means for geometrically transforming the acquired image data based on the calculated height distribution information. The desired linear area to be detected is
A distortion correction device, wherein the distortion correction device is a linear region located farther from the imaging unit. 2. The object according to claim 1, wherein the desired line-shaped area includes an end of the object or a continuous line such as a ruled line or a character string described on an imaging surface of the object. Distortion correction device. 3. The apparatus according to claim 1, further comprising a position information acquisition unit configured to acquire information on a relative position between the imaging unit and the object, wherein the information on the relative position includes information on a vertical distance between the imaging unit and the object and horizontal information. The distortion correction device according to claim 1, wherein the distortion correction device includes information on a distance. 4. The detection unit includes: an extraction unit configured to extract two linear regions from the acquired image data; and a line located at a position farther from the imaging unit among the extracted two linear regions. 4. The distortion correction device according to claim 1, further comprising: selecting means for selecting a linear region, wherein the selected linear region is set as the desired linear region. 5. 5. The method according to claim 1, wherein the selecting unit includes a tilt detecting unit configured to detect a tilt of each of the two extracted line-shaped regions. A line located at a position farther from the imaging unit based on the detected tilt. The distortion correction apparatus according to claim 4, wherein a shape region is selected. 6. An image data acquiring step of acquiring image data of an object photographed by an imaging unit; a detecting step of detecting a desired linear region from the acquired image data; Line-shaped area, and, based on information about the relative position between the imaging unit and the object,
A calculating step of calculating height distribution information of the imaging surface of the object; anda converting step of geometrically converting the obtained image data based on the calculated height distribution information, The distortion correction method, wherein the detected desired linear region is a linear region located farther from the imaging unit. 7. The object according to claim 6, wherein the desired line-shaped area includes an end portion of the object or a continuous line such as a ruled line or a character string described on a photographing surface of the object. Distortion correction method. 8. A position information obtaining step of obtaining information on a relative position between the imaging unit and the object, wherein the information on the relative position is information on a vertical distance between the imaging unit and the object and horizontal information. The distortion correction method according to claim 6, further comprising information on a distance. 9. The detecting step includes: extracting two linear regions from the acquired image data; and a line located at a position farther from the imaging unit among the extracted two linear regions. 9. The distortion correction method according to claim 6, further comprising: selecting a linear region, wherein the selected linear region is set as the desired linear region. 10. The selection step includes a tilt detection step of detecting a tilt of each of the two extracted linear regions. A line located farther from the imaging unit based on the detected tilt. The distortion correction method according to claim 9, wherein a shape region is selected. 11. A computer-readable recording medium storing a distortion correction program for causing a computer to execute the distortion correction method according to claim 6.