JP2016171429A - Image correction device, image correction method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform, in a shorter time, curvature correction on an image obtained by picking up a curved paper surface.SOLUTION: A CPU acquires a synthesis image by processing: a step S32 of extracting an outline of a page from the inside of an image; a step S36 of performing keystone correction on distortion of the image on the basis of four corners of the extracted outline; a step S38 of dividing the image on which the keystone correction has been performed into a correction necessary area and a correction unnecessary area with respect to correction in the longitudinal direction; a step S40 of performing curvature correction in the longitudinal direction on the correction necessary area in the longitudinal direction; a subsequent step S42 of synthesizing the correction unnecessary area and the correction necessary area having been corrected in the longitudinal direction before dividing the image on which the keystone correction has been performed into a correction necessary area and a correction unnecessary area with respect to correction in the lateral direction; a step S44 of performing curvature correction in the lateral direction on the correction necessary area in the lateral direction; and a step S46 of synthesizing the correction unnecessary area and the correction necessary area having been corrected in the lateral direction.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image correction apparatus, an image correction method, and a program.

  2. Description of the Related Art Conventionally, a technique has been proposed in which a book is photographed with a camera while turning the page as it is without being cut, and digitized. When a book is opened in a natural state and photographed from above, the character strings and diagrams in the image are distorted due to the bending of the pages of the book, and it is difficult to read the character strings and diagrams in the image.

  Therefore, for example, Patent Document 1 proposes an image processing technique for correcting the curved surface (distortion) of a book page based on the upper and lower contour lines of the book. Patent Document 2 proposes a technique for correcting a curved surface (distortion) of a book page using a distortion component, a base point, a mesh, projective transformation, and the like.

JP 2013-004088 A JP 2012-170049 A

  In recent years, portable information terminals with camera functions such as smartphones and tablets have become widespread, and there is a technology that corrects the curvature of an image of a book page and generates an image with no distortion in character strings and diagrams in the image. desired. However, the algorithm according to the prior art has a problem that it takes time if processing is performed on a portable information terminal having low processing capability.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to correct a curve of an image obtained by capturing a curved sheet in a shorter time.

  An image correction apparatus according to the present invention includes an extraction unit that extracts a contour of a page from an image, a contour correction unit that corrects distortion of the image based on four corners of the contour extracted by the extraction unit, and the contour A region dividing unit that divides the image corrected by the correcting unit into a correction necessary region and a correction unnecessary region, and the correction necessary region divided by the region dividing unit from the upper end to the lower end of the image. A dividing unit that divides the divided regions so as to be arranged in a horizontal direction orthogonal to the binding direction, and a dividing direction divided by the dividing unit in the binding direction. A first correction unit that corrects a length; an estimation unit that estimates a deflection amount in a normal direction of the divided region in an actual page for each of the divided regions corrected by the first correcting unit; Second correction means for correcting the lateral length of each of the divided regions based on the deflection amount estimated by the means, and the n corrections corrected by the first correction means and the second correction means. The image processing apparatus includes: an acquisition unit configured to combine the divided regions and combine the combined region and the correction unnecessary region divided by the region dividing unit to acquire a combined image.

  An image correction method according to the present invention includes a step of extracting a page outline from within an image, a step of correcting distortion of the image based on four corners of the extracted outline, and an image in which the distortion is corrected, A step of dividing the correction required area into a correction unnecessary area, and a divided area extending along the stitch direction from the upper end to the lower end of the divided image. Dividing in a direction so as to be arranged in n directions, correcting the length in the binding direction for each of the divided areas divided in the number n, and correcting the length in the binding direction. Further, for each of the divided regions, a step of estimating a normal direction deflection amount of the divided region on an actual page, and correcting the lateral length of each divided region based on the estimated deflection amount. Step and before Thereby synthesizing the corrected n pieces of divided regions, characterized in that it comprises the steps of synthesizing the said combined area and the divided correction unnecessary area to obtain a composite image.

  The program according to the present invention provides a computer that controls the image correction apparatus with an extraction function for extracting the outline of a page from within an image, and an outline for correcting distortion of the image based on the four corners of the outline extracted by the extraction function. A correction function, a region dividing function for dividing the image corrected by the contour correcting function into a correction required region and a correction unnecessary region, and the correction required region divided by the region dividing function from the upper end to the lower end of the image A dividing function for dividing the divided areas so that n divided areas extending along the stitch direction are arranged in a lateral direction perpendicular to the stitch direction, and for each of the divided areas divided by the dividing function, A first correction function for correcting the length in the eye direction, and for each of the divided areas corrected by the first correction function, the amount of deflection in the normal direction of the divided area in the actual page Correction by the second correction function, the first correction function, and the second correction function for correcting the horizontal length for each of the divided regions based on the deflection amount estimated by the estimation function. And an acquisition function for obtaining a composite image by combining the n divided regions and combining the combined region and the correction unnecessary region divided by the region dividing function. And

  According to the present invention, it is possible to correct the curvature of an image obtained by imaging a curved paper surface in a shorter time.

It is a block diagram which shows the structure of the portable information terminal 1 as an image correction apparatus by embodiment of this invention. It is a flowchart for demonstrating the whole operation | movement by the portable information terminal 1 of this embodiment. It is a flowchart for demonstrating the operation | movement of the image processing by this embodiment. It is a schematic diagram which shows the example of the image at the time of image | photographing the page of a book from right above. It is a schematic diagram which shows the example of the image at the time of image | photographing the page of a book in the diagonal direction from the lower side. It is a conceptual diagram which shows the image G before performing the curvature correction of this embodiment. It is a conceptual diagram which shows the image G1 after the keystone correction by this embodiment. It is a conceptual diagram for demonstrating the division | segmentation process of the vertical direction performed before the curvature correction by this embodiment. It is a conceptual diagram for demonstrating the division | segmentation process of the horizontal direction performed prior to the curvature correction process by this embodiment. It is explanatory drawing which shows outline R1 and image G1 after trapezoid correction | amendment, and rectangular outline R2 which consists of four corners r1, r2, r3, r4 after trapezoid correction | amendment. It is a conceptual diagram for demonstrating the conversion type used by the vertical direction correction process by this embodiment. It is a conceptual diagram for demonstrating the discontinuity which arises in the correction required area | regions 30a and 30b and the correction unnecessary area | region 31 after the vertical direction correction | amendment by this embodiment. It is a flowchart for demonstrating the operation | movement of the vertical direction correction process by this embodiment. It is explanatory drawing which shows the outline R1 and image G1 after trapezoid correction | amendment, and the rectangular outline R2 which consists of the target points r1 ', r2', r3 ', r4' of the four corners after trapezoid correction | amendment. It is a flowchart for demonstrating operation | movement of the horizontal direction correction | amendment process by this embodiment. It is explanatory drawing which shows the deflection amount d (z) in the division area which concerns on this embodiment. It is a conceptual diagram which shows the relationship between ds (i), dz (i), and dx '(i) concerning this embodiment. It is a conceptual diagram for demonstrating operation | movement of the horizontal direction correction | amendment process by this embodiment. In this embodiment, it is a conceptual diagram which shows the example of the image after correction | amendment which changes with the curvature degree of a page.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

A. Configuration of Embodiment FIG. 1 is a block diagram showing a configuration of a portable information terminal 1 as an image correction apparatus according to an embodiment of the present invention. In the figure, the portable information terminal 1 includes, for example, a smartphone or a tablet terminal. The portable information terminal 1 includes a communication unit 10, an imaging unit 11, a ROM 12, a RAM 13, a display unit 14, an operation unit (touch panel) 15, a recording medium 16, and a CPU 17.

  The communication unit 10 is connected to a network such as the Internet using mobile communication, Bluetooth (registered trademark), and / or wireless LAN (WiFi), for example. The imaging unit 11 includes a lens block including an optical lens group and an imaging element such as a CCD or a CMOS, and captures an image entered from the lens block by the imaging element. In particular, in the present embodiment, the imaging unit 11 images a page of a book.

  The ROM 12 stores programs executed by the CPU 17 described later, various parameters necessary for operations, and the like. The RAM 13 stores data such as temporary data when the CPU 17 described later executes a program, various application programs, and various parameters necessary for executing the application. In particular, in the present embodiment, the RAM 13 stores captured images, corrected corrected images, and the like.

  The display unit 14 includes a liquid crystal display, an organic EL (Electro Luminescence) display, and the like, and displays icons associated with specific functions and applications, application screens, various menu screens, and the like. The operation unit (touch panel) 15 detects direct contact or proximity of a finger or a stylus (pen). The operation unit (touch panel) 15 may include mechanical switches such as a power button and a volume button. In this embodiment, parameters for correcting the curved surface (distortion) of the captured image are input from the operation unit (touch panel) 15.

  The recording medium 16 stores various data such as captured image data. The CPU 17 controls the operation of each unit by executing the program stored in the ROM 12 described above. In particular, in the present embodiment, the CPU 17 executes curvature correction on the captured image by executing an image processing program.

  The present embodiment is an image processing technique that partially corrects a curved surface (distortion) of an image of a book or booklet (hereinafter abbreviated as a book) imaged by the imaging unit 11 or another device having an imaging function. is there. In the present embodiment, curvature correction is necessary in the same image (one book image) by taking advantage of the feature that distortion is small in the center part of the photographed one book image and the part away from the binding part. The area is divided into an area (necessary area for correction) and an unnecessary area (unnecessary area for correction), and after the curvature correction is performed on the necessary area for correction, the necessary area for correction and the unnecessary area for correction are connected again. As a result, the number of pixels to be curved is reduced and the image processing speed can be increased.

  Further, in the present embodiment, in order to connect the correction necessary area and the correction unnecessary area without a sense of incongruity, that is, when the correction necessary area and the correction unnecessary area are connected after the curvature correction, the image is not shifted. In order to suppress the effect of the curvature correction on the vicinity of the boundary between the correction necessary area and the correction unnecessary area, the degree of correction is made relatively small in the vicinity of the boundary by multiplying the weight coefficient in the curvature correction processing.

B. Operation of Embodiment Next, the operation of the above-described embodiment will be described.
FIG. 2 is a flowchart for explaining the overall operation of the portable information terminal 1 of the present embodiment. In the portable information terminal 1, the CPU 17 first stores a book image from the recording medium 16 in the RAM 13, inputs parameters for curvature correction from the operation unit (touch panel) 15, or performs curvature correction by another algorithm. Are automatically derived (step S10). Next, the CPU 17 performs image processing for curvature correction based on the parameters (step S12). Details of image processing for curvature correction will be described later.

  When the image processing ends, the CPU 17 outputs the image-processed image, that is, the book image subjected to the curvature correction based on the parameters to the display unit 14 (step S14). The CPU 17 determines whether or not the processed image satisfies a predetermined condition (step S16).

  In the present embodiment, the user confirms the image subjected to curvature correction displayed on the display unit 14 and determines whether sufficient curvature correction has been performed, or whether the degree of curvature correction is too strong or too weak. Instructed from the operation unit (touch panel) 15. However, the present invention is not limited to this, and the CPU 17 may determine whether the distortion of the image after curvature correction is within an allowable range.

  When the processed image does not satisfy the predetermined condition (NO in step S16), the curvature correction parameter for increasing or decreasing the degree of curvature correction from the operation unit (touch panel) 15 by the user is set. Input (step S18). When the CPU 17 determines whether or not a predetermined condition is satisfied, the CPU 17 sets again a parameter for executing curvature correction under a different condition.

  In any case, when the curvature correction parameters are reset, the CPU 17 returns to step S12, performs the above-described image processing for curvature correction, and then displays the book image subjected to the curvature correction on the display unit 14. Repeat to output to. This process is repeatedly executed until the distortion of the image after curvature correction falls within the allowable range, that is, until it is determined that sufficient curvature correction has been performed.

  On the other hand, when the processed image satisfies the predetermined condition (YES in step S16), the CPU 17 outputs the image after the curvature correction to the recording medium 16 (step S20). Next, the CPU 17 determines whether there is another image to be processed (step S22).

  If there is another image to be processed (YES in step S22), the process returns to step S10 and the above-described processing is repeated for the next image. On the other hand, if there is no other image to be processed (NO in step S22), the process ends.

  In the above-described processing, the determination processing in step S16 may be skipped for continuous processing or speed improvement.

  FIG. 3 is a flowchart for explaining the image processing operation according to the present embodiment. FIG. 4 is a schematic diagram showing an example of an image when a book page is photographed from directly above. FIG. 5 is a schematic diagram illustrating an example of an image when a page of a book is photographed obliquely from below.

  First, the CPU 17 reads an image to be processed from the recording medium 16 (step S30). Any image may be used, but an image taken vertically from a position directly above the paper surface is preferable. As shown in FIG. 4, in the image 20 taken perpendicularly from the position directly above the paper surface, the area 21 near the center of the page has little change before and after the curvature correction, and therefore an area that does not need to be corrected. Can be. That is, it can be said that the region 21 can be ignored because the distortion in the vertical direction is small and there is no abrupt change in the horizontal direction. On the other hand, as shown in FIG. 5, an image 22 taken from an oblique lower side without being perpendicular to the paper surface is curved near the center, and there are fewer portions that do not require curvature correction. That is, in the image 22, there is distortion in all areas, and curvature correction is required in all areas.

  Even when the calibration is not performed, the distortion generated from the lens is generated over a wide range, so that an unnecessary area for curvature correction is reduced. Therefore, in order to enjoy the effect of the curvature correction of the present embodiment, as shown in FIG. 4, the calibrated imaging unit (for example, the imaging unit 11) is photographed as vertically as possible from the center of the paper as much as possible. It is desirable.

  Next, the CPU 17 extracts a contour in the image by using known image processing (step S32). FIG. 6 is a conceptual diagram showing an image G before performing the curvature correction of the present embodiment. As shown in FIG. 6, when the points r1 and r4 at both ends of the upper reference line SBUL of the image G and the points r2 and r3 at both ends of the lower reference line SBDL are connected by a straight line, the shape is trapezoidal. Yes. In addition, the open book is distorted in a certain direction around the binding (line segment r1-r2), and all the paper surfaces that are equidistant from the binding (line r1-r2) are the same. It is distorted in the height direction.

  Next, the CPU 17 acquires the coordinate values of the four corners r1, r2, r3, r4 from the contour of the image G (step S34). Next, the CPU 17 performs known trapezoidal correction (perspective transformation) on the contour of the image G using the projection matrix SA based on the acquired coordinate values of the four corners r1, r2, r3, r4 (step S36). ). In this trapezoidal correction, not only the contour of the image G is trapezoidally corrected, but the entire image (all pixels) of the image G is corrected following the contour deformation by the trapezoidal correction.

  FIG. 7 is a conceptual diagram showing an image G1 after keystone correction according to the present embodiment. In FIG. 7, for easy understanding of the drawing, a quadrangle formed by connecting four target points r1 ′, r2 ′, r3 ′, and r4 ′ with straight straight lines is illustrated facing the front.

  The projection matrix SA is a matrix that projects four reference points onto four target points, and the four reference points are points on both ends of the upper reference line SBUL and the lower reference line SBDL of the image G, respectively. The points at both ends of the corrected upper reference line SBUL ′ and the corrected lower reference line SBDL ′ of the image G1 projected by the projection matrix SA are four target points r1 ′, r2 ′, It corresponds to r3 ′ and r4 ′.

  In addition, since the four target points are given as the four corner points of the clean quadrangle, the four target points r1 ′, r2 ′, r3 ′, and r4 ′ are straightened as shown in FIG. The shape connected by a straight line is converted from the trapezoidal shape shown in FIG. 6 into a clean quadrangle (rectangular shape), but the distortion is not completely removed. In the curvature correction processing according to the present embodiment, correction is performed on the image after perspective transformation (after trapezoidal correction) by performing two processes of vertical curvature correction and horizontal curvature correction.

  Next, the CPU 17 divides the image after the trapezoid correction into the correction required areas 30a and 30b and the correction unnecessary area 31 with respect to the vertical correction (step S38).

  FIG. 8 is a conceptual diagram for explaining vertical division processing performed prior to the curvature correction processing according to the present embodiment. As shown in FIG. 8, a region where there is much distortion in the vertical direction is generally the peripheral portion of the image. That is, the upper and lower areas are areas to be corrected, and these areas are extracted as the correction required areas 30a and 30b. The areas other than the correction necessary areas 30a and 30b are set as the correction unnecessary areas 31. Therefore, the image is divided into a total of three images, that is, the images of the two correction necessary regions 30a and 30b (upper and lower) and the image of the one correction unnecessary region 31.

  The method of determining the range of the correction required areas 30a and 30b (where to set the boundary lines BL1 and BL2 to be divided) can be specified manually by the user from the operation unit (touch panel) 15 or at a predetermined fixed position. A method of using a boundary line or deriving and setting a boundary line of an arbitrary page from a reference page can be considered.

  Next, the CPU 17 performs vertical curvature correction on the divided correction required areas 30a and 30b (step S40). In particular, in the present invention, an arithmetic expression used for vertical curvature correction is multiplied by a weight coefficient term that is inversely proportional to the distance from the boundary lines BL1 and BL2. Thereby, it is possible to connect the correction required areas 30a and 30b and the correction unnecessary area 31 without a sense of incongruity. Details of the vertical curvature correction process will be described later.

  Next, the CPU 17 divides the image after the trapezoidal correction into the correction required area 32 and the correction unnecessary area 33 with respect to the horizontal correction (step S42).

  FIG. 9 is a conceptual diagram for explaining horizontal division processing performed prior to the curvature correction processing according to the present embodiment. As shown in FIG. 9, the portion having a large amount of distortion in the lateral direction is generally in the vicinity of the binding portion at the center of the book (the right end in the drawing). Other portions need to be corrected in the horizontal direction, but the change in the interval is slow, so it is not very noticeable. Therefore, the image is divided into two images, that is, a correction required area 32 in the vicinity of the binding portion and the other correction unnecessary area 33.

  As with the vertical division, the method for determining the range of the divided region (where to set the boundary line BL3 to be divided) is determined manually by the user from the operation unit (touch panel) 15 or at a predetermined fixed position. A method of using a line or deriving a boundary line BL3 of an arbitrary page from a reference page is conceivable, but is not particularly limited.

  Next, the CPU 17 performs lateral curvature correction on the divided correction required area 32 (step S44). In particular, in the present invention, similarly to the vertical curvature correction, an arithmetic expression used for the horizontal curvature correction is multiplied by a term of a weighting factor that is inversely proportional to the distance from the boundary line BL3. Thereby, the correction required area | region 32 and the correction unnecessary area | region 33 can be joined without a sense of incongruity. The details of the lateral curvature correction process will be described later.

  Next, the CPU 17 synthesizes the n divided areas U1 that have been subjected to the vertical direction correction process and the horizontal direction correction process, and creates an entire image (step S46). Specifically, by accumulating the lateral length corrected by the lateral direction correction process, n divided areas U1 after the height direction correction process are combined and stored in the recording medium 16. This composition processing may cause a change in the number of pixels. In that case, the vertical and horizontal widths are resized to be the same as the width of the original image, and the number of pixels is made equal. Further, the aspect ratio of the generated image is not necessarily equal to the actual paper surface. Therefore, the aspect ratio is adjusted as appropriate so that the image is not correct or unnatural. Thereafter, the CPU 17 returns to step S14 shown in FIG.

  In this embodiment, the images divided at the completion of correction are combined to simplify the explanation, but the next horizontal division processing and curvature correction processing are performed on each image while being divided, and finally all A method of combining the images is also conceivable.

  FIG. 10 shows the contour R1 and the image G1 of the correction required area 30a (30b) after trapezoid correction, and the target points r1 ′, r2 ′, r3 ′, r4 ′ at the four corners of the correction required area 30a (30b) after trapezoid correction. It is explanatory drawing which shows the rectangular outline R2 which consists of. In the figure, h1 is the height of the rectangular contour R2, and H1 is the height of the contour R1 of the correction required region 30a (30b). In the outline R1 and the image G1 of the correction required area 30a (30b) after the trapezoid correction, the curve in the binding direction (Y direction) remains. If the page is curved, the center portion of the page is close to the image pickup means (camera), so that the image is taken larger than the left and right edges of the page.

  For example, a region U extending along the stitch direction from the upper end to the lower end of the image G1 in the vicinity of the approximate center in the lateral direction of the image G1 is larger (longer) in the stitch direction than the rectangular contour R2. That is, the length in the binding direction of the page varies in the horizontal direction of the page. In order to make this uniform and within the rectangular contour R2, the vertical correction processing is executed on the divided vertical correction required region 30a (30b).

(Vertical curvature correction processing)
FIG. 11 is a conceptual diagram for explaining a conversion formula used in the vertical direction correction processing according to the present embodiment. In the figure, F is a focal point set for convenience. A ′ is the upper point of an arbitrary region U in the contour R1 of the correction required region 30a (30b) after trapezoid correction, B ′ is the lower point of the arbitrary region U in the region U, and P ′ is on the A′-B ′ line. An arbitrary point, A is an upper point of a portion corresponding to the region U in the rectangular contour R2, B is a lower point of a portion corresponding to the region U in the rectangular contour R2, and P corresponds to a point P on the line AB. Shows the point. The coordinates are shown in parentheses.

The following relationship can be seen from FIG.
ΔA'FB'∝ΔAFB
∴FP: FP '= h1: H1
ΔAFP∝ΔA'FP '
∴AP: A'P '= h1: H1
∴AP = (h1 / H1) ・ A'P '
Here, assuming that the Y coordinate value of point A is ya, the Y coordinate value of point P is yp, the Y coordinate value of point A ′ is ya ′, and the Y coordinate value of point P ′ is yp ′,
A'P '= ya'-yp'
AP = ya-yp
It becomes.
∴yp = h1 / H1 * (yp′−ya ′) + ya (1)

However, this equation (1) cannot be directly applied to the correction required region 30a (30b).
FIG. 12 is a conceptual diagram for explaining discontinuities that occur in the correction required area 30a (30b) and the correction unnecessary area 31 after the vertical correction according to the present embodiment. Curves are also actually generated in the divided correction unnecessary area 31, and after correction in the vertical direction, the correction required area 30a (30b) to which the expression (1) is applied and the correction unnecessary area 31 are combined, as shown in FIG. Thus, it becomes discontinuous near the boundary line BL1.

  Therefore, in the vicinity of the boundary line BL1, the conversion rate is suppressed, and a conversion equation having a conversion rate similar to the equation (1) is set as the distance from the boundary line BL1 increases. As an example, a conversion formula including the following weights is given.

yp = (1-m) * {h1 / H1 * (yp'-ya ') + ya} + m * yp' (2)
m = (1 + k * | yp'-ybound |) ^-1 (3)

  Here, ybound is the y coordinate of the boundary lines BL1 and BL2, and k is a coefficient with 0 <k. The coefficient k must be set so that the conversion rate is almost the same as that in the expression (1) at the end of the page. For example, if the coefficient k is a value close to 0, conversion is suppressed over the entire area, and the conversion effect is lost.

Assuming that the boundary of the conversion area opposite to the coupling portion is yc and the coordinate when converted to the plane is yc ′, yc only needs to be a result that does not change in either equation (1) or (2). .
From number (1)-number (2),
yc1-yc2 = h1 / H1 * (yc′−ya ′) + ya− (1-m) * {h1 / H1 * (yc′−ya ′) + ya} −m * yc ′ = 0 (4)

  However, in this equation (4), the coefficient k becomes indefinite. So, the right side is not 0, but smaller than 1 pixel,

h1 / H1 * (yc′−ya ′) + ya− (1-m) * {h1 / H1 * (yc′−ya ′) + ya} + m * yc ′ <1 (5)
Then, the coefficient k is expressed by Equation (6).

  k> {H1 / h1 * (yc'-ya ')-yc' + ya-1} / | yc'-ybound | (6)

  Therefore, the coefficient k that satisfies the condition of Expression (6) is used. The coefficient k is determined by using an amount twice the lower limit value of the coefficient k, or H1 / h1 is 1.2 (empirical maximum amount of one image) to simplify the derivation of H1 / h1. Or a fixed amount. Using this coefficient k, a transformation matrix is created from the transformation equation (2), and vertical curvature correction is executed.

  The conversion equation derived here is merely an example, and if the correction is suppressed in the vicinity of the boundary lines BL1 and BL2 and the suppression term attenuates as it moves away, what value is the coefficient k? It is not limited.

The Y coordinate value of P after conversion of an arbitrary point P can be obtained by the above-described equation (2).
Actually, correction is performed by applying the gradation at the point of the corresponding contour R1 to the gradation of each point in the mesh in the rectangular contour R2.

  FIG. 13 is a flowchart for explaining the operation of the vertical direction correction processing according to this embodiment. The CPU 17 obtains the length h1 in the stitch direction of the rectangular contour R2 (step S60). Specifically, the interval between the upper and lower corners on the right side or the left side of the contour R1 in the correction required region 30a (30b) after the trapezoid correction is set to the length h1. Next, the CPU 17 horizontally divides the contour R1 in the correction required area 30a (30b) after trapezoid correction, and sets the abscissa x (i) of each divided area U1 (step S62).

  Next, the CPU 17 sets i to 1 (step S64), and acquires the vertical length H1 at x (i) based on the contour R1 in the correction required region 30a (30b) (step S66). Next, the CPU 17 derives the Y coordinate value (yp ′) corresponding to the P ′ point at the boundary between the divided areas U1 using the equation (2), and converts the image of the P ′ point (step S68). ).

  The CPU 17 determines whether or not the conversion process has been completed at all points in the stitch direction in x (i) (step S70). If not completed (NO in step S70), the process returns to step S68, and the image conversion using equation (2) is repeated.

  On the other hand, if it is over (YES in step S70), the CPU 17 determines whether i has reached n-1 (step S72). If i is not n−1 (NO in step S72), the CPU 17 sets i = i + 1 (step S74), returns to step S66, and repeats the above-described processing. As a result, the vertical correction is performed on all the boundaries between the n-1 divided areas U1 in the correction required area 30a (30b). On the other hand, when i becomes n−1 (YES in step S72), the corrected correction required areas 30a and 30b and the correction unnecessary area 31 are combined (step S76), and the vertical direction correction process is terminated. . Thereafter, the CPU 17 returns to step S42 shown in FIG.

(Horizontal curvature correction processing)
FIG. 14 is an explanatory diagram showing the contour R1 and the image G1 after the trapezoid correction, and the rectangular contour R2 including the target points r1 ′, r2 ′, r3 ′, and r4 ′ at the four corners after the trapezoid correction. In the contour R1 and the image G1 after the trapezoidal correction, the curve in the binding direction remains, so it is necessary to perform a process for returning the curve W1 to the straight line L1, as shown in FIG. At this time, since the correction amount varies depending on the location in the horizontal direction, the horizontal correction is executed. Similarly to the vertical curvature correction, when the horizontal curvature correction is performed, a discontinuous surface is generated in the vicinity of the boundary line BL3. Therefore, the conversion formula used in the horizontal curvature correction is also weighted. .

  FIG. 15 is a flowchart for explaining the operation of the horizontal direction correction processing according to this embodiment. The CPU 17 obtains the length h in the binding direction of the rectangular contour R2 (step S80). Next, the CPU 17 divides the trapezoid-corrected contour R1 into n parts in the horizontal direction, and sets the abscissa x (i) of each divided region U1 (step S82). Next, the CPU 17 sets i to 1 (step S84), and estimates the deflection amount dz (i) at x (i) (step S86).

  16A and 16B are explanatory views showing the amount of deflection d (z) in the divided region U1, FIG. 16A is a cross-sectional view of an actual page, and FIG. 16B is a front view showing an image after contour correction. is there. As shown in FIG. 16A, the deflection amount dz (i) in the normal direction (Z direction) of the actual page is strictly speaking that the i-th divided region U1 and the (i-1) th divided region U1. And the difference in the Z direction.

However, this derivation requires a distance and an angle from the image pickup means (camera) to the paper surface, and measurement thereof is troublesome. Here, if the width of the divided area U1 is very small, the length H (i) of the i-th divided area U1 in the binding direction and the length H (i-1) of the i-1th divided area U1. Is approximately equal to the deflection amount dz (i). Using this relationship, the deflection amount dz (i) of the divided region U1 is estimated.
Specifically, the amount of deflection dz (i) is estimated by Expression (7).
dz (i) = g · {H (i) −H (i−1)} (7)

  Here, g is an arbitrary coefficient. By using this coefficient g, the image after the horizontal direction correction process can be adjusted in the horizontal direction. Here, it is tentatively derived with g = 1.

Next, the CPU 17 obtains the actual width dx ′ (i) (step S88).
dx (i) is the width of the i-th divided region U1 after trapezoidal correction.
dx (i) = x (i) -x (i-1) = {x (n) -x (0)} / n (8)

The actual width dx ′ (i) has been modified by the height z (i) of x (i). When z (i) is high, the width increases as H increases. That is, the actual width dx ′ (i) is small. This relationship is summarized in the following equation (9).
dx ′ (i) = h / H (i) · dx (i) (9)

Next, the CPU 17 calculates ds (i) (step S90).
ds (i) is the minute length of the actual page in the i-th divided region U1.

FIG. 17 is a conceptual diagram showing the relationship between ds (i), dz (i), and dx ′ (i).
As shown in FIG. 17, ds (i) is a square square based on the length dz (i) in the Z direction obtained in step S86 and the length dx ′ (i) in the X direction obtained in step S88. It is derived by the theorem.
ds (i) = {dz (i) ^ 2 + dx '(i) ^ 2} ^ 1/2 (10)

  Next, the CPU 17 determines whether i has reached the position of the boundary line BL3 (step S92). If i has not reached the position of the boundary line BL3 (NO in step S92), the CPU 17 sets i = i + 1 (step S94) and returns to step S86. As a result, dz (i), dx ′ (i), and ds (i) are obtained for all n divided regions U1.

  On the other hand, when i has reached the position of the boundary line BL3 (YES in step S92), the CPU 17 obtains the horizontal length S of the actual page (step S96). More specifically, the horizontal length S of the actual page is derived by calculating the sum of all ds (i) of the n divided areas U1. Next, the CPU 17 sets i to 1 again (step S98).

  Next, the CPU 17 obtains ds (i) of the divided area U1 from the 1st to the i-th, derives the accumulated value S (i) (step S100), and associates x (i) with x ′ (i). The image is converted (step S102).

The converted point of the i-th minute slit is x ′ (i) = i, the point before conversion is x (i), the length of the reference curve is S, and the length of the reference curve up to the i-th is S ( i) The horizontal length of the rectangular area is D, and S and S (i) from which the conversion formula is derived in advance,
x (i) = D * S (i) / S (11)
And

  FIG. 18 is a conceptual diagram for explaining the operation of the horizontal direction correction processing according to the present embodiment. In the case of the horizontal curvature correction, as shown in FIG. 18, in addition to giving the same weight as the vertical curvature correction, the i increment is counted from the correction required region side. That is, if the stitch is on the right side, the image is scanned from the right side, contrary to the general case. As a result, the count of i can be terminated at the boundary line BL3, and the processing time can be shortened.

The following is an example of a conversion formula in lateral curvature correction.
x (i) = (1−m) * D * S (i) / S + m * x ′ (i) (12)
m = (1-k * | x ′ (i) −xbound |) ^ − 1 (13)

  Here, xbound is the x coordinate of the boundary line BL3, and k is a constant where 0 <k. It is assumed that the value of k satisfies the condition of the above-described formula (6). Using this k, a transformation matrix is created from the transformation equation (12), and lateral curvature correction is executed. Thereby, it is possible to connect the correction required area 32 and the correction unnecessary area 33 without a sense of incongruity in the synthesis process.

  Next, the CPU 17 determines whether or not the conversion process has been completed for all points in the stitch direction in x (i) (step S104). If not completed (NO in step S104), the process proceeds to step S102. Returning, the image conversion in which x (i) is made to correspond to x ′ (i) is repeated according to the above-described equations (12) and (13).

  On the other hand, when the conversion process is completed (YES in step S104), the CPU 17 determines whether i has reached the position of the boundary line BL3 (step S106). If i has not reached the position of the boundary line BL3 (NO in step S106), i = i + 1 is set (step S108), the process returns to step S100, and the above-described processing is repeated. As a result, the horizontal correction is executed for all n divided areas U1 in the correction required area 32. On the other hand, if i has reached the position of the boundary line BL3 (YES in step S106), the corrected correction required area 32 and the correction unnecessary area 33 are combined (step S110), and the horizontal direction correction processing is ended. To do. Thereafter, the CPU 17 returns to step S46 shown in FIG.

  In the present embodiment, the coefficient g (provisionally g = 1) is used for Expression (6) when determining dz (i) in the lateral correction processing. Although the above-described processing makes the divided areas U1 uniform in the horizontal direction, there is a problem in the accuracy of the overall length of the width. For example, FIG. 19A shows a case where correction is performed on a page P1 having a low degree of curvature, and FIG. 19B shows that the same page P1 as in FIG. The case where correction is performed is shown.

  As is clear from FIGS. 19A and 19B, the corrected images Q1 and Q2 have different lengths in the horizontal direction. This is because even if the actual page width is the same, the difference between H (i) and H (i-1) is greatly different depending on the degree of curvature. In order to suppress this variation, processing can be performed in consideration of the coefficient g.

In other words, since the page aspect ratio (for example, the prescribed paper size) is known, the ratio between the horizontal width of the image synthesized in step S46 of FIG. 3 and the horizontal width obtained from the page aspect ratio is the coefficient g. Equivalent to. Therefore, an image without distortion at a desired aspect ratio can be obtained by scaling the image synthesized in step S46 in the horizontal direction corresponding to this ratio. Specifically, it is preferable that the corrected image is enlarged or reduced in the horizontal direction so as to have a predetermined size (for example, a prescribed paper size).
Note that the present invention is not limited to the above embodiment, and can be modified as appropriate.

  According to the present embodiment described above, the correction required areas 30a, 30b, and 32 that need to be corrected in a book image and the correction unnecessary areas 31 and 33 that do not need correction are divided into correction required areas 30a and 30b. , 32, the vertical direction correction and the horizontal direction correction are performed, so that an image obtained by imaging a curved paper surface can be corrected in a shorter time.

  Further, according to the present embodiment described above, when the vertical correction and the horizontal correction are performed on the correction necessary areas 30a, 30b, and 32, the degree of correction is compared in the vicinity of the boundary with the correction unnecessary areas 31 and 33. Therefore, the corrected necessary areas 30a, 30b, and 32 and the unnecessary correction areas 31, 33 can be combined without a sense of incongruity.

  In addition, according to the above-described embodiment, the boundary for dividing the correction required areas 30a, 30b, and 32 and the correction unnecessary areas 31 and 33 is set according to the user's instruction operation. The curvature correction according to the degree can be executed.

  In addition, according to the above-described embodiment, the correction required areas 30a, 30b, and 32 and the correction unnecessary areas 31 and 33 are set based on a predetermined division range, so that correction is required more easily. The areas 30a, 30b, and 32 and the correction unnecessary areas 31 and 33 can be divided.

7
Further, according to the above-described embodiment, the correction required areas 30a, 30b, and 32 and the correction unnecessary areas 31 and 33 are derived based on the division range of the reference image, so that correction is required more easily. The areas 30a, 30b, and 32 and the correction unnecessary areas 31 and 33 can be divided.

As mentioned above, although several embodiment of this invention was described, this invention is not limited to these, The invention described in the claim, and its equal range are included.
Below, the invention described in the claims of the present application is appended.

(Appendix 1)
The invention according to attachment 1 includes an extracting unit that extracts a contour of a page from an image, a contour correcting unit that corrects distortion of the image based on four corners of the contour extracted by the extracting unit, and the contour correction A region dividing unit that divides the image corrected by the unit into a correction necessary region and a correction unnecessary region; and the correction necessary region divided by the region dividing unit in a binding direction from an upper end to a lower end of the image. A dividing unit that divides the divided regions so that n divided regions are arranged in a lateral direction orthogonal to the binding direction, and a length in the binding direction for each of the divided regions divided by the dividing unit. A first correction unit that corrects the height, an estimation unit that estimates a deflection amount in a normal direction of the divided region in an actual page for each divided region corrected by the first correcting unit, and an estimation unit. Second correction means for correcting the lateral length of each of the divided areas based on the amount of deflection estimated in the above, and the n corrections corrected by the first correction means and the second correction means. An image correction apparatus comprising: an acquisition unit configured to combine a divided region and combine the combined region and the correction unnecessary region divided by the region dividing unit to obtain a combined image. is there.

(Appendix 2)
The invention according to attachment 2 is characterized in that the first correction means and the second correction means adjust the degree of correction in the vicinity of the boundary between the correction necessary area and the correction unnecessary area. It is an image correction apparatus of description.

(Appendix 3)
According to the third aspect of the present invention, the region dividing unit divides the image into three regions of an upper end portion, a central portion, and a lower end portion in the vertical direction, and the regions of the upper end portion and the lower end portion are divided into the regions. The image correction apparatus according to appendix 1 or 2, wherein the correction required area is used and the central portion is the correction unnecessary area.

(Appendix 4)
In the invention according to appendix 4, the region dividing unit divides the image into the vicinity of the binding portion of the page and the other two regions in the horizontal direction, and sets the region near the binding portion as the correction necessary region, The image correction apparatus according to any one of appendices 1 to 3, wherein the other area is set as the correction unnecessary area.

(Appendix 5)
The invention according to appendix 5 further includes designation means for designating the correction required area and the correction unnecessary area divided by the area dividing means. It is an image correction apparatus of description.

(Appendix 6)
The invention according to attachment 6 further includes setting means for setting the correction required area and the correction unnecessary area divided by the area dividing means based on a predetermined division range. The image correction apparatus according to any one of appendices 1 to 4.

(Appendix 7)
The invention according to appendix 7 is characterized in that the correction required area and the correction unnecessary area divided by the area dividing means are derived based on a division range of a reference image. It is an image correction apparatus as described in any one.

(Appendix 8)
The invention according to appendix 8 includes a step of extracting a contour of a page from an image, a step of correcting distortion of the image based on four corners of the extracted contour, and correcting an image in which the distortion is corrected A step of dividing into a necessary area and a correction unnecessary area; and a horizontal direction in which the divided area extending from the upper end to the lower end of the image along the stitch direction is orthogonal to the stitch direction. The n-divided area, the step of correcting the length in the binding direction for each of the divided areas divided into the n, and the length in the binding direction are corrected. For each of the divided regions, estimating a normal direction deflection amount of the divided region on an actual page, and correcting the lateral length of each divided region based on the estimated deflection amount And the corrected With synthesizing n number of segmented regions, an image correction method characterized by comprising the steps of synthesizing the said combined area and the divided correction unnecessary area to obtain a composite image.

(Appendix 9)
According to the ninth aspect of the present invention, the computer that controls the image correction apparatus corrects the distortion of the image based on the extraction function for extracting the outline of the page from the image and the four corners of the outline extracted by the extraction function. An area correction function that divides the image corrected by the outline correction function, the image that has been corrected by the outline correction function into a correction required area and a correction unnecessary area, and the correction required area that is divided by the area dividing function from the upper end of the image A division function that divides the division area extending along the stitch direction to the lower end so that n pieces are arranged in a lateral direction orthogonal to the stitch direction, and for each of the division areas divided by the division function, A first correction function for correcting the length in the binding direction, and for each of the divided areas corrected by the first correction function, an amount of deflection in the normal direction of the divided area on an actual page is estimated. The correction function is corrected by the second correction function, the first correction function, and the second correction function that correct the length in the horizontal direction for each of the divided regions based on the amount of deflection estimated by the estimation function. An acquisition function of combining the n divided regions and combining the combined region and the correction unnecessary region divided by the region dividing function to acquire a composite image is realized. It is a program.

DESCRIPTION OF SYMBOLS 1 ... Portable information terminal, 10 ... Communication part, 11 ... Imaging part, 12 ... ROM, 13 ... RAM, 14 ... Display part, 15 ... Operation part (touch panel), 16 ... Recording medium, 17 ... CPU, 30a, 30b, 32: Correction required area 31, 33: Correction unnecessary area

Claims (9)

Extraction means for extracting the outline of the page from the image;
Contour correcting means for correcting distortion of the image based on the four corners of the contour extracted by the extracting means;
Area dividing means for dividing the image corrected by the contour correcting means into a correction required area and a correction unnecessary area;
The correction required areas divided by the area dividing means are arranged so that n divided areas extending along the stitch direction from the upper end to the lower end of the image are arranged in the horizontal direction perpendicular to the stitch direction. A dividing means for dividing;
First correcting means for correcting the length in the binding direction for each of the divided areas divided by the dividing means;
Estimating means for estimating the amount of deflection in the normal direction of the divided area on an actual page for each of the divided areas corrected by the first correcting means;
Second correction means for correcting the lateral length of each of the divided regions based on the amount of deflection estimated by the estimation means;
The n number of divided regions corrected by the first correcting unit and the second correcting unit are combined, and the combined region and the correction unnecessary region divided by the region dividing unit are combined and combined. An acquisition means for acquiring an image;
An image correction apparatus comprising: The first correction unit and the second correction unit are:
Adjusting the degree of correction in the vicinity of the boundary between the correction necessary region and the correction unnecessary region;
The image correction apparatus according to claim 1. The region dividing means includes
The image is vertically divided into three regions of an upper end portion, a central portion, and a lower end portion, the upper end portion and the lower end portion are defined as the correction necessary region, and the central portion is defined as the correction unnecessary region. To
The image correction apparatus according to claim 1, wherein the image correction apparatus is an image correction apparatus. The region dividing means includes
The image is divided in the horizontal direction into the vicinity of the binding portion of the page and the other two regions, the region near the binding portion is set as the correction necessary region, and the other region is set as the correction unnecessary region.
The image correction apparatus according to claim 1, wherein the image correction apparatus is an image correction apparatus. Further comprising designation means for designating the correction necessary area and the correction unnecessary area divided by the area dividing means;
The image correction apparatus according to claim 1, wherein the image correction apparatus is an image correction apparatus. Further comprising setting means for setting the correction required area and the correction unnecessary area divided by the area dividing means based on a predetermined division range.
The image correction apparatus according to claim 1, wherein the image correction apparatus is an image correction apparatus. Deriving the correction required area and the correction unnecessary area divided by the area dividing means based on a division range of a reference image;
The image correction apparatus according to claim 1, wherein the image correction apparatus is an image correction apparatus. Extracting a page outline from within the image;
Correcting distortion of the image based on the four corners of the extracted contour;
Dividing the distortion-corrected image into a correction-necessary region and a correction-unnecessary region;
Dividing the divided correction required areas so that n divided areas extending along the stitch direction from the upper end to the lower end of the image are arranged in a lateral direction perpendicular to the stitch direction;
Correcting the length in the binding direction for each of the divided areas divided into n pieces;
Estimating the amount of deflection in the normal direction of the divided area on an actual page for each of the divided areas in which the length in the binding direction is corrected;
Correcting the lateral length of each of the divided regions based on the estimated amount of deflection;
Combining the corrected n divided regions and combining the combined region and the divided correction unnecessary region to obtain a combined image;
An image correction method comprising: In the computer that controls the image correction device,
An extraction function that extracts the outline of the page from the image,
A contour correction function for correcting distortion of the image based on the four corners of the contour extracted by the extraction function;
A region dividing function for dividing the image corrected by the contour correcting function into a correction required region and a correction unnecessary region;
The correction required areas divided by the area dividing function are arranged so that n divided areas extending along the stitch direction from the upper end to the lower end of the image are arranged in the horizontal direction perpendicular to the stitch direction. Split function to split,
A first correction function for correcting the length in the binding direction for each of the divided areas divided by the division function;
An estimation function for estimating the amount of deflection in the normal direction of the divided area on an actual page for each of the divided areas corrected by the first correction function;
A second correction function for correcting the lateral length of each of the divided regions based on the amount of deflection estimated by the estimation function;
The n regions divided by the first correction function and the second correction function are combined, and the combined region and the correction unnecessary region divided by the region dividing function are combined and combined. Acquisition function to acquire images,
A program characterized by realizing.

Images