JP2016149097A - Image processing device, image processing method, program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an image file in which distortion correction is properly performed.SOLUTION: A portable terminal (10) includes a formatting processing part (101) for generating an image file obtained by formatting image data of a photographed image in a prescribed format, and a drawing command generation part (100) for creating a drawing command to control a computer so as to display the photographed image of the image file after performing distortion correction when displaying the photographed image, and adding the drawing command to the image file. The drawing command generation part (100) acquires first difference information showing difference between in the case of performing distortion correction of a correction object range (60) of the photographed image in projective transformation and in the case of performing the distortion correction in affine transformation, and divides the correction object range 60 to set a plurality of partial areas to create the drawing command to perform the distortion correction by a matrix for affine transformation determined from the partial areas in each partial area in the case that the difference is not within an allowable range.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing apparatus that generates an image file obtained by formatting image data obtained by photographing into data of a predetermined type.

  With the spread of digital cameras, the number of users who record content recorded on bulletin boards or blackboards by taking pictures with digital cameras is increasing. However, in such a method of use, there are many cases where a subject to be photographed cannot be photographed from the front, and a distorted photographed image may be acquired. On the other hand, as shown in Patent Document 1, for example, many techniques for correcting (distortion correction) a distorted captured image are conventionally known.

JP 2005-115711 A JP 2010-161764 A

  By the way, image data of a photographed image obtained by photographing with a digital camera is usually converted into an image file (for example, PDF file) formatted into a predetermined type of data and stored. Here, since the performance of a computer is limited in a digital camera or a mobile terminal equipped with a digital camera, if an image file is generated from image data of a captured image that has been subjected to distortion correction, it takes time to generate the image file. For example, when there are many processing objects, it becomes a problem.

  On the other hand, in Patent Document 2, when displaying image data before image processing and an image of the image data, the computer is controlled to perform the display in a state where the image processing is performed on the image data. A technique for generating an image file in which a drawing command is embedded has been proposed. By using this technology, it is possible to quickly generate an image file even on a portable terminal or the like with limited computer performance.

  Therefore, an image in which image data before distortion correction and a drawing command for controlling the computer to perform the display in a state where the image data is subjected to distortion correction when the image of the image data is displayed are embedded. By generating the file, it is considered possible to shorten the time until the generation of the image file.

  Here, as a distortion correction method, projective transformation is known as disclosed in Patent Document 1. However, in an image file, all coordinate transformations that fall within the category of projective transformation cannot be used. Among transformations, there is a transformation that can use only coordinate transformation called affine transformation (a kind of projective transformation) (for example, PDF (Portable Document Format) file). In such an image file, you can embed a drawing command that instructs affine transformation, but you cannot embed a drawing command that instructs projection transformation other than affine transformation, so correct distortion that is difficult to deal with by affine transformation. The problem that is difficult arises.

  The present invention has been made in view of the above-described problem, and in an image processing apparatus that generates an image file adapted to perform distortion correction by affine transformation when an image is displayed, distortion correction is appropriately performed. The purpose is to generate an image file to be performed.

  In order to solve the above problems, an image processing apparatus according to an aspect of the present invention includes a formatting processing unit that generates an image file in which image data of a captured image is formatted in a predetermined format, and the captured image of the image file. A command addition unit for creating a drawing command for controlling the computer to perform the display after performing distortion correction when displaying the image, and adding the drawing command to the image file. Determining a first difference information indicating a difference between a case where a correction target range is set for the photographed image and the correction target range is subjected to distortion correction by projective transformation and a case where the distortion correction is performed by affine transformation; When the difference is within an allowable range, the drawing command for performing the distortion correction is created by an affine transformation matrix determined from the correction target range, and the difference is allowable When not within the range, the correction target range is divided to set a plurality of partial areas, and the drawing command for performing the distortion correction is created for each partial area by an affine transformation matrix determined from the partial areas. To do.

  According to one aspect of the present invention, when distortion correction is performed on a correction target range using a single affine transformation matrix, a correction error (in contrast to a case where correction is performed by projective transformation without being limited to affine transformation). When the projection transformation is limited to the affine transformation, the correction target range is divided into a plurality of partial areas, and an affine transformation matrix is set for each partial area. Therefore, it is possible to realize distortion correction with a correction error suppressed (by dividing the corresponding range in one distortion correction matrix to reduce the range, the corresponding range in one distortion correction matrix) The correction error can be reduced and the correction error can be suppressed as a whole). Therefore, when generating an image file in which distortion correction is performed by affine transformation when an image is displayed, an image file in which distortion correction is appropriately performed can be generated.

It is a figure for demonstrating projective transformation. It is a functional block diagram which shows the structure of the portable terminal of this embodiment. It is a figure which shows the picked-up image before distortion correction. FIG. 4 is a diagram illustrating an image obtained by performing projective transformation on the captured image of FIG. 3 without being limited to affine transformation. It is the figure which showed the parallelogram set using the left side and lower side of the correction object range set to the picked-up image of FIG. It is the figure which showed the image which correct | amended the picked-up image of FIG. 3 by the affine transformation which transforms the parallelogram shown in FIG. 5A into another parallelogram. It is the figure which showed the parallelogram set using the right side and upper side of the correction | amendment target range set to the picked-up image of FIG. It is the figure which showed the image which correct | amended the picked-up image of FIG. 3 by the affine transformation which transforms the parallelogram shown in FIG. 6A into another parallelogram. It is a flowchart which shows the flow of a process of the file generation mode of this embodiment. It is the schematic diagram which showed the external appearance structure of the portable terminal of this embodiment. It is the schematic diagram which showed the projective transformation (deformation 1) from the square PQRS to the square TUUV. FIG. 9B is an enlarged view showing the vicinity of the center of the square TUUV in FIG. 9A. It is explanatory drawing which showed four division ranges obtained by dividing | segmenting a correction | amendment object range, and four post-correction division | segmentation ranges obtained by dividing | segmenting a display range after correction. It is explanatory drawing which showed the division range obtained by further dividing one division range of the four division ranges shown by FIG. It is explanatory drawing which showed two partial areas produced | generated from the division | segmentation range, and two post-correction partial areas produced | generated from the division | segmentation range after a correction | amendment. It is explanatory drawing which showed eight partial area | regions obtained from the correction target range, and eight post-correction partial areas obtained from the display area after correction. FIG. 8 is a schematic diagram illustrating a drawing command designated in S107 of FIG. It is the schematic diagram which showed the drawing command embedded in the PDF file obtained via S108-S118. FIG. 10 is a schematic diagram illustrating a drawing command that is a drawing command according to the fourth embodiment and is embedded in a PDF file obtained through S108 to S118. It is the figure which showed the display malfunction which arises in the boundary of the corrected partial area | regions adjacent to each other. It is an image displayed based on the mask area | region set by the process of Embodiment 4 of this invention, and is a figure which shows that the display malfunction of FIG. 17A is eliminated. It is explanatory drawing which showed the division | segmentation process of Embodiment 2 of this invention. It is a figure which shows the output image which carried out distortion correction by projective transformation, and the output image which was divided | segmented into eight partial area | regions and was distortion-corrected by affine transformation.

[Description of prerequisite technology]
Before describing the embodiment of the present invention, projection transformation (coordinate transformation), which is a prerequisite technology of the embodiment of the present invention, will be described first.

  Normally, the coordinate value of the two-dimensional coordinate is expressed as (x, y), but the homogeneous coordinate is expressed as (wx, wy, w) using a real number w (where w ≠ 0). For example, when expressing a two-dimensional coordinate with (x, y) = (2, 3) as a homogeneous coordinate, it is usually expressed as (2, 3, 1) with w = 1, but (4, 6, 2), (6, 9, 3), etc., can be expressed in different forms. (2, 3, 1), (4, 6, 2), (6, 9, 3) are all the same position (point) ). When this homogeneous coordinate is used, the two-dimensional projective transformation is expressed as a 3 × 3 matrix (hereinafter simply referred to as “matrix”) as shown in Equation 1.

  In Equation 1, (x, y, 1) represents coordinate values before conversion, (x ′, y ′, 1) represents coordinate values after conversion, and “˜” are the same in homogeneous coordinates. It is a symbol indicating a position (point).

  In the matrix operation on the left side of Equation 1, when the above-described w is obtained, w = xg + yh + 1. Since this w does not always become 1, by multiplying the coordinate value after conversion by w (replaces the right side of Equation 1 with (wx ′, wy ′, w)), the left side and the right side match. , Equation 2 can be obtained.

  The two-dimensional projective transformation corresponds to a transformation from an arbitrary quadrangle to another quadrangle (as an example, a transformation from a trapezoid to a rectangle is possible). Of the projective transformations, the transformation with g = 0 and h = 0 in the matrix of Equation 1 corresponds to the affine transformation. The affine transformation is a transformation that transforms an arbitrary parallelogram on the plane into another parallelogram, but cannot cope with other transformations. That is, the affine transformation is a kind of coordinate transformation among various coordinate transformations included in the category of projective transformation.

  Next, a method of obtaining matrix coefficients for realizing Modification 1 (deformation from any square to another square) shown in FIG. 1 will be described.

Modification 1 can be realized by sequentially applying some basic projective transformations (deformation 2 to modification 5). Specifically, as illustrated in FIG. 1, the deformation 1 that transforms the quadrangular PQRS into the quadrangular TUWW is the same conversion as that in which the modifications 2 to 5 are sequentially applied.
Deformation 1: Deformation of quadrangular PQRS to quadrilateral TUUV 2: Deformation of translating quadrilateral PQRS so that S coincides with the origin Deformation 3: Deformation of quadrilateral PQRS with S coincident with origin into unit square Deformation 4: Deformation of unit square Is transformed into a square TUWW where W coincides with the origin. Deformation 5 where the translation of the square TUWW where W coincides with the origin is translated.

  Next, modifications 3 and 4 will be described. The deformations 3 and 4 correspond to conversion between an arbitrary square whose unit vertex coincides with the origin and a unit square.

  First, a procedure for obtaining a matrix for performing modification 4 will be described. The deformation 4 is a deformation from a unit square to a quadrangle TUVW in which W coincides with the origin. That is, in the modification 4, the four vertices (0, 1, 1), (1, 1, 1), (1, 0, 1), (0, 0, 1) constituting the unit square are one vertex. Corresponds to conversion (movement) to four vertices (x1, y1, 1), (x2, y2, 1), (x3, y3, 1), (0, 0, 1) of an arbitrary rectangle whose origin is To do. Therefore, by substituting the coordinate values of the four vertices of the unit square as the coordinate values of the four points before the conversion and substituting the coordinate values of the four vertices of the arbitrary quadrangle as the coordinate values of the four points after the conversion, Equation 4 is obtained.

  By solving the equations listed in Equation 4 as simultaneous equations, Equation 5 can be obtained. By calculating Expression 5, the coefficients a to h of the matrix of the modification 4 can be obtained, and thereby the matrix of the modification 4 can be obtained.

  In Equation 5, the condition that the denominator of the equation for calculating the coefficient g is 0 is that the three points after the movement of the three points other than the origin of the unit square are aligned on a straight line. It deviates from the condition of “transformation from a square to an arbitrary quadrilateral whose one vertex is the origin”. In addition, there is an expression A or an expression B as an expression for calculating h. In consideration of the calculation accuracy, the expression A is used when | x1-x2 |> | y1-y2 | It is preferable to use B.

  The matrix of modification 3 is obtained as described below. The deformation 3 is a deformation from the quadrangle PQRS in which S coincides with the origin to a unit square. That is, the above-described modification 4 corresponds to a transformation from a unit square to an arbitrary quadrangle whose one vertex is located at the origin, whereas transformation 3 is a transformation from an arbitrary quadrangle whose one vertex is located at the origin to the unit square. It corresponds to the deformation. Therefore, with respect to the deformation 3, (1) First, by the same method as the method for obtaining the matrix of the deformation 4, the reverse deformation of the deformation 3 (from the unit square to the square PQRS in which S coincides with the origin) is performed. What is necessary is just to obtain | require the matrix for performing, and obtain | require the inverse matrix of the matrix for performing (2) deformation | transformation contrary to modification 3. This inverse matrix corresponds to the matrix of modification 3. In this way, the matrix of the modification 3 can be obtained using the equations 2, 4, and 5 in the same manner as the method for obtaining the modification 4.

  The matrix of modification 1 can be obtained by sequentially multiplying the matrices of modification 2 to modification 5 obtained as described above. That is, if the coordinate value of each vertex of the quadrilateral PQRS before deformation and the coordinate value of each vertex of the post-deformation (target) quadrangle TUWW are determined, the quadrangle PQRS is transformed into the quadrangle TUVW using the coordinate values of these vertices. It is possible to obtain the matrix of modification 1 for the purpose. Specifically, it is as follows.

  First, a matrix of deformation 2 is obtained by calculating Equation 3 using the coordinate value of the vertex S and the coordinate value of the origin of the quadrangle PQRS. Similarly, a matrix of deformation 5 is obtained by calculating Equation 3 using the coordinate value of the vertex W and the coordinate value of the origin of the quadrangle TUUV.

  Subsequently, the coordinate value of each vertex of the quadrangle PQRS where the vertex S is located at the origin is obtained using the coordinate value of each vertex of the quadrangle PQRS and the example of the second modification. Then, using the coordinate value of each vertex of the quadrangle PQRS where the vertex S is located at the origin and the coordinate value of each vertex constituting the unit square, the matrix of modification 3 is obtained by calculating Equation 5.

  Also, using the coordinate value of each vertex of the quadrangle TUUV and the matrix of deformation 5, the coordinate value of each vertex of the quadrangle TUWW where the vertex W is located at the origin is obtained. Then, using the coordinate value of each vertex of the quadrangle TUWW where the vertex W is located at the origin and the coordinate value of each vertex constituting the unit square, the matrix of the modification 4 is obtained by calculating Equation 5.

  Finally, the matrix of modification 1 (that is, coefficients a to h of the matrix of modification 1) can be obtained by multiplying the matrices of modification 1 to modification 4 obtained as described above.

  In the present specification, the matrix of modification 1 indicating the projective transformation is referred to as a distortion correction matrix. The coordinate transformation of deformation 1 is projective transformation. However, when the quadrangle PQRS before deformation and the quadrangle TUUV after deformation are parallelograms, the coefficients g and h of the matrix of deformation 1 are both 0, It means that the deformation 1 is an affine transformation (a kind of projective transformation).

Embodiment 1
(Outline of the embodiment)
When a subject to be photographed (for example, a rectangular paper affixed to a bulletin board) is photographed obliquely with a digital camera or a portable terminal (for example, a smartphone) equipped with a digital camera, a distorted photographed image (the subject to be photographed is distorted). Captured image) may be obtained (for example, FIG. 3).

  When converting image data of such a captured image into a PDF file (that is, when embedding the image data of the captured image in a PDF file), when displaying the image data of the captured image before distortion correction and the captured image If a PDF file in which a drawing command for controlling the computer to perform the display is embedded after performing distortion correction on the image data is created, it is not necessary to perform distortion correction when creating the PDF file. Processing time can be reduced.

  However, PDF files can only perform affine transformations among projective transformations. Therefore, even if the distortion can be corrected by projective transformation that is not limited to affine transformation, there is a problem that the PDF file may not be able to properly correct the distortion when it cannot be corrected by affine transformation.

  FIG. 3 is a diagram illustrating a captured image before distortion correction. The photographed image in FIG. 3 is an image obtained by photographing the paper pasted on the bulletin board. A square 200 shown in FIG. 3 is a correction target range set by the user, and is a portion where the paper is shown. In the captured image shown in FIG. 3, it can be seen that the image (paper) shown in the correction target range is strongly distorted.

  FIG. 4 is a diagram showing an image obtained by performing projective transformation on the range (correction target range) of the rectangle 200 in FIG. 3 without being limited to affine transformation. The rectangle 200a in FIG. 4 corresponds to the rectangle after projective transformation of the rectangle 200 in FIG. Comparing FIG. 3 and FIG. 4, it can be seen that if the projection transformation is performed without being limited to the affine transformation, the distortion is appropriately corrected even if the distortion is strong.

  FIG. 5B is a diagram illustrating an image obtained by performing affine transformation on the range (correction target range) of the rectangle 200 in FIG. 3. Comparing FIG. 3, FIG. 4, and FIG. 5B, it can be seen that there are cases where distortion that can be corrected appropriately by projective transformation cannot be corrected by affine transformation.

  Note that the distortion correction matrix of the affine transformation is a matrix for transforming the parallelogram into another parallelogram, but the quadrangle 200 that is the correction target range in FIG. 3 is not a parallelogram. From the coordinate values of the four vertices of the rectangle 200, a distortion correction matrix for affine transformation cannot be obtained. Therefore, as shown in FIG. 5A, a parallelogram 200b (thick line portion) is set which includes a left side 201 of the quadrangle 200, a lower side 202 of the quadrangle 200, a right side parallel to the left side 201, and an upper side parallel to the lower side 202. Then, the affine transformation distortion correction matrix is obtained using the coordinate values of the vertices of the parallelogram 200b as the pre-conversion coordinate values. Here, the parallelogram 200b ′ shown in FIG. 5B corresponds to the parallelogram after the parallelogram 200b of FIG. 5A is deformed by affine transformation. An image obtained by performing affine transformation on the correction target range surrounded by the rectangle 200 in FIG. 3 is shown inside the parallelogram 200b ′ in FIG. 5B. In this way, even when a correction target range that is not a parallelogram is set as in the quadrangle 200 in FIG. 3, an image in which the correction target range is subjected to affine transformation can be obtained.

  5A and 5B, the parallelogram 200b is set using the left side 201 and the lower side 202 of the rectangle 200 of the correction target range. However, any one of the vertices of the rectangle 200 of the correction target range is used. A parallelogram can be set using the first side and the second side extending from the vertex. In other words, as described above, the first side and the second side may be the left side 201 and the lower side 202, the left side 201 and the upper side 203, or the right side 204 and the upper side 203. It may be the right side 204 and the lower side 202 (see FIG. 3).

  For example, when a parallelogram is set using the right side 204 and the upper side 203 in FIG. 3, processing is performed as shown in FIGS. 6A and 6B. That is, as shown in FIG. 6A, a parallelogram 200c is set, which includes a right side 204 of a quadrangle 200, an upper side 203 of the quadrangle 200, a left side parallel to the right side 204, and a lower side parallel to the upper side 203. A distortion correction matrix for affine transformation is obtained using the coordinate value of each vertex of the shape 200c as the coordinate value before conversion. Here, the parallelogram 200c ′ (thick line portion) shown in FIG. 6B corresponds to the parallelogram after the parallelogram 200c of FIG. 6A is deformed by affine transformation. Then, an image obtained by performing affine transformation on the correction target range surrounded by the rectangle 200 in FIG. 3 is shown inside and around the parallelogram 200c ′ in FIG. 6B. Here, according to FIG. 6B, although the distortion in the upper right part is corrected, a new distortion is generated in the lower left part, and as a result, it is understood that the distortion is not completely corrected.

  As described above, there are cases in which distortion that can be corrected by projection transformation that is not limited to affine transformation cannot be corrected by affine transformation, so only affine transformation can be specified with drawing commands. The problem with PDF files is that distortion may not be corrected properly.

  Therefore, in order to avoid this problem, the mobile terminal according to the present embodiment first sets a correction target range for the captured image and obtains first error information. The first error information is a value indicating an error (correction error) when distortion correction is performed by limiting the projection transformation to affine transformation with respect to the case where the correction target range is distortion-corrected by projective transformation (in other words, The first error information is also a value (first difference information) indicating a difference between the case where the correction target range is subjected to distortion correction by projective transformation and the case where distortion is corrected by affine transformation. Details of the first error information will be described later.

  After obtaining the first error information, the mobile terminal according to the present embodiment performs threshold processing on the first error information to determine whether or not the error of the first error information is within an allowable range.

  When determining that the error of the first error information is within an allowable range, the mobile terminal according to the present embodiment performs a affine transformation using a distortion correction matrix set from the correction target range, and displays a captured image. A drawing command for controlling the image is generated, and the drawing command is embedded in the PDF file together with the captured image.

  On the other hand, when it is determined that the error of the first error information is not within the allowable range, the mobile terminal according to the present embodiment generates a plurality of partial areas by dividing the correction target range, and is set from the partial areas. A drawing command for controlling the computer to display a captured image after performing affine transformation using the distortion correction matrix is generated for each partial region, and each generated drawing command is embedded in the PDF file together with the captured image.

  As a result, when affine transformation is performed on a correction target range using a single distortion correction matrix, a correction error increases (that is, when distortion correction cannot be performed properly), a plurality of correction ranges are provided. It is possible to realize distortion correction with reduced correction error by dividing the area into partial areas and setting a distortion correction matrix for each partial area (dividing the corresponding range with one distortion correction matrix) Thus, the correction error in the corresponding range can be reduced with one distortion correction matrix, and the correction error can be suppressed as a whole).

  Hereinafter, the portable terminal of this embodiment will be described in detail.

(Overall configuration of mobile device)
A configuration of a portable terminal will be described in detail as an embodiment of the image processing apparatus of the present invention. FIG. 2 is a block diagram illustrating a schematic configuration of the mobile terminal according to the present embodiment.

  A portable terminal 10 shown in FIG. 2 is a portable computer having a camera function for photographing a photographing object and generating a photographed image and a display function for displaying an image, and examples thereof include a smartphone and a tablet. As shown in FIG. 2, the mobile terminal 10 includes at least an information processing unit 11, a touch panel 12, a photographing unit 13, and a storage unit 14.

  Further, although not shown, the mobile terminal 10 includes various types of hardware that are normally provided in the mobile terminal, such as a communication interface, an Internet transmission / reception unit, a voice operation unit, a voice output unit, a television broadcast receiver, and a GPS (Global Positioning System). Wear.

  The information processing unit 11 is a block that extracts various types of information and programs (control programs and applications for performing various types of control) stored in the storage unit 14 and performs various types of information processing. For example, a CPU (Central Processing Unit) ). That is, the information processing unit 11 is a block that performs various information processing in the mobile terminal 10 such as execution of various application programs, operation control (including display control) of various hardware included in the mobile terminal 10, and image processing. In the present embodiment, in particular, the information processing unit 11 executes a file generation mode for converting image data of a captured image obtained by the imaging unit 13 into a PDF file. This point will be described in detail.

  The storage unit 14 receives various application programs executed by the information processing unit 11, an OS (Operating System) program, a control program, various data (setting values, tables, etc.) read when the programs are executed, and acquired by the imaging unit 13. This is a storage area for storing image data of captured images. As the storage unit 14, various conventionally known storage means such as read-only memory, random access memory, flash memory, EPROM (Erasable Programmable ROM), EEPROM (registered trademark) (Electrically EPROM) and the like can be used. . The storage unit 14 includes a working memory that temporarily stores data being processed by the information processing unit 11.

  The touch panel 12 is an apparatus in which a display unit 12a that displays an image and an input unit 12b that generates an input signal corresponding to a user's touch operation are integrated. As the touch panel 12, a known one can be used. That is, the display unit 12a is configured by an LCD (Liquid Crystal Display) or the like, and the input unit 12b is configured by a capacitance sensor or the like integrated with the display unit 12a. When the user performs a touch operation on the touch panel 12, an input instruction corresponding to the touch operation is transmitted to the information processing unit 11, and the information processing unit 11 performs display control of the touch panel 12 according to the input instruction. Become. For example, when a tap operation is performed, an image corresponding to the tapped icon or file is displayed.

  In addition, since the portable terminal 10 of this embodiment is what is called a smart phone and a tablet, although the touch panel 12 is provided as an image display means, an image display means is not restricted to the touch panel 12. FIG. That is, in the mobile terminal 10, the display unit and the input unit may be configured separately. In this case, the display unit is an LCD or the like, and the input unit is a keyboard or the like.

  The photographing unit 13 is a digital camera made up of a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). The photographing unit 13 performs a photographing process according to a photographing instruction signal input by the user, and generates image data indicating a photographed image. Image data generated by the imaging unit 13 is subjected to predetermined processing such as A / D (analog to digital) conversion and shading correction, and is temporarily stored in the storage unit 14.

(File generation mode)
Next, the file generation mode will be described in detail. The file generation mode is an image file in which the user designates a photographed image to be processed from among photographed images stored in the storage unit 14, and the image data of the designated photographed image is formatted in a predetermined format. It is a mode to create and save. In the present embodiment, the image file is a PDF (Portable Document Format) file.

  As illustrated in FIG. 2, the information processing unit 11 includes at least a drawing command generation unit 100 that executes a file generation mode and a formatting processing unit 101. In FIG. 2, the information processing unit 11, the touch panel 12, the photographing unit 13, and the storage unit 14 are hardware, but each unit included in the information processing unit 11 (the drawing command generation unit 100 and the formatting processing unit 101). These are functional blocks indicating each function of software (application) executed by the information processing unit 11.

  The formatting processing unit 101 is a block that creates a PDF file in which image data of a captured image to be processed designated by the user is embedded.

  When displaying an image of the PDF file, the drawing command generation unit (command addition unit) 100 generates a drawing command for controlling the computer to display the image after correcting the distortion of the image. This block adds (embeds) the drawing command to the PDF file generated by the formatting processing unit 101.

  Next, the flow of processing in the file generation mode will be described below using the flowchart of FIG.

  When the user operates the touch panel 12 of the mobile terminal 10 and inputs an instruction to shift to the file generation mode, the formatting processing unit 101 shifts the mobile terminal 10 to the file creation mode (S101).

  After S101, the formatting processing unit 101 selects a captured image designated by the user as a processing target from among the captured images stored in the storage unit 14 (S102). The selection of the photographed image is performed as follows, for example. In response to a user instruction, the formatting processing unit 101 sequentially displays the captured image previews stored in the storage unit 14 on the touch panel 12, and selects the captured image of the preview designated by the user as a processing target.

  When a captured image to be processed is selected in S102 of FIG. 7, the drawing command generation unit 100 sets a rectangular correction target range in the captured image to be processed and sets a corrected display range (S103). .

  The correction target range is a range surrounding a distorted shooting target in the shot image, and is set by the user. The correction target range is set as follows, for example. The drawing command generation unit 100 displays the captured image 50 on the touch panel 12 of the mobile terminal 10 as shown in FIG. The user designates four points to be the vertices of the correction target range among the captured images 50 displayed on the touch panel 12 by a touch operation. Then, as illustrated in FIG. 8, the drawing command generation unit 100 sets, as the correction target range 60, a quadrangle having four vertices designated by the user's touch operation.

  Next, the corrected display range in S103 will be described. The corrected display range indicates (1) the target shape (target range) of the corrected correction target range 60 when distortion correction is performed on the correction target range 60, and (2) a PDF file. It is also the range displayed when opening. In this embodiment, since a PDF file is used, the post-correction display range is set from a standard paper size (A3, A4, etc.). Here, since the shape of the standard paper size is a rectangle (parallelogram), the display range after correction set in this way can be a target shape after correction by affine transformation.

  The corrected display range is set as follows, for example. As illustrated in FIG. 8, the drawing command generation unit 100 causes the touch panel 12 to display a list box 471 for allowing the user to select a corrected size as the corrected display range. The user selects a desired paper size from among the paper sizes shown in the list box 471, and the drawing command generation unit 100 sets the paper size selected by the user as the corrected display range. When the user does not select a desired paper size, the drawing command generation unit 100 sets a default paper size.

  For example, as shown in FIG. 10, a rectangular corrected display range 70 is set for the correction target range 60.

  After S103 in FIG. 7, the drawing command generation unit 100 refers to the correction target range 60 and the corrected display range 70 set in S103 and obtains first error information (first difference information) for the correction target range 60. Calculate (S104). The first error information will be described in detail below.

  The first error information indicates an error (correction error) when distortion correction is performed by limiting the projection transformation to the affine transformation in contrast to the case where the correction target range is distortion-corrected by the projection transformation without being limited to the affine transformation. (In other words, the first error information can be said to be a value indicating a difference between a case where the correction target range is subjected to distortion correction by projective transformation and a case where distortion is corrected by affine transformation). In S104, the correction target range 60 shown in FIG. 8 is corrected by projective transformation (not limited to affine transformation) that transforms into the corrected display range 70 (see FIG. 10), and distortion correction is performed by affine transformation. A value indicating the difference between the two is obtained. Hereinafter, the first error information will be described in more detail.

  FIG. 9A is a schematic diagram showing a case where the projective transformation (deformation 1) is performed from the quadrangle PQRS to the quadrangle TUWW, with the quadrangle PQRS as the correction target range and the quadrangle TUWW as the corrected display range, and FIG. It is the figure which expanded and showed the center vicinity of the square TUUV.

  The point Z1 in FIG. 9A is the center of gravity (physical center of gravity) of the quadrangle PQRS that is the correction target range, and the midpoints of these sides are set between the side PQ and the side SR that are in the opposite side relationship in the quadrangle PQRS. This corresponds to the intersection of the connected straight line H1 and the straight line H2 connecting the midpoints of these sides between the side Ps and the side QR of the quadrangle PQRS.

  A point Z3 in FIG. 9A is a position after the conversion of the point Z1 when performing the projective conversion from the quadrangle PQRS to the quadrangle TUWW.

  The point Z2 in FIG. 9A is the center of gravity (physical center of gravity) of the quadrangle TUWW, which is the corrected display range, and the midpoints of these sides between the side TU and the side WV that are in a side-to-side relationship in the quadrangle TUWW. And a straight line H4 that connects the midpoints of these sides between the side TW and the side UV that are in a side-to-side relationship with the quadrangle TUWW.

  Here, if the projective transformation (deformation 1) is an affine transformation, the point Z1 is moved to the point Z2, and therefore the point Z2 and the point Z3 coincide. That is, the distance d (straight line distance) between the point Z2 and the point Z3 shown in FIG. 9B is zero. On the other hand, when the projective transformation (deformation 1) is a coordinate transformation other than the affine transformation, since the point Z2 and the point Z3 do not coincide with each other, the distance d between the point Z2 and the point Z3 shown in FIG. (The distance d is evaluated in terms of the number of pixels (number of dots) at 72 dpi, which is the reference resolution in the PDF file). Note that the point Z2 and the point Z3 coincide with each other in the case of affine transformation because the affine transformation has the property that the centroid before transformation is transferred to the centroid after transformation. In this case, the point Z2 and the point Z3 do not coincide with each other because the center of gravity of the square before conversion is not moved to the center of gravity of the square after conversion.

  That is, the distance d between the point Z2 and the point Z3 is equal to the projective transformation in the case of correcting the distortion of the correction target range by the projective transformation without being limited to the affine transformation when the quadrangle PQRS is corrected to the square TUUV. Can be considered as a representative value of the error in the corrected position of a predetermined pixel (the center of gravity of the quadrangle PQRS).

  If the quadrangle PQRS and the quadrangle TUUV are both parallelograms, the projective transformation is inevitably affine transformation, so the distance d is zero.

  On the other hand, if the quadrangle TUUV is a parallelogram and the quadrangle PQRS is a non-parallelogram, the projective transformation is a coordinate transformation other than the affine transformation. However, in this case, if the distance d is small, the quadrangle PQRS can be considered as a shape approximating a parallelogram. In such a case, correction is performed using a projective transformation other than affine transformation and correction is performed using affine transformation. The difference in correction accuracy is small compared with the case where it is performed, and it can be considered that there is no problem even if correction is performed by affine transformation.

  However, even if the quadrangle TUUV is a parallelogram, when the distance d increases, the quadrangle PQRS becomes a shape away from the parallelogram, and correction is performed by projection transformation other than affine transformation and correction by affine transformation. The difference in correction accuracy is conspicuous between the two, and it is considered that when the correction is performed by affine transformation, the difference becomes conspicuous.

  Therefore, in the present embodiment, the drawing command generation unit 100 obtains first error information indicating the distance d between the point Z2 and the point Z3 as a correction error for the correction target range 60 in S104, and in S105, It is determined whether the correction error is within an allowable range.

  Specifically, the drawing command generation unit 100 performs a distortion correction example in the case of performing projective transformation (not limited to affine transformation) on the correction target range 60 set in S103 to a corrected display range 70 (for example, A4 paper). Then, the coordinate value of the physical center of gravity of the correction target range 60 and the coordinate value of the physical center of gravity of the corrected display range 70 are obtained. Note that the method for obtaining the distortion correction example is as described in the section [Prerequisite Technology]. That is, the distortion correction matrix is obtained by using the coordinate values of the four vertices of the correction target range 60 as the coordinate values before the conversion and the four vertices of the corrected display range 70 as the coordinate values after the conversion. Since the distortion correction matrix obtained here (distortion correction matrix for converting the correction target range 60 to the corrected display range 70) is used in the subsequent processing, the distortion correction matrix is stored in the storage unit 14 as the matrix A. Further, if the quadrangle PQRS in FIG. 9A is the correction target range 60 and the quadrangle TUWW is the corrected display range 70, Z1 in FIG. 9A is the physical center of gravity of the correction target range 60, and Z2 in FIG. 9A is the corrected display range 70. It becomes the physical center of gravity.

  Subsequently, the drawing command generation unit 100 uses the matrix A and the coordinate value of the physical centroid (centroid position) of the correction target range 60 to move the physical centroid of the correction target range 60 using the matrix A. Is obtained (the position Z3 in FIG. 9A corresponds to the post-movement position). Then, the drawing command generation unit 100 uses the physical center of gravity (center of gravity position) of the corrected display range 70 (target range) and the post-movement position as the first error information for the correction target range 60 set in S103. Find the distance d.

  After S104, the drawing command generation unit 100 determines whether or not the correction error indicated in the first error information is within an allowable range by performing threshold processing on the first error information obtained in S104. (S105). For example, the drawing command generation unit 100 compares the first error information with a preset threshold value, and determines that the correction error is within an allowable range if the first error information is less than the threshold value. If the information is equal to or greater than the threshold, it is determined that the correction error is not within the allowable range. When the first error information (distance d) is converted into the number of pixels (number of dots) at 72 dpi that is the reference resolution in the PDF file and evaluated, the threshold value compared with the first error information is, for example, 1 Is set.

  If the drawing command generation unit 100 determines that the correction error is within the allowable range (YES in S105), the drawing command generation unit 100 designates a range to be a mask area (non-display area) when displaying the captured image (S106). In S106, the drawing command generation unit 100 designates a range outside the corrected display range set in S103 as a mask region.

  Subsequently, the drawing command generation unit 100 generates a drawing command for the correction target range 60 (S107). Specifically, it is as follows.

  First, the drawing command generation unit 100 obtains a distortion correction matrix for affine transformation using the correction target range 60 set in S103. Specifically, a distortion correction matrix is obtained as follows. The drawing command generation unit 100 specifies a vertex whose inner angle is closest to the right angle in the correction target range 60, and among the four sides of the correction target range 60, the first side and the second side extending from the specified vertex are determined. Extract. The drawing command generation unit 100 generates a parallelogram composed of a first side, a second side, a third side parallel to the first side, and a fourth side parallel to the second side. Set. For example, when the range indicated by reference numeral 200 shown in FIG. 3 is the correction target range 60, the left side 201 and the lower side 202 are extracted as the first side and the second side, and the parallelogram 200b shown in FIG. 5A is set. It has become so. When there are a plurality of vertices whose interior angles are closest to the right angle, any one is randomly selected.

  The drawing command generation unit 100 extracts two sides extending from a vertex whose inner angle is closest to the right angle in the correction target range 60, but extracts two sides extending from any vertex of the correction target range 60. In particular, the inner angle is not limited to the two sides extending from the vertex closest to the right angle.

  Then, the drawing command generation unit 100 sets the coordinate values of the four vertices of the parallelogram set as described above as coordinate values before conversion, and sets the four corrected display ranges (for example, A4) set in S103. An affine transformation distortion correction matrix is obtained using the vertex coordinate values as converted coordinate values.

  After the distortion correction matrix is obtained, the drawing command generation unit 100 generates a drawing command for the correction target range 60. Specifically, the drawing command generation unit 100 performs affine transformation in accordance with the obtained distortion correction matrix on the image data, and hides the mask area in S106 (the area outside the corrected display range 70). A drawing command for controlling the computer to perform display processing is generated (hides the mask area) (S107).

  FIG. 14 is a schematic diagram showing the drawing command specified in S107. (1) in FIG. 14 indicates that the mask area is not displayed (the range to be drawn is limited to the range specified by the command), and (2) in FIG. 14 indicates the affine transformation. FIG. 14 (3) shows image drawing (display processing).

  When S107 is performed, the process proceeds to S120. In step S120, the formatting processing unit 101 creates a PDF file in which image data of a captured image that has not been processed by the distortion correction matrix is created, and the drawing command generation unit 100 converts the drawing command generated in step S107 to PDF. Append to file. Thereafter, the PDF file is stored in the storage unit 14.

  When the user inputs an instruction to open the PDF file generated via S106 and S107, a drawing process corresponding to the drawing command generated in S107 is performed. That is, the computer that processes the PDF file performs affine transformation on the image data using a single distortion correction matrix, and displays an image of the image data while masking the range outside the corrected display range 70. It has become. That is, the image within the corrected display range 70 is displayed.

  Next, the case of NO in S105 will be described. When the drawing command generation unit 100 determines that the correction error of the first error information obtained for the correction target range 60 is not within the allowable range (NO in S105), the drawing command generation unit 100 performs division processing on the corrected display range to perform 4 Four post-correction division ranges are generated, and division processing is performed on the correction target range 60 to generate four division ranges (S108). Hereinafter, the processing content of S108 will be described in detail.

  First, as shown in FIG. 10, the drawing command generation unit 100 divides the corrected display range 70 into four to generate four corrected divided ranges. Specifically, the drawing command generation unit 100 detects the midpoint of each of the upper side 71, the lower side 72, the right side 73, and the left side 74 of the corrected display range 70. In FIG. 10, the midpoint 71m is the midpoint of the upper side 71, the midpoint 72m is the midpoint of the lower side 72, the midpoint 73m is the midpoint of the right side 73, and the midpoint 74m is the midpoint of the left side 74. is there.

  Subsequently, the drawing command generation unit 100 sets a straight line that connects the midpoints of the two sides that are opposite to each other in the corrected display range 70. That is, a straight line h3 that connects the midpoints of these sides between the upper side 71 and the lower side 72, and a straight line h4 that connects the midpoints of these sides between the left side 74 and the right side 73. Is set. In addition, the drawing command generation unit 100 detects an intersection 70k between the straight line h3 and the straight line h4. 10 corresponds to the physical center of gravity of the corrected display range 70 obtained in S104. Specifically, if the square TUUV in FIG. 9A is the corrected display range 70, the intersection 70k in FIG. 10 corresponds to the physical center of gravity Z2 in FIG. 9A, and the straight line h3 in FIG. 10 corresponds to the straight line H3 in FIG. The straight line h4 in FIG. 10 corresponds to the straight line H4 in FIG. 9A. That is, the drawing command generation unit 100 stores the calculation process when the physical center of gravity of the corrected display range 70 is obtained in S104, and refers to the calculation process, so that the midpoints 71m, 72m, 73m, 74m, straight lines h3 and h4, and intersection 70k may be detected.

  Next, as shown in FIG. 10, the drawing command generation unit 100 uses the four vertices 75 to 78 of the corrected display range 70, the midpoints 71 m, 72 m, 73 m, and 74 m, and the intersection 70 k to correct the corrected display range. 70 is divided into four post-correction division ranges. Specifically, a corrected divided range 70a composed of a quadrangle having an upper right vertex 75, a middle point 71m, a middle point 73m, and an intersection 70k as vertices in the corrected display range 70, and a lower right vertex 76 in the corrected display range 70. And a corrected divided range 70b made up of a quadrangle having a vertex at a middle point 73m, a middle point 72m, and an intersection 70k, and a lower left vertex 77, a middle point 72m, a middle point 74m, and an intersection 70k as a vertex. And a post-correction divided range 70d composed of a quadrangle whose vertexes are the upper left vertex 78, the middle point 74m, the middle point 71m, and the intersection point 70k of the corrected display range 70.

  That is, the corrected display range 70 includes (1) a corrected divided range 70a surrounded by the upper side 71, the right side 73, the straight line h4, and the straight line h3, and (2) the right side 73, the lower side 72, and the straight line h4. And a post-correction divided range 70b surrounded by a straight line h3, (3) a lower side 72, a left side 74, a straight line h4, a post-correction divided range 70c surrounded by a straight line h3, and (4) a left side 74, It is divided into a post-correction division range 70d surrounded by the upper side 71, the straight line h4, and the straight line h3.

  After the corrected display range 70 is divided into four and the corrected divided ranges 70a to 70d are generated as described above, the drawing command generation unit 100 divides the correction target range 60 into four, and sets the four divided ranges. Generate. Specifically, it is as follows.

  First, the drawing command generation unit 100 reads out from the storage unit 14 the matrix A (distortion correction matrix for projectively converting the correction target range 60 to the corrected display range 70) used for generating the first error information in S104. The inverse matrix of matrix A is obtained. Since the inverse matrix of the matrix A is used in the subsequent processes, the drawing command generation unit 100 stores the inverse matrix of the matrix A as the matrix B in the storage unit 14.

  Subsequently, the drawing command generation unit 100 performs coordinate conversion using the matrix B on the five points of the midpoints 71m, 72m, 73m, 74m, and the intersection 70k after the corrected display range 70, thereby changing the point. 61 m, 62 m, 63 m, 64 m, and 60 k are obtained (see FIG. 10). Point 61m is the conversion result of midpoint 71m, point 62m is the conversion result of midpoint 72m, point 63m is the conversion result of midpoint 73m, point 64m is the conversion result of midpoint 74m, and point 60k Is the conversion result of the intersection 70k. That is, the points 61m, 62m, 63m, 64m, and 60k correspond to the positions of the midpoints 71m, 72m, 73m, 74m, and the intersection 70k before the projective transformation from the correction target range 60 to the corrected display range 70, respectively. ing. Note that the points 61m, 62m, 63m, 64m, and 60k are not necessarily converted to positions that coincide with one pixel on the captured image (processing target) in which the correction target range 60 is shown. Although it may not be an integer value, it is used as a small number without being converted to an integer.

  The point 61m is not necessarily located on the midpoint of the side 61. Similarly, the point 62m is not necessarily located on the midpoint of the lower side 62. Similarly, the point 63m is not necessarily located on the midpoint of the right side 63. Similarly, the point 64m is not necessarily located on the midpoint of the left side 64. Further, the point 60k does not necessarily coincide with the physical center of gravity of the correction target range 60.

  Subsequently, as illustrated in FIG. 10, the drawing command generation unit 100 uses the four vertices 65 to 68 and the points 61 m, 62 m, 63 m, 64 m, and 60 k of the correction target range 60 to generate four correction target ranges 60. Divide into division ranges.

  Specifically, the correction target range 60 includes a divided range 60 a formed of a quadrangle having the upper right vertex 65, the point 61 m, the point 63 m, and the point 60 k as vertices, and the lower right vertex 66 of the correction target range 60. , A divided range 60b composed of a quadrangle with points 63m, 62m, and 60k as vertices, and a divided range 60c composed of a quadrangle with vertices at the lower left vertex 67, points 62m, 64m, and 60k of the correction target range 60. And the upper left vertex 68 of the correction target range 60, the point 64m, the point 61m, and the point 70k are divided into a divided range 60d composed of a quadrangle.

  After S108, the drawing command generation unit 100 selects any one of the four divided ranges 60a to 60d as a processing target. Then, the drawing command generation unit 100 calculates second error information (second difference information) for the divided range selected as the processing target (S109). The second error information is a value indicating a correction error like the first error information, but differs from the first error information in that the target is not a correction target range but a divided range. That is, the second error information is a value indicating an error (correction error) when correcting the distortion by limiting the projection transformation to the affine transformation in contrast to correcting the division range by the projective transformation (in other words, The second error information can be said to be a value indicating a difference between the case where the division range is subjected to distortion correction by projective transformation and the case where distortion is corrected by affine transformation. Hereinafter, the process of S109 will be specifically described.

  For example, it is assumed that the drawing command generation unit 100 selects the rightmost and uppermost divided range 60a as the processing target among the four divided ranges 60a to 60d divided from the correction target range 60 of FIG. In this case, the drawing command generation unit 100 extracts the corrected divided range 70a corresponding to the divided range 60a to be processed from the four corrected corrected ranges 70a to 70d divided from the corrected display range 70. That is, the drawing command generation unit 100 refers to the positional relationship of the divided range 60a with respect to the correction target range 60 (the rightmost and uppermost divided range), and includes the four corrected divided ranges 70a to 70d of the corrected display range 70. Among them, the post-correction divided range 70a having the same positional relationship as the above-described positional relationship with respect to the corrected display range 70 is extracted.

  Then, the drawing command generation unit 100 obtains the coordinate value of the physical centroid of the divided range 60a and the coordinate value of the physical centroid of the corrected divided range 70a. For example, if the quadrangle PQRS in FIG. 9A is the divided range 60a and the quadrangle TUWW is the corrected divided range 70a, Z1 in FIG. 9A becomes the physical center of gravity of the divided range 60a, and Z2 in FIG. 9A is the physical in the corrected divided range 70a. Become the center of gravity. Further, the drawing command generation unit 100 uses the matrix A stored in the storage unit 14 to obtain the coordinate value of the position (post-movement position) after moving the physical center of gravity of the divided range 60a. ing.

  Then, the drawing command generation unit 100 uses the physical center of gravity (center of gravity position) of the corrected divided range 70a (target range) and the physical center of gravity (center of gravity position) of the divided range 60a as the second error information regarding the divided range 60a. Is moved by the matrix A, and the distance from the post-movement position is obtained.

  After the second error information is calculated as described above (S109), the drawing command generation unit 100 performs threshold processing on the second error information obtained in S109, thereby processing target divided range 60a. It is determined whether or not the correction error is within an allowable range (S110). The threshold processing in S110 is the same as the threshold processing in S105, and the threshold is also assumed to be the same value. That is, the second error information is compared with a preset threshold, and if the second error information is less than the threshold, it is determined that the correction error is within the allowable range, and if the second error information is equal to or greater than the threshold. It is determined that the correction error is not within the allowable range.

  When the drawing command generation unit 100 determines that the correction error for the divided range 60a is within the allowable range (YES in S110), the second error information has been calculated for all the divided ranges generated at the present time. It is determined whether or not (S111). Here, since the second error information has been calculated only for the divided range 60a out of the four divided ranges 60a to 60d shown in FIG. 10A, the drawing command generation unit 100 performs all the divisions currently generated. It is determined that the second error information has not been calculated for the range (NO in S111), and the processing target is changed from the divided range 60a to the next divided range (the second error information is an uncalculated divided range) ( (S112), the process after S109 is repeated.

  In S110, when the drawing command generation unit 100 determines that the correction error for the division range 60a to be processed is not within the allowable range (NO in S110), the process proceeds to S108 and the correction error is within the allowable range. The non-divided divided range 60a (see FIG. 10) and the corrected divided range 70a corresponding to the divided range 60a are further divided. Specifically, as illustrated in FIG. 11, four new divided ranges 60a1 to 60a4 are generated from the divided range 60a, and four new corrected divided ranges 70a1 to 70a4 are generated from the corrected divided range 70a. It is like that.

  The division method for the division range 60a and the corrected division range 70a in S108 is the same as the division method for the correction target range 60 and the corrected display range 70 (the division target is from the correction target range 60 and the corrected display range 70). The processing contents are the same except that the divided range 60a and the corrected divided range 70a are replaced.

  That is, the division method of the divided range 60a and the corrected divided range 70a shown in FIG. 11 is as follows. First, as shown in FIG. 11, the drawing command generation unit 100 includes a midpoint 171m of the upper side of the corrected divided range 70a, a midpoint 172m of the lower side of the corrected divided range 70a, and a right side of the corrected divided range 70a. A point 173m and a midpoint 174m on the left side of the post-correction divided range 70a are set.

  In addition, the drawing command generation unit 100 sets a straight line that connects the midpoints of the two sides that are opposite to each other in the corrected divided range 70a. Specifically, a straight line h13 connecting the midpoint 171m and the midpoint 172m and a straight line h14 connecting the midpoint 173m and the midpoint 174m are set. In addition, the drawing command generation unit 100 detects an intersection 170k between the straight line h13 and the straight line h14 (physical center of gravity of the corrected divided range 70a).

  Subsequently, as illustrated in FIG. 11, the drawing command generation unit 100 uses the four vertices of the corrected divided range 70a, the midpoints 171m, 172m, 173m, 174m, and the intersection 170k to generate four corrected corrected ranges 70a. This is divided into two post-correction division ranges 70a1 to 70a4. Specifically, a corrected divided area 70a1 composed of a quadrangle having apexes at the upper right vertex (reference numeral 75), the midpoint 171m, the midpoint 173m, and the intersection 170k of the corrected divided area 70a, and the right of the corrected divided area 70a. A corrected divided range 70a2 composed of a quadrangle having a lower vertex (reference numeral 73m), a middle point 173m, a middle point 172m, and an intersection point 170k, and a lower left vertex (reference numeral 70k) and a middle point 172m of the corrected divided area 70a. From a post-correction divided range 70a3 composed of a quadrangle having a point 174m and an intersection 170k as vertices, and a quadrangle having a vertex at the upper left vertex (reference numeral 71m), the midpoint 174m, the midpoint 171m, and the intersection 170k having a vertex A post-correction divided range 70a4 is generated.

  As described above, the post-correction division range 70a is divided into four to generate post-correction division ranges 70a1 to 70a4, and then the division range 60a is divided into four to generate four division ranges 60a1 to 60a4. Specifically, it is as follows.

  First, the drawing command generation unit 100 uses the matrix B stored in the storage unit 14 to coordinate the five points of the midpoints 171m, 172m, 173m, 174m, and the intersection 170k after the corrected divided range 70a. By performing the conversion, points 161m, 162m, 163m, 164m, and 160k are obtained (see FIG. 11). Point 161m is the conversion result of midpoint 171m, point 162m is the conversion result of midpoint 172m, point 163m is the conversion result of midpoint 173m, point 164m is the conversion result of midpoint 174m, and point 160k Is the conversion result of the intersection 170k. That is, the points 161m, 162m, 163m, 164m, and 160k correspond to the positions of the midpoints 171m, 172m, 173m, 174m, and the intersection point 170k before projective transformation from the division range 60a to the post-correction division range 70a, respectively. Yes. In FIG. 11, for convenience of explanation, the points 161m to 164m are located at the midpoint on the side of the divided range 60a, but are not necessarily located at the midpoint on the side of the divided range 60a. This is similar to the case of the correction target range 60 in FIG. Further, the point 160k is not necessarily located at the physical center of gravity of the divided range 60a, as in the case of the correction target range 60 in FIG.

  Subsequently, as illustrated in FIG. 11, the drawing command generation unit 100 uses the four vertices 161m, 162m, 163m, 164m, and 160k of the divided range 60a to divide the divided range 60a into the four divided ranges 60a1 to 60a4. Divide into

  Specifically, the division range 60a includes a division range 60a1 composed of a quadrangle having apexes at the upper right vertex (reference numeral 65) of the division range 60a, the points 161m, 163m, and 160k, and the lower right vertex of the division range 60a ( 63m), a division range 60a2 composed of a quadrangle having points 163m, 162m, and 160k as vertices, and a quadrangle having vertices at the lower left vertex (reference 60k), points 62m, 64m, and 160k of the division range 60a. Is divided into a divided range 60a4, and a divided range 60a4 made of a quadrangle having apexes at the upper left vertex (reference numeral 61m), the point 164m, the point 161m, and the point 160k of the divided range 60a.

  In this way, further division is performed on the post-correction division range 70a and the division range 60a, and new post-correction division ranges 70a1 to 70a4 and division ranges 60a1 to 60a4 are generated.

  When the drawing command generation unit 100 generates a new division range and a corrected division range as described above (S108), the division in which the second error information is not calculated among all the division ranges generated at the present time. The range is set as a processing target, and the processing after S109 is repeated.

  In S111, when the drawing command generation unit 100 determines that the second error information has been calculated for all the divided ranges generated at the present time, the drawing command generation unit 100 shifts the processing to S113. The fact that the second error information has been calculated for all the divided ranges generated at the time of S111 means that the correction errors of all the divided ranges generated at the time of S111 are within the allowable range. That's right.

  Next, S113 will be described. In S113, the drawing command generation unit 100 performs a process of further dividing the division range generated so far into two partial areas. This process will be described in detail below.

  For example, it is assumed that the division ranges generated so far are four as shown in FIG. 10, and one of them is the division range 60a. Here, as shown in FIG. 12, the drawing command generation unit 100 sets one diagonal line for the divided range 60a, and the divided range 60a is divided into a partial area 60aa and a partial area 60ab that face each other across the diagonal line. To divide. Here, the diagonal line may be any of a straight line connecting the upper left vertex and the lower right vertex and a straight line connecting the upper right vertex and the lower left vertex, but in this embodiment, the upper left vertex and the lower right vertex are connected. Set to a straight line.

  That is, as shown in FIG. 12, the divided range 60a includes a partial area 60aa that is a triangle formed by the diagonal line of the divided range 60a, the upper side of the divided range 60a, and the right side of the divided range 60a, and the diagonal line and the divided range 60a. It is divided into a partial area 60ab which is a triangle composed of the lower side and the left side of the division range 60a.

  The drawing command generation unit 100 divides not only the divided range 60a but also other divided ranges into two partial areas in the same manner. Therefore, the four divided regions (squares) shown in FIG. 10 are divided into eight partial regions (triangles) as shown in FIG.

  In addition, the drawing command generation unit 100 also divides the four corrected divided ranges shown in FIG. 10 into two corrected partial regions in the same manner as the four divided ranges. That is, as shown in FIG. 12, the corrected divided range 70a corresponding to the divided range 60a includes a diagonal line connecting the upper left vertex and the lower right vertex, the upper side of the corrected divided range 70a, and the right side of the corrected divided range 70a. The corrected partial area 70aa, which is a triangle, and the corrected partial area 70ab, which is a triangle including the diagonal line, the lower side of the corrected divided area 70a, and the left side of the corrected divided area 70a. Further, not only the post-correction division range 70a but also the other post-correction division ranges are similarly divided into two partial areas. Therefore, the four corrected divided areas (rectangles) shown in FIG. 10 are divided into eight corrected partial areas (triangles) as shown in FIG.

  After the partial area and the corrected partial area are set as shown in FIG. 13, the drawing command generation unit 100 selects any one of the partial areas as a processing target. For example, it is assumed that the partial area 60ab shown in FIG. 12 is selected as a processing target.

  Subsequently, the drawing command generation unit 100 specifies a range to be a mask area (non-display area) when displaying a captured image for the partial area to be processed (S114). In S114, the drawing command generation unit 100 designates a range outside the corrected partial area 70ab corresponding to the partial area 60ab to be processed as a mask area.

  After S114, the drawing command generation unit 100 generates a drawing command for the partial area 60ab to be processed (S115). Specifically, it is as follows.

  First, the drawing command generation unit 100 obtains a distortion correction matrix for affine transformation using the partial area 60ab to be processed. Specifically, a distortion correction matrix is obtained as follows. In order to obtain a distortion correction matrix for affine transformation from the triangular partial region 60ab, it is first necessary to set a parallelogram including the triangular partial region 60ab. Specifically, it is as follows.

  The drawing command generation unit 100 extracts the first side and the second side other than the side corresponding to the diagonal line of the divided range 60a out of the three sides forming the triangular partial region 60ab (see FIG. 12). A parallelogram composed of a first side and a second side, a side parallel to the first side, and a side parallel to the second side is set. In FIG. 12, the first side is the left side of the division range 60a, the second side is the lower side of the division range 60a, the side parallel to the first side is the broken line J, A side parallel to the second side is a broken line K.

  Then, the drawing command generation unit 100 uses the coordinate values of the four vertices of the parallelogram set from the partial area 60ab as described above as coordinate values before conversion, and the corrected partial area 70aa corresponding to the partial area 60ab belongs to it. Using the coordinate values of the four vertices of the post-correction divided range 70a as the post-conversion coordinate values, a distortion correction matrix for affine transformation is obtained.

  After the distortion correction matrix is obtained, the drawing command generation unit 100 generates a drawing command for the partial region 60ab. Specifically, the drawing command generation unit 100 performs affine transformation corresponding to the distortion correction matrix obtained from the partial region 60ab on the image data, and specifies the mask region specified in S114 (range outside the corrected partial region 70ab). Is hidden (the mask area is hidden), and a drawing command for controlling the computer to perform display processing is generated (S115).

  After S115, the drawing command generation unit 100 determines whether drawing commands have been generated for all partial areas (S116).

  When drawing commands have not been generated for all partial areas (NO in S116), the drawing command generation unit 100 selects the next partial area from the partial area 60ab (the part that has not generated a drawing command) as the processing target. After changing to (region) (S117), the processing after S114 is repeated.

  If the generation of the drawing command is completed for all the partial areas (YES in S116), the process proceeds from S116 to S120. In S120, the formatting processing unit 101 creates a PDF file in which image data of a captured image that has not been processed by the distortion correction matrix is created, and the drawing command generating unit 100 generates a drawing command generated for each partial area. All are added to the PDF file. Thereafter, the PDF file is stored in the storage unit 14.

  FIG. 15 is a schematic diagram showing a drawing command embedded in the PDF file obtained through S108 to S118. (4) to (6) in FIG. 15 are drawing commands for a certain partial area, and (7) to (9) in FIG. 15 are drawing commands for another partial area. (4) and (7) indicate that the mask area is not displayed, (5) and (8) indicate affine transformation, and (6) and (9) indicate image drawing (display processing). ).

  When the user inputs an instruction to open the PDF file generated through the processing after S108, the drawing processing corresponding to the drawing command generated for each partial area in S115 is performed. For example, the computer that processes the PDF file processes the drawing command set for the partial area 60aa (see FIG. 12 or 13) as follows. The computer performs affine transformation on the image data (data of the entire image) using the distortion correction matrix set from the partial area 60aa, and masks the range outside the corrected partial area 70aa corresponding to the partial area 60aa. Displays an image of image data. Such processing is performed for all drawing commands generated for each partial area. As a result, the image of each partial area in FIG. 13 is coordinate-converted into an image of each corrected partial area, and an image obtained by combining all the corrected partial area images (image of the corrected display range 70) is displayed. become.

  According to the file generation process of the present embodiment described above, when affine transformation is performed on a correction target range using a single distortion correction matrix, a correction error increases (that is, distortion correction is appropriately performed). If the correction target range is divided into a plurality of partial areas and a distortion correction matrix is set for each partial area, it is possible to realize distortion correction with reduced correction errors (per one). By dividing and reducing the corresponding range in the distortion correction matrix, it is possible to reduce the correction error in the corresponding range with one distortion correction matrix and to suppress the correction error as a whole). Therefore, when generating a PDF file in which distortion correction is performed by affine transformation when displaying an image, a PDF file in which distortion correction is appropriately performed can be generated.

  FIG. 19A shows an output image obtained by correcting the distortion of the correction target image of FIG. 3 by projective transformation. FIG. 19B shows the eight correction target images of FIG. It is an output image that is divided into partial areas and subjected to distortion correction by affine transformation for each partial area. As shown in FIG. 19, even in the case of affine transformation, it can be understood that an output image comparable to that in the case of projective transformation can be obtained by dividing into a plurality of partial regions and performing conversion for each partial region.

  Further, when a distortion correction matrix is obtained from the generated rectangular division range 60a and affine transformation is performed (a parallelogram is set by the same method as in FIG. 6 to obtain a distortion correction matrix), near the boundary of the division range. There may be a problem that distortion becomes large. On the other hand, in the present embodiment, the above-described inconvenience can be suppressed by further dividing the quadrangular division range into triangles and obtaining a distortion correction matrix for each partial area with each triangle as a partial area. is there.

  Further, according to S108 to S112 of the present embodiment, the drawing command generation unit 100 obtains the second error information for each divided range generated by the dividing process, and the error indicated by the second error information is within the allowable range. The process of dividing the division range that is not present and generating further division ranges is repeated until the error of the second error information falls within the allowable range in all division ranges. As a result, the correction error of the corresponding range (partial region) in one distortion correction matrix can be suppressed within the allowable range, and the distortion correction is performed in order to suppress the correction error within the allowable range in all partial regions. It can be performed with high accuracy.

  Also, according to the present embodiment, when a PDF file is generated, a distortion correction matrix is obtained, but correction (pixel conversion) using the distortion correction matrix is not performed, so that the load on the computer can be reduced, and an image file can be quickly obtained. (The processing for obtaining the distortion correction matrix is not heavy, but pixel conversion is heavy).

  Further, according to the present embodiment, since the image data embedded in the PDF file is data (original data) before distortion correction, there is an advantage that data before distortion correction can be left. That is, when the distortion correction in the generated PDF file is not appropriate, it is possible to recreate the PDF file using the original data.

  In the present embodiment, the generated PDF file is temporarily stored in the storage unit 14, but the PDF file is temporarily stored in the storage unit 14 and then transmitted to the transmission destination designated by the user. Of course, it doesn't matter if it comes to be. The transmission destination is not particularly limited as long as it is an apparatus in which an application capable of displaying an image of a PDF file is installed. That is, general-purpose personal computers, various portable terminals, server devices, multifunction devices, and the like can be given.

  In this embodiment, the corrected display range 70 is selected from the standard paper size in S103. However, the display range 70 is not limited to the standard paper size. For example, the user uses the touch panel 12 to select an arbitrary rectangle. It may be set as a display range after correction.

  Also, according to the PDF file generated in the file generation mode of the present embodiment, a plurality of partial areas are formed through the processing of S108 to S116, and distortion correction row examples set for each partial area Accordingly, distortion correction is performed for each partial region. Here, the fact that the distortion correction is performed for each partial area in accordance with the distortion correction row example set for each partial area is a drawing described in the PDF file even if it is not known from the displayed image. It is clear from analyzing the command. That is, as shown in FIG. 15, if it is specified that a drawing command related to distortion correction and mask designation is described for each partial area, the process proceeds from S108 to S116 in the file generation mode of this embodiment. It is clear that this is a generated PDF file.

[Embodiment 2]
According to S108 to S111 of the first embodiment, the division process is repeated until the correction error reaches the allowable range in all the divided ranges. In such a form, the divided ranges and the corrected divided ranges are not changed. The number may be too large, and in this case, a processing delay occurs when displaying an image of the PDF file. Therefore, instead of adopting S108 to S111, after S105, the drawing command generation unit 100 determines the division range and the corrected division range according to the correction error (distance d) of the first error information obtained in S104. The number may be set, the set number of divided ranges and the corrected divided ranges may be generated, and the process may proceed to S113.

  For example, as illustrated in FIGS. 18A to 18C, the drawing command generation unit 100 generates a dividing straight line (broken line) that equally divides two sides that are opposite to each other in the corrected display range. The corrected display range is divided based on the dividing straight line.

  When the distance d that is the correction error of the first error information is 1 <d ≦ 5, n = 2 is set as shown in FIG. In this case, processing is performed in the same manner as the first division processing in the first embodiment, the corrected display range 701 is divided into four, and the correction target range 601 is also divided into four. However, in the first embodiment, the second division is performed in some cases, but the second division is not performed here.

  When the distance d is 5 <d ≦ 10, n = 3 is set as shown in FIG. 18B, and the corrected display range 702 is divided into nine corrected divided ranges. Also, in this case, the intersections of the dividing straight lines (broken lines) that divide each side of the corrected display range 702 into n equal parts, and the intersections of the dividing straight lines (broken lines) and the sides of the corrected display range 702 are: The coordinates are converted by the inverse matrix of the distortion correction matrix for projective conversion of the correction target range 602 to the corrected display range 702. A black circle portion shown on the corrected display range 702 in FIG. 18B is an intersection before conversion by an inverse matrix, and a black circle portion shown on the correction target range 602 in FIG. 18B is an inverse matrix. This is the intersection after conversion.

  Subsequently, as illustrated in FIG. 18B, the division range corresponding to each post-correction division range on the post-correction display range 702 is generated by dividing the correction target range 602.

  Specifically, if all four vertices are the intersections (black circles) as in the first corrected divided range 702a in FIG. 18B, the intersections obtained by transforming these intersections with the inverse matrix are the vertices. A rectangular first divided range 602a is formed as a divided range corresponding to the first corrected divided range 702a.

Also, as in the second corrected divided range 702b in FIG. 18B, when one vertex coincides with the vertex (white circle portion) of the corrected display range 702, the positional relationship with the vertex of the corrected display range 702 Is formed as a divided range corresponding to the second post-correction divided range 702b. The second divided range 602b has one vertex corresponding to the vertex (white circle portion) of the correction target range 602 having the same.
For example, when one vertex of the second corrected divided range 702b matches the lower left vertex of the corrected display range 702, the second divided range 602b having the lower left vertex of the correction target range 602 as one vertex is formed. In this case, the three vertices of the second divided range 602b coincide with the points obtained by converting the three vertices of the second corrected divided range 702b (the intersection (black circle)) with the inverse matrix.

  In this way, each divided range corresponding to each corrected divided range on the corrected display range 702 can be formed.

  When the distance d is d> 10, n = 4 is set as shown in FIG. 18C, and the corrected display range 703 is divided into 16 corrected divided ranges. Also in this case, the intersection of the straight lines for dividing (broken lines) that divide each side of the corrected display range 703 into n equal parts, and the intersection of the straight lines for dividing (broken lines) and the respective sides of the corrected display range 703 are as follows. The coordinate conversion is performed by the inverse matrix of the distortion correction matrix for projective conversion of the correction target range 603 to the corrected display range 703. Subsequently, by dividing the correction target range 603 using the intersection obtained by the coordinate conversion, each divided range corresponding to each corrected divided range on the corrected display range 703 is generated. Note that the method for generating the divided range is the same as that in the case of FIG.

  According to the division method of the present embodiment described above, the number of division ranges is set (adjusted) according to the correction error of the first error information. In the case of this form, the correction error of the divided range is slightly larger than that of the first embodiment, but the number of post-correction divided ranges and the upper limit of the number of divided ranges are determined. And the problem that the number of division ranges becomes too large can be suppressed.

[Embodiment 3]
Moreover, as a configuration for suppressing the problem that the number of division ranges and the number of division ranges after correction are too large, the following forms can be cited in addition to the second embodiment.

  In the present embodiment, a specified number (for example, 32) is preset as a threshold for the number of division ranges. The drawing command generation unit 100 determines whether or not the total number of division ranges exceeds the specified value when the division processing is performed before the division processing of S108 is performed after determining NO in S110. Yes.

  If the drawing command generation unit 100 determines that the specified number is not exceeded, it performs the dividing process of S108 as it is and proceeds to S109.

  In contrast, if the drawing command generation unit 100 determines that the specified number is exceeded, the drawing command generation unit 100 displays a warning image on the touch panel 12 to notify the user that the specified number will be exceeded if it is divided as it is. In this warning image, (1) the first option for performing the division process even if the total number of the divided ranges exceeds the specified value, and (2) the total number of the divided ranges are defined by stopping the subsequent divided processes. A UI for allowing the user to select one of the second option for shifting to S113 while keeping the value within the range and (3) the third option for stopping the generation of the PDF file itself is displayed. Yes.

  When the user selects the first option, even if the total number of division ranges exceeds the specified value, the division process is repeated until the error of the second error information is within the allowable range for all the division ranges as in the first embodiment. It is. When the user selects the second option, S108 to S112 are skipped, and the process proceeds to S113. In this case, among all the generated divided ranges, a divided range in which the error of the second error information exceeds the allowable range remains, but the total number of divided ranges can be suppressed. When the user selects the third option, the file generation mode itself is interrupted (the generation of the PDF file itself is stopped).

  According to the present embodiment, priority is given to correction accuracy by performing division even when the total number of division ranges exceeds a specified value, or whether display speed is given priority by suppressing the total number of division ranges to a specified value, Alternatively, there is an advantage that the user can select whether to stop the generation of the image file itself.

[Embodiment 4]
When displaying an image of a PDF file generated through the processing after S108, a display defect that a gap occurs at the boundary between corrected partial areas (see FIG. 13) depending on the type of the PDF display program and the display magnification. May occur. FIG. 17A is a diagram showing the display defect. In FIG. 17A, line-shaped noise is shown as a display defect.

  Therefore, the drawing command generation unit 100 includes a drawing command generated for a certain partial area including a corrected partial area (corrected target partial area) corresponding to the partial area and a range wider than the corrected partial area. The mask area may be set so that display processing (drawing) is performed, and the mask area may be set in the same manner for all drawing commands. In this way, the display range based on the first corrected partial area is extended to the second corrected partial area located next to the first corrected partial area, so the first corrected part A problem that a gap is generated at the boundary between the region and the second corrected partial region can be suppressed. FIG. 17B shows a display image when the mask area is set as described above. In FIG. 17B, the display defect of FIG. 17A is eliminated.

  When the mask area is specified as in the present embodiment, the corrected partial areas overlap at the boundary between the corrected partial areas. However, the overlapped part has contents drawn later. Since the pixels are overwritten and originally located at substantially the same position in the pre-correction image, there is no sense of incongruity.

  FIG. 16 is a diagram showing a drawing command generated for each partial region by the processing of this embodiment. In the drawing command of FIG. 16, the setting for hiding the mask area of (4) of FIG. 15 in the first embodiment is changed, and the range drawn by drawing (display processing) of the image is an oblique boundary line. The mask area is designated so that a range expanded by about 0.5 pixels on the basis of 72 dpi is displayed in the direction.

  If it is known in advance that a display failure may occur depending on the type or display magnification of the PDF display program used in the computer that performs the display, the processing of this embodiment may be selected.

[Modification]
In the above, the PDF file is clearly shown as a format that can use only the affine transformation in the projective transformation. Of course, the format is not limited to the PDF file as long as it can use only the affine transformation in the projective transformation. For example, in many file formats that assume two-dimensional graphics drawing, only affine transformation can be used among projective transformations (for example, svg format that performs two-dimensional graphics drawing). It is effective to apply form processing.

  Further, the file generation mode executed by the mobile terminal 10 according to the present embodiment may be performed by, for example, a server device or a multifunction device. In this case, the drawing command generation unit 100 and the formatting processing unit 101 illustrated in FIG. 2 are provided in the server device or the multifunction peripheral. The user transmits a captured image captured by the mobile terminal 10 or the digital camera to the server device or the multifunction device. The server device or the multifunction device executes a file generation mode on the received captured image to generate a PDF file, and the PDF file is stored in a storage destination (such as a mobile terminal, various terminals, or a PDF file) designated by the user. Send. In this way, the file generation mode of this embodiment can be used even by a user of a portable terminal or a digital camera in which an application that executes the file generation mode is not installed. In particular, multi-function machines often do not have projection conversion hardware installed, and have the advantage of being applied to multi-function machines.

  In the above embodiment, the mobile terminal 10 creates an image file in a format that can use only affine transformation among projective transformations. However, the portable terminal 10 is not limited to affine transformation but has a format that can use projective transformation. An image file may be created. That is, in this case, the mobile terminal 10 distorts the captured image by projective transformation (not limited to affine transformation) when drawing (displaying) the image data of the captured image before distortion correction and the captured image of the image file. After the correction, an image file is created in which a drawing command for controlling the computer is displayed so that the captured image is displayed. And the portable terminal 10 transmits the said image file to the transmission destination (a server, another terminal, etc.) designated by the user according to a user instruction. Alternatively, in the portable terminal 10 having a function of creating an image file in a format that can use projective transformation, affine transformation is used when drawing (displaying) the image data of the photographed image before distortion correction and the photographed image of the image file. And a mode for creating an image file in which a drawing command for controlling the computer is displayed so as to display the photographed image after performing distortion correction on the photographed image, and the image file is transmitted to a user-specified destination (server, It may be configured to transmit to another terminal or the like. According to this configuration, when there are many photographed images to be processed or when multitasking is performed, there is an advantage that the processing load can be reduced. Further, the processing for creating the image file may be performed not by the mobile terminal 10 but by a server device or a personal computer that can communicate with the mobile terminal 10 or a personal computer (that is, by the mobile terminal 10). The captured image may be transmitted to a server device or a personal computer, and an image file may be created on the server device or the personal computer).

  Moreover, in the above embodiment, the portable terminal 10 performs drawing after performing distortion correction (affine transformation) when drawing a shot image that has not been subjected to distortion correction and a PDF file opened. As described above, the PDF file in which the drawing command for controlling the computer is embedded is created. However, the following modifications may be made. First, when the mobile terminal 10 shifts to the file generation mode, the mobile terminal 10 causes the user to select either the simple processing mode or the normal processing mode. When the simple mode is selected, as in the present embodiment, the mobile terminal 10 performs distortion correction (affine transformation) when a captured image that has not been subjected to distortion correction and when a PDF file is opened to draw a captured image. A PDF file in which a drawing command for controlling the computer is embedded so as to draw is created. On the other hand, when the normal processing mode is selected, the mobile terminal 10 performs distortion correction on the captured image using not only affine transformation but also projective transformation, and a PDF file in which the captured image after distortion correction is embedded. Create In this way, even if the file creation process takes a long time, a user's request for a file in which a captured image whose distortion has been corrected is embedded, and a user's request that does not like the file creation process taking a long time are requested. Can be satisfied.

  The information processing unit 11 of the mobile terminal 10 includes a character recognition unit that performs character recognition processing on the captured image, and a transparent text processing unit that creates transparent text based on the result of the character recognition processing, and is formatted. The processing unit 101 may create an image file by embedding transparent text in addition to the drawing command.

  In the first and second embodiments, the correction target range 60 (or the divided range 60a) and the corrected display range 70 (or the corrected divided range 70a) are divided by the method described with reference to FIGS. However, the division method is not limited to the method described with reference to FIGS. In short, as long as the pre-division square is divided into a plurality of post-division squares smaller than the pre-division square, there is no limitation on the division method itself. For example, the corrected display range 70 and the correction target range 60 may be divided into two instead of four. Further, the correction target range 60 may be divided before the post-correction display range 70 is divided. In this case, first, a dividing line that passes through the midpoint of the upper and lower sides of the correction target range 60 and a dividing line that passes through the midpoint of the left and right sides of the correction target range 60 are set. The correction target range 60 is divided into four by these dividing lines. Subsequently, the four midpoints obtained in the process of dividing the correction target range 60 into four and the intersections of the dividing lines are converted with a distortion correction matrix for projectively converting the correction target range 60 into the corrected display range 70, The corrected display range 70 is divided into four by a line segment connecting these points.

[Example of software implementation]
The information processing unit 11 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software using a CPU (Central Processing Unit).

  In the latter case, the information processing unit 11 includes a CPU that executes instructions of a program that is software that implements each function, and a ROM (Read Only Memory) in which the program and various data are recorded so as to be readable by a computer (or CPU). Alternatively, a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) that expands the program, and the like are provided. And the objective of this invention is achieved when a computer (or CPU) reads the said program from the said recording medium and runs it. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. The present invention can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[Summary]
An image processing apparatus (portable terminal 10) according to aspect 1 of the present invention displays a format processing unit 101 that generates an image file in which image data of a captured image is formatted in a predetermined format, and the captured image of the image file. A command adding unit (drawing command generating unit 100) for creating a drawing command for controlling the computer to perform the display after performing distortion correction, and adding the drawing command to the image file. The adding unit sets first correction information indicating a difference between a case where the correction target range is set for the photographed image and the correction target range is subjected to distortion correction by projective transformation and the case where the distortion correction is performed by affine transformation. When the difference is within the allowable range, the drawing command for performing the distortion correction using an affine transformation matrix determined from the correction target range is obtained. And when the difference is not within an allowable range, the correction target range is divided to set a plurality of partial areas, and the drawing command for performing the distortion correction by an affine transformation matrix determined from the partial areas is set to the partial area It is created every time.

  According to the configuration of the present invention, when distortion correction is performed on a correction target range using a single affine transformation matrix, a correction error (in contrast to a case where correction is performed by projective transformation without being limited to affine transformation). If the projection transformation is limited to the affine transformation and the correction is large, the correction target range is divided into a plurality of partial areas, and an affine transformation matrix is set for each partial area. Thus, it is possible to realize distortion correction with reduced correction error (by dividing the corresponding range in one distortion correction matrix to reduce the size of the corresponding range in one distortion correction matrix. The correction error can be reduced and the correction error can be suppressed as a whole). Therefore, when generating an image file in which distortion correction is performed by affine transformation when displaying an image, there is an effect that it is possible to generate a PDF file in which distortion correction is appropriately performed.

  In addition to the configuration of aspect 1, the image processing apparatus according to aspect 2 of the present invention divides the correction target range into a plurality of quadrangular division ranges when the difference is not within an allowable range, Each division range is divided into triangles facing each other across one diagonal line in each division range, and the triangle is used as the partial region.

  If the affine transformation is applied to the quadrangular division range, there is a problem that distortion increases in the vicinity of the boundary between the division ranges. However, as in the aspect 2 of the present invention, the quadrangular division range is further divided into triangles. By performing affine transformation using each triangle as a partial region, there is an effect that the above problems can be suppressed.

  In the image processing apparatus according to aspect 3 of the present invention, in addition to the configuration of aspect 2, when the command adding unit corrects the distortion of the divided range by projective transformation and generates the distortion by affine transformation. The second difference information indicating the difference from the correction case is obtained, and the process of dividing the divided range where the difference indicated by the second difference information is not within the allowable range and generating the divided range is performed for all the divided ranges. It repeats until the difference of the second difference information falls within an allowable range.

  According to the aspect 3 of the present invention, the correction error of the corresponding range (partial region) in one distortion correction matrix can be suppressed within a predetermined allowable range (threshold), and in all the partial regions In order to suppress the correction error within an allowable range, there is an effect that distortion correction can be performed with high accuracy.

  In addition to aspect 2, the image processing apparatus according to aspect 4 of the present invention is configured such that the command adding unit sets the number of the divided ranges in accordance with the difference in the first difference information. The upper limit of the number of ranges is defined.

  If the number of the divided ranges (that is, the number of partial areas) is too large, there is a problem that processing delay occurs when displaying the image of the image file. According to the aspect 4 of the present invention, Since the upper limit of the number is determined, the problem that the number of division ranges becomes too large can be suppressed.

  In addition to any one of the first to fourth aspects, the image processing apparatus according to the fifth aspect of the present invention is configured such that the first difference information includes a barycentric position of a target range after distortion correction set for the correction target range; The center of gravity position of the correction target range is a value corresponding to a distance from the position after being moved by the projective transformation.

  According to the fifth aspect of the present invention, a numerical value indicating a difference (correction error) between when the correction target range is distortion-corrected by projective transformation and when the distortion is corrected by affine transformation is easily used as the first difference information. Can be requested.

  In addition to aspect 3, the image processing apparatus according to aspect 6 of the present invention is configured such that the second difference information includes a centroid position of a target range after distortion correction set for the division range and a centroid position of the division range. It is a value according to the distance from the position after being moved by the projective transformation.

  According to the sixth aspect of the present invention, a numerical value indicating a difference (correction error) between the case where the division range is subjected to distortion correction by projective transformation and the case where the distortion correction is performed by affine transformation is easily used as the second difference information. Can be sought.

  In the image processing apparatus according to aspect 7 of the present invention, in addition to aspects 3 and 6, when the command adding unit divides a divided range in which the difference in the second difference information is not within an allowable range, the total number of divided ranges is defined. If the value exceeds the value, before performing the division, (1) the first option for performing the division even if the total number of division ranges exceeds the specified value; and (2) canceling the division and calculating the total number of the division ranges. The user is allowed to select one of a second option to be kept within a specified value and (3) a third option to stop the generation of the image file.

  As described above, when the number of the divided ranges (that is, the number of partial areas) is too large, there is a problem that processing delay occurs when displaying the image of the image file. On the other hand, according to aspect 7 of the present invention, priority is given to the accuracy of correction by performing division even when the total number of division ranges exceeds the specified value, or the total number of division ranges is suppressed to the specified value. The user can select whether to give priority to the display speed or to cancel the image file generation itself.

  In an image processing apparatus according to an eighth aspect of the present invention, in addition to the first to seventh aspects, when the difference indicated by the first difference information is not within an allowable range, the drawing command generated for the partial region is Distortion correction using an affine transformation matrix set from a partial area is performed on the image data, and display processing is performed with the outside of the corrected target partial area set for the partial area as a mask area. A command for controlling a computer, wherein the mask area is set so as to display a range that includes the corrected target partial area and is wider than the corrected target partial area.

  According to the image processing device of aspect 8 of the present invention, when the image file is opened and displayed, a display defect that a gap is generated at the boundary between the corrected target partial area and the corrected target partial area that are adjacent to each other is suppressed. There is an effect that can be done.

  According to the ninth aspect of the present invention, an image file obtained by formatting image data of a captured image into a predetermined format is generated, and the display is performed after distortion correction is performed when the captured image of the image file is displayed. An image processing method for creating a drawing command for controlling a computer and adding the drawing command to the image file, wherein a correction target range is set for the captured image, and the correction target range is subjected to distortion correction by projective transformation. First difference information indicating a difference between the case and the case of correcting the distortion by affine transformation is obtained, and when the difference is within an allowable range, the distortion correction is performed using an affine transformation matrix determined from the correction target range. When the drawing command is created and the difference is not within the allowable range, the correction target range is divided to set a plurality of partial areas, and an area defined from the partial areas is set. The fin conversion matrix, characterized in that to create the drawing command for the distortion correction for each of the partial regions.

  According to the ninth aspect of the present invention, similarly to the configurations of the first to eighth aspects, when generating an image file in which distortion correction is performed by affine transformation when an image is displayed, distortion correction is appropriately performed. The effect is that a PDF file can be generated.

  The image processing apparatus according to aspects 1 to 8 of the present invention may be realized by a computer, and in this case, a program for causing the computer to realize the image processing apparatus by causing the computer to operate as each unit of the image processing apparatus. And a computer-readable recording medium on which it is recorded also fall within the scope of the present invention.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

  The present invention can be applied to an image processing apparatus that generates an image file in which image data obtained by photographing with a digital camera or the like is formatted in a predetermined format. Examples of the image processing device include a mobile terminal (smart phone, tablet, node type personal computer, mobile phone, etc.) equipped with a digital camera and a photographing device. Examples of the image processing apparatus include a computer, a multi-function peripheral, a server apparatus, and the like that can process a captured image received from a digital camera or a portable terminal.

10 Mobile terminal (image processing device)
DESCRIPTION OF SYMBOLS 11 Information processing part 12 Touch panel 60 Correction object range 60a Division range 60a1-60a4 Division range 60aa Partial region 60ab Partial region 70 After-correction display range 70a After-correction division range 70aa After-correction partial region (after-correction target partial region)
70ab partial area after correction (target partial area after correction)
100 Drawing command generation part (command addition part)
101 Formatting processor

Claims (11)

A formatting processing unit that generates an image file in which image data of a captured image is formatted into a predetermined format;
A command addition unit for creating a drawing command for controlling the computer to perform the display after performing distortion correction when displaying the captured image of the image file, and adding the drawing command to the image file; Prepared,
The command adding unit sets a correction target range for the photographed image, and shows a difference between a case where the correction target range is subjected to distortion correction by projective transformation and a case where the distortion correction is performed by affine transformation. Seeking information,
If the difference is within an allowable range, create the drawing command to perform the distortion correction by an affine transformation matrix determined from the correction target range,
When the difference is not within the allowable range, the correction target range is divided to set a plurality of partial areas, and the drawing command for performing the distortion correction is generated for each partial area by an affine transformation matrix determined from the partial areas An image processing apparatus.   When the difference is not within the allowable range, the command adding unit divides the correction target range into a plurality of quadrangular divided ranges, and each divided range is opposed to each other with a diagonal line interposed therebetween. The image processing apparatus according to claim 1, wherein the image processing apparatus is divided into triangles, and the triangle is used as the partial region.   The command adding unit obtains second difference information indicating a difference between a case where the division range is corrected by the projective transformation and a case where the distortion correction is performed by the affine transformation. It repeats the process of dividing a divided range in which the difference indicated in the difference information is not within the allowable range and further generating a divided range until the difference of the second difference information is within the allowable range in all the divided ranges. The image processing apparatus according to claim 2. The command adding unit is configured to set the number of the divided ranges according to the difference of the first difference information,
The image processing apparatus according to claim 2, wherein an upper limit of the number of the divided ranges is set.   The first difference information is a distance between a centroid position of a target range after distortion correction set for the correction target range and a position after moving the centroid position of the correction target range by the projective transformation. The image processing apparatus according to claim 1, wherein the image processing apparatus has a value corresponding to the value.   The second difference information corresponds to a distance between a centroid position of a target range after distortion correction set for the divided range and a position after the centroid position of the divided range is moved by the projective transformation. The image processing apparatus according to claim 3, wherein the image processing apparatus is a value.   When dividing the divided range in which the difference of the second difference information is not within the allowable range when the total number of divided ranges exceeds a specified value, the command adding unit (1) before dividing, (1) the total number of divided ranges A first option for performing the division even if the value exceeds a specified value; (2) a second option for stopping the division and keeping the total number of the divided ranges within a specified value; and (3) generating the image file. The image processing apparatus according to claim 3, wherein the user is allowed to select one of the third options to be canceled. When the difference indicated by the first difference information is not within an allowable range, the drawing command generated for the partial area performs distortion correction on the image data using an affine transformation matrix set from the partial area. And a command for controlling the computer to perform display processing on the outside of the corrected target partial area corresponding to the partial area as a mask area,
8. The mask area according to claim 1, wherein the mask area is set so as to display a range that includes the corrected target partial area and is wider than the corrected target partial area. Image processing apparatus. Generate an image file in which the image data of the shot image is formatted into a predetermined format,
An image processing method for creating a drawing command for controlling a computer to perform the display after performing distortion correction when displaying the captured image of the image file, and adding the drawing command to the image file. ,
A correction target range is set for the captured image, and first difference information indicating a difference between a case where the correction target range is subjected to distortion correction by projective transformation and a case where the distortion correction is performed by affine transformation is obtained,
If the difference is within an allowable range, create the drawing command to perform the distortion correction by an affine transformation matrix determined from the correction target range,
When the difference is not within the allowable range, the correction target range is divided to set a plurality of partial areas, and the drawing command for performing the distortion correction is generated for each partial area by an affine transformation matrix determined from the partial areas Image processing method.   A program that causes a computer to function as each unit of the image processing apparatus according to claim 1.   The computer-readable recording medium which recorded the program of Claim 10.