JP2012134662A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of calculating a distortion aberration coefficient corresponding to the lens and a zooming rate in imaging.SOLUTION: An imaging device 10 comprises: an imaging part 20 which has a lens for imaging and is capable of imaging using the lens; and a distortion aberration coefficient calculation part 50 which extracts feature points from plural pieces of image information obtained by imaging in respectively different imaging attitudes by the imaging part 20, searches a feature point extracted from one piece of image information and a feature point extracted from another piece of image information for a set of feature points indicating the same point on a subject, estimates an imaging attitude in imaging of another piece of image information mentioned above, on the basis of the set of the feature points, and calculates the distortion aberration coefficient of the lens on the basis of the estimated imaging attitude.

Description

  The present invention relates to a photographing apparatus.

  Distortion occurs in an image taken using a lens. In a camera using a chemical film, it has been difficult to correct such distortion, but in a digital still camera that has been widespread in recent years, for example, as described in Patent Documents 1 and 2 In addition, it has become possible to correct distortion.

  Here, the correction of distortion aberration of an image acquired by a digital still camera has two stages: acquisition of aberration information and image geometric conversion.

[Acquisition of aberration information]
The coordinates of each point for the optical imaging point (actual projection point) on the imaging surface (on the surface of the imaging device) and the imaging point (true projection point) when it is assumed that there is no distortion Aberration information represents the correspondence between the two.

  Although this aberration information can be expressed in various forms, an expression using a distance (image height) r from the projection center (intersection of the optical axis and the surface of the image sensor) is often used.

Assume that the actual projection point is (x _r , y _r ), the true projection point is (x _i , y _i ), and the coordinates of the projection center are (x _c , y _c ). At this time, the true image height r _i and the actual image height r _r are expressed by Equation 1.

…（１）

... (1)

Here, the relationship between r _i and r _r is expressed by Equation 2 using distortion coefficients (distortion aberration coefficients) h ₁ and h ₂ .

…（２）

... (2)

In consideration of measurement errors, the fifth order is necessary and sufficient. Even order terms are not used.
[Geometric transformation of image]
As an image geometric transformation, a method of interpolating corresponding points and a method of rectangular transformation are widely used.

  In the method of interpolating corresponding points, a true projection point and an actual projection point are made to correspond one-to-one with respect to an arbitrary point in the image, and the pixel value of the actual projection point is interpolated. Although the accuracy is high, the amount of calculation is relatively large.

  The rectangle conversion method divides an image into a plurality of rectangular areas and creates an output image by rectangular conversion. Although the accuracy is slightly reduced, the amount of calculation is relatively small.

JP 2009-290863 A International Publication No. 2008 / 044591A1 JP-A-6-141237

  By the way, the amount of distortion differs for each lens and zoom ratio. Therefore, in order to correct distortion aberration accurately, it is necessary to acquire a distortion aberration coefficient corresponding to the lens and zoom ratio at the time of shooting. In this regard, Patent Documents 1 and 2 disclose a technique for acquiring a distortion aberration coefficient registered (stored) in advance in a lens unit. However, the techniques disclosed in Patent Documents 1 and 2 cannot acquire the distortion aberration coefficient when the distortion aberration coefficient is not registered in the lens unit.

  Patent Document 3 discloses a technique for acquiring images using two imaging devices and performing geometric transformation on overlapping regions of these images. However, this technique performs geometric transformation so that images are not doubled in the overlapping region, and does not correct distortion.

  Therefore, the conventional technique has a problem that it may not be possible to acquire a distortion coefficient corresponding to a lens and a zoom ratio at the time of shooting.

  Accordingly, the present invention has been made in view of the above problems, and an object of the present invention is to obtain a distortion aberration coefficient corresponding to a lens and a zoom ratio at the time of shooting more reliably than in the past. It is another object of the present invention to provide a new and improved photographing apparatus.

  In order to solve the above-described problem, according to a certain aspect of the present invention, it is obtained by performing photographing with different photographing postures by photographing parts having photographing lenses and capable of photographing using the lenses, A feature point extraction unit that extracts feature points from each of a plurality of pieces of image information in which some areas overlap each other, a first feature point extracted from one image information, and a first feature point extracted from other image information Based on the two feature points, a feature point correlation unit that searches for a set of feature points indicating the same point on the subject, and a shooting posture when other image information is shot is estimated based on the set of feature points An imaging apparatus is provided, comprising: an attitude estimation unit; and an aberration coefficient calculation unit that calculates a distortion aberration coefficient of the lens based on the imaging attitude estimated by the attitude estimation unit.

  The photographing apparatus according to this viewpoint can estimate the photographing posture based on the image information and calculate the distortion aberration coefficient regardless of the lens and zoom ratio at the time of photographing. Therefore, the photographing apparatus can more reliably calculate the distortion aberration coefficient corresponding to the lens and zoom ratio at the time of photographing than before. In addition, since the photographing apparatus can calculate the distortion aberration coefficient when the photographer photographs an actual landscape, the distortion aberration coefficient can be calculated without using a chart or sample whose geometric information is known. .

  Here, the feature point correlation unit includes a predetermined number or more of the first feature points existing in the overlapping region of the one image information and the other image information to form a set of feature points. In the case, it is determined that the one image information and the other image information are correlated, and the posture estimation unit determines that the one image information and the other image information are correlated based on the set of feature points. The shooting posture when other image information is shot may be estimated. The photographing apparatus according to this aspect estimates the photographing posture when one piece of image information and other image information are correlated, so that the photographing posture can be estimated more reliably.

  In addition, the posture estimation unit performs singular value decomposition on the left side of the following expression (3) to obtain a product U · S · V of the matrices U and V and the diagonal matrix S having singular values as diagonal components. The posture matrix R represented by the following expression (4) may be calculated by converting the column components of the matrix V corresponding to the column of the smallest singular value to a00 to a22.

…（３）

... (3)

Here, n is a natural number, i1 _n and j1 _n are the x and y coordinates of the feature points extracted from the one image information, and i2 _n and j2 _n are extracted from the other image information. These are the x coordinate and y coordinate of the feature point.

…（４）

... (4)

  Since the photographing apparatus according to this aspect calculates the photographing posture as the posture matrix R, the photographing posture can be estimated more reliably.

The aberration coefficient calculation unit calculates the true image height of the feature point extracted from one image information based on the posture matrix R, and the coordinates (i1 _n , j1 _n ) to calculate the actual image height of the feature point extracted from one image information, and based on the calculated true image height and actual image height, and the following equation (5) The distortion aberration coefficients h ₁ and h ₂ may be calculated.

…（５）

... (5)

Even if the lens and the zoom ratio are changed, the photographing apparatus according to this aspect can estimate the photographing posture based on the image information obtained after the change and calculate the distortion aberration coefficients h ₁ and h ₂ .

  Further, a lens information acquisition unit that acquires identification information for identifying a lens, zoom rate information regarding a zoom rate when the imaging unit captures the subject, information acquired by the lens information acquisition unit, and an aberration coefficient An aberration information storage unit that stores the distortion aberration coefficient calculated by the calculation unit in comparison with each other may be provided.

  The photographing apparatus according to this aspect can easily determine under what conditions each distortion aberration coefficient stored in the aberration information storage unit is calculated.

  An aberration correction unit that searches the aberration information storage unit for a distortion aberration coefficient corresponding to the information acquired by the lens information acquisition unit, and corrects the image information acquired by the imaging unit based on the searched distortion aberration coefficient; You may make it provide.

  Since the photographing apparatus according to this aspect can perform correction using a distortion aberration coefficient corresponding to a currently used lens and zoom ratio, correction can be performed more accurately.

  As described above, the photographing apparatus according to the present invention can more reliably obtain the distortion aberration coefficient corresponding to the lens and the zoom ratio at the time of photographing than before.

It is a block diagram which shows the structure of the imaging device which concerns on embodiment of this invention. It is a block diagram which shows the structure of the distortion aberration coefficient calculation part which concerns on the same embodiment. It is a flowchart which shows the procedure of the process by an imaging device. It is a flowchart which shows the procedure of the process by an imaging device. It is explanatory drawing which shows the guide frame displayed on a display part. It is explanatory drawing which shows the image obtained when image | photographing an actual landscape and this landscape. It is explanatory drawing which shows the image which the imaging | photography person image | photographed. It is explanatory drawing which shows the extracted feature point. It is explanatory drawing which shows the imaging | photography attitude | position of an imaging device. It is explanatory drawing which shows matching of a feature point. It is explanatory drawing which shows a response | compatibility of the coordinate point by distortion aberration correction. It is explanatory drawing which shows the concept of pixel interpolation. It is explanatory drawing which shows the kind of distortion aberration. It is a graph which shows the relationship between the distance from a projection center, and a distortion ratio for every kind of distortion aberration.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[Configuration of Shooting Device 10]
First, a schematic configuration of the photographing apparatus 10 will be described with reference to FIG. The photographing apparatus 10 is a so-called digital still camera, and includes a photographing unit 20, a lens information acquisition unit 30, a camera control unit 40, a distortion aberration coefficient calculation unit 50, an aberration information storage unit 60, and an aberration correction unit 70. And an image storage unit 80 and a display unit 90. Note that the photographing apparatus 10 includes hardware such as a CPU, a ROM, and a RAM. When the CPU reads and executes a program stored in the ROM, the processing by each functional block is realized.

  The photographing unit 20 includes a lens unit 21, an image sensor 22, and an image processing unit 23. The lens unit 21 is detachable from the photographing apparatus 10, and includes one or a plurality of lenses L (see FIG. 9), an adjustment mechanism that adjusts the zoom ratio, focus, aperture, and shutter speed, and the lens unit. A storage unit that stores various information about the lens unit 21, and a lens unit control unit that controls the lens unit 21 such as control of the adjustment mechanism and output of information stored in the storage unit. The information stored in the storage unit may be, for example, lens ID (identification information) for identifying the lens L, shooting information regarding the current zoom rate (that is, at the time of shooting), focus, aperture, and shutter speed. It is done. The lens unit 21 captures light outside the photographing apparatus 10 and causes the captured light to enter the image sensor 22 via the lens L.

  The image sensor 22 generates an electrical signal (Raw data) corresponding to the light incident on the image sensor 22 and outputs it to the image processing unit 23.

  The image processing unit 23 converts the electrical signal given from the image sensor 22 into image information. In this image information, coordinates and pixel values (indicating the color and brightness of the pixels) are set for each pixel. Here, the coordinates of each pixel are defined on the image sensor 22 and given as coordinates on the xy plane with the projection center as the origin. Both the x coordinate and the y coordinate are integers. The projection center is a point where the optical axis of the lens L intersects the image sensor 22. Since distortion is generated by the lens of the lens unit 21, image distortion due to distortion occurs in the image information generated by the image processing unit 23.

  The image processing unit 23 outputs the image information to the distortion aberration coefficient calculation unit 50 and the aberration correction unit 70. Further, the image processing unit 23 displays image information on the display unit 90.

  The lens information acquisition unit 30 acquires the lens ID and shooting information from the lens unit 21 and outputs them to the camera control unit 40.

  The camera control unit 40 controls each component of the photographing apparatus 10. In addition, the camera control unit 40 outputs the lens ID and the photographing information given from the lens information acquisition unit 30 to the aberration information storage unit 60 and the aberration correction unit 70.

  The distortion aberration coefficient calculation unit 50 calculates a distortion aberration coefficient necessary for correcting the distortion aberration based on the image information given from the image processing unit 23, and stores aberration information related to the distortion aberration coefficient in the aberration information storage unit 60. Output.

  The aberration information storage unit 60 stores the lens ID and shooting information given from the camera control unit 40 and the aberration information given from the distortion aberration coefficient calculation unit 50 in association with each other. That is, since the amount of distortion varies depending on the lens and zoom ratio at the time of shooting, the value of the distortion aberration coefficient necessary for correcting distortion also varies depending on the lens and zoom ratio at the time of shooting. Therefore, in the present embodiment, the aberration information storage unit 60 stores the lens (that is, the lens ID) and the zoom rate at the time of shooting in association with each other and the distortion aberration coefficient.

  The aberration correction unit 70 searches the aberration information storage unit 60 for aberration information corresponding to the lens ID and the shooting information given from the camera control unit 40. Then, the aberration correction unit 70 corrects the image information given from the image processing unit 23 based on the searched aberration information. The aberration correction unit 70 outputs the corrected image information to the image storage unit 80. The image storage unit 80 stores the image information given from the aberration correction unit 70.

  The display unit 90 has a display 100 (see FIG. 5) and displays various images on the display 100.

[Configuration of Distortion Aberration Coefficient Calculation Unit 50]
Next, the configuration of the distortion aberration coefficient calculation unit 50 will be described with reference to FIG. The distortion aberration coefficient calculation unit 50 includes a feature point extraction unit 51, a feature point correlation unit 52, a posture estimation unit 53, and an aberration coefficient calculation unit 54.

  The feature point extraction unit 51 extracts feature points from the image information given from the image processing unit 23. Here, the feature point constitutes the contour of the subject among the pixels constituting the image information. The technique for extracting feature points is not particularly limited, and a known technique is arbitrarily applied. The feature point extraction unit 51 generates feature point extraction information related to the image information from which the feature point coordinates, pixel values, and feature points are extracted, and outputs the feature point extraction information to the feature point correlation unit 52.

  The feature point correlator 52 receives the feature point extraction information corresponding to the one image information and the feature point extraction information corresponding to the other image information from the feature point extraction unit 51, from the one image information. A set of feature points indicating the same point on the subject is searched for from the extracted first feature points and second feature points extracted from other image information. Here, as will be described later, one image information and another image information partially overlap.

  The feature point correlation unit 52 has a predetermined number or more (for example, 80 of the total number of first feature points existing in the overlapping region) among the first feature points existing in the overlapping region of one image information and other image information. %) Of the first feature points constitute a set of feature points, it is determined that these pieces of image information are correlated. When one image information and other image information correlate, the feature point correlation unit 52 generates feature point correlation information related to the searched feature point set, and the posture estimation unit 53 and the aberration coefficient calculation unit 54. Output to.

  The posture estimation unit 53 calculates a posture matrix R represented by the following equation (4) based on the feature point correlation information given from the feature point correlation unit 52.

…（４）

... (4)

  Next, the posture estimation unit 53 outputs the posture matrix information regarding the calculated posture matrix R to the aberration coefficient calculation unit 54.

The aberration coefficient calculation unit 54 calculates the distortion aberration coefficients h ₁ and h ₂ based on the information given from the feature point correlation unit 52 and the posture estimation unit 53. The aberration coefficient calculation unit 54 generates aberration information related to the calculated distortion aberration coefficient and outputs the aberration information to the aberration information storage unit 60.

[Distortion coefficient calculation processing]
Next, the distortion aberration coefficient calculation process performed by the imaging apparatus 10 will be described with reference to the flowchart shown in FIG. Note that the timing for performing the distortion aberration coefficient calculation processing is, for example, when the photographing apparatus 10 is turned on, when the lens unit 21 is replaced, or when a correction instruction is issued from the photographer (in this case, such designation is performed). It is conceivable that a switch for performing the operation is provided separately in the photographing apparatus 10). It is assumed that the photographer fixes the zoom rate and does not replace the lens unit 21 while the distortion aberration coefficient calculation process is performed.

  In step S <b> 10 illustrated in FIG. 3, the image processing unit 23 displays guide frames 101 to 105 illustrated in FIG. 5A on the display screen 100. The guide frames 101 to 105 are numbered, and these numbers mean the order of the guide frames. That is, the guide frames 101 to 105 become the first, second,..., 55th guide frames in order from the guide frame 101.

  The upper right quarter of the guide frame 101 overlaps with the lower left quarter of the guide frame 102, and the lower right quarter of the guide frame 101 is the upper left quarter of the guide frame 103. It overlaps with the area. Similarly, the guide frame 101 and other guide frames 104 and 105 partially overlap each other. As described above, the guide frame 101 and the other guide frames overlap each other by 1/4 (25%) of their regions, but preferably overlap at least 20% or more of the regions. Further, the number of guide frames is not limited to five.

  On the other hand, the lens unit 21 constantly takes in light from the outside and makes it incident on the image sensor 22, and the image sensor 22 outputs an electrical signal to the image processing unit 23. The image processing unit 23 generates image information based on the electrical signal.

  In step S20, the image processing unit 23 activates the first guide frame 101 and displays the reduced image information on the guide frame 101 as shown in FIG. As a result, the image processing unit 23 displays a live view, that is, a current shooting target landscape on the guide frame 101. For example, when the landscape shown in FIG. 6A (including house A1, trees A2 to A4, buildings A5 to A7, and factory A8) is to be photographed, the image processing unit 23 displays a part of this landscape live. Display as a view.

  In step S <b> 30, the image processing unit 23 stands by until there is a photographing instruction (that is, pressing of the shutter button) from the photographer. On the other hand, the photographer issues a shooting instruction with reference to the live view displayed on the guide frame 101 (that is, performs shooting). The image processing unit 23 proceeds to step S40 when there is a photographing instruction from the operator.

  In step S <b> 40, the image processing unit 23 sets the image information displayed on the guide frame 101 when there is a shooting instruction as the first image information, and outputs the first image information to the feature point extraction unit 51. For example, when the image processing unit 23 displays the image information P1 shown in FIG. 6B as a live view on the guide frame 101 and receives an imaging instruction from the operator, the image processing unit 23 displays the image information P1. The image information is the first image. As shown in FIG. 9, the image information P1 is acquired when the photographing unit 20 performs photographing with the photographing posture L1 and the viewpoint (the focal point of the lens L) B1. The shooting posture is indicated by, for example, the position and orientation of the lens L, and is specifically indicated by the posture matrix R.

  In step S50, the feature point extraction unit 51 extracts feature points from the first piece of image information. Here, the feature points are pixels constituting the contour of the subject as described above. For example, when the first piece of image information is the image information P1 shown in FIG. 7A, the feature point extraction unit 51, as shown in FIG. 8A, houses A1, trees A2, A3, and building A6. Feature points Q11 to Q12 that constitute the contours are extracted. Next, the feature point extraction unit 51 generates feature point extraction information regarding the extracted feature point coordinates, pixel values, and image information from which the feature points have been extracted, and outputs the feature point extraction information to the feature point correlation unit 52.

  In step S60, the image processing unit 23 activates the next guide frame. For example, when the guide frame 101 is active, the image processing unit 23 activates the next guide frame 102 as shown in FIG. 5B. Then, the image processing unit 23 displays the live view on the active guide frame as in step S20. At this time, the image information already fixed in the guide frame is continuously fixed in the guide frame. Further, an alpha blend process is performed by the image processing unit 23 on the overlapping area of the two pieces of image information (including live view). As a result, the overlapped area is displayed by superimposing two pieces of image information.

  In step S <b> 70, the image processing unit 23 stands by until there is a photographing instruction from the photographer. On the other hand, the photographer adjusts the photographing posture of the photographing apparatus 10 so that a part of the live view displayed in the active guide frame matches the first image information, and performs photographing in this state. For example, when the guide frame 102 becomes active, the shooting posture of the shooting apparatus 10 is adjusted so that the lower left 1/4 portion of the live view matches the upper right 1/4 portion of the first image information, Shooting is performed in this state. The image processing unit 23 proceeds to step S80 when there is a photographing instruction from the operator.

  In step S <b> 80, the image processing unit 23 outputs the image information displayed on the active guide frame when the shooting instruction is given as N-th image information to the feature point extraction unit 51. Here, N is an integer of 2 to 5, and indicates the order of the active guide frames.

  For example, when the active guide frame is the guide frame 102, N = 2. When the image processing unit 23 displays the image information P2-1 shown in FIG. 6B as a live view on the guide frame 102 and receives an imaging instruction from the operator, the image processing unit P2-1. Is acquired as the image information of the second sheet. The lower left 1/4 portion of the image information P2-1 matches the upper right 1/4 portion of the image information P1. That is, in the overlapping area P12 of the first image information P1 and the second image information P2-1, both images match. Note that the image information P2-1 is acquired when the photographing unit 20 performs photographing in the photographing posture L2 and the viewpoint B2, as illustrated in FIG.

  On the other hand, when the image processing unit 23 displays the image information P2-2 shown in FIG. 6C on the guide frame 102 as a live view, when there is a shooting instruction from the operator, the image information P2 -2 is acquired as the image information of the second sheet. The lower left 1/4 portion of the image information P2-2 is different from the upper right 1/4 portion of the image information P1. That is, in the overlapping area P12 of the first image information P1 and the second image information P2-2, both images are different.

  In step S90, the feature point extraction unit 51 extracts feature points from the N-th image information. For example, when the image information P2-1 shown in FIG. 7B is acquired as the second piece of image information (N = 2), the feature point extraction unit 51, as shown in FIG. A feature point Q21 constituting the contours of A1 and trees A2 and A3 is extracted. Next, the feature point extraction unit 51 generates feature point extraction information regarding the extracted feature point coordinates, pixel values, and image information from which the feature points have been extracted, and outputs the feature point extraction information to the feature point correlation unit 52. FIG. 7B shows a part of the image information 2-1.

  In step S100, the feature point correlation unit 52 converts the feature point extraction information corresponding to the first image information given from the feature point extraction unit 51 and the feature point extraction information corresponding to the Nth image information. Based on the first feature point extracted from the first image information and the second feature point extracted from the Nth image information, a set of feature points indicating the same point on the subject is searched. To do. The feature point correlator 52 has a predetermined number or more (for example, one) of the first feature points existing in the overlapping region of the first image information and the N-th image information among the first feature points. When the first feature points of 80% or more of the total number of the first feature points existing in the overlapping area of the image information of the eye and the image information of the Nth sheet constitute a set of feature points, these images It is determined that the information is correlated.

  For example, the first feature points Q11 and Q12 shown in FIG. 8A are extracted from the first piece of image information P1 shown in FIG. 7A, and the second (N = 2) shown in FIG. 7B. 8) is extracted from the image information P2-1 of FIG. 8B, all the first feature points Q11 and all the second feature points Q21 are one sheet. It exists in the overlapping area P12 of the image information of the eye and the image information of the second sheet. Further, all the first feature points Q11 and all the second feature points Q21 constitute the same point on the house A1 and the trees A2 and A3, and thus constitute a set of feature points. Accordingly, since all the first feature points Q11 constitute a set of feature points, the feature point correlation unit 52 determines that the first image information P1 and the second image information P2-1 are correlated. . FIG. 10 shows a straight line C connecting feature points that constitute a set of feature points.

  On the other hand, when the first image information is the image information P1 and the two images are the image information P2-2 shown in FIG. 6C, the feature points extracted from these image information are the feature points. Since the set is not configured, the feature point correlation unit 52 determines that these pieces of image information are not correlated.

  Next, when it is determined that the first image information and the Nth image information are correlated, the feature point correlation unit 52 proceeds to step S110, and when it is determined that these are not correlated, step S140 is performed. Proceed to

  In step S110, the feature point correlation unit 52 determines whether or not the correlation between the first image information and the fifth image information is obtained. If it is determined that these correlations are obtained, step S120 is performed. If it is determined that these correlations are not taken, the process returns to step S60.

  In step S120, the feature point correlation unit 52 generates feature point correlation information regarding a set of feature points obtained from each of the first image information and the second to fifth images, and the posture estimation unit 53 and the aberration It outputs to the coefficient calculation part 54.

  Next, the posture estimation unit 53 uses the posture matrix R described above for each combination of image information (a combination of the first image information and the second to fifth image information) based on the feature point correlation information. To calculate. For example, the posture estimation unit 53 calculates the posture matrix R corresponding to the combination of the first image information and the second image information as follows. In other words, the posture estimation unit 53 performs singular value decomposition on the left side of the following expression (3), whereby the product U · S · V of the matrices U and V and the diagonal matrix S having singular values as diagonal components. The posture matrix R is calculated by converting the column components of the matrix V corresponding to the column with the smallest singular value to a00 to a22.

…（３）

... (3)

Here, n is a natural number, i1 _n and j1 _n are the x coordinate and y coordinate of the feature point extracted from the first image information, and i2 _n and j2 _n are the second image information. X and y coordinates of feature points extracted from.

  Here, the derivation process of Expression (3) will be described. The posture matrix R is expressed by the following equation (6).

…（６）

(6)

  Here, w is a parameter and f is a focal length. Even if f is 1, generality is not lost. k is a natural number of 1 to n.

  When the parameter w is eliminated from the equation (6), the following equations (7) and (8) are obtained.

…（７）

... (7)

…（８）

(8)

  Assuming that the posture matrix R can be expressed as the above-described equation (4), the following equation (9) is obtained by expanding the equation (8).

…（９）

... (9)

  When Expression (9) is collected on the left side, Expression (10) is obtained.

…（１０）

(10)

  By substituting 1 to n for k in equation (10) and rewriting them in the form of a matrix, equation (3) is obtained.

  Note that the posture matrix R corresponding to other combinations of image information is also calculated in the same manner as described above. Therefore, a total of four attitude matrices R are calculated. Hereinafter, the posture matrix R corresponding to the combination of the first image information and the Nth image information is also referred to as an “Nth posture matrix R”.

  The posture estimation unit 53 outputs posture matrix information regarding the calculated posture matrix R to the aberration coefficient calculation unit 54.

In step S130, the aberration coefficient calculation unit 54 calculates distortion aberration coefficients h ₁ and h ₂ based on the information given from the feature point correlation unit 52 and the posture estimation unit 53. Hereinafter, a specific calculation method will be described.

  The aberration coefficient calculation unit 54 separates the projection distortion and the distortion aberration based on the feature point correlation information and the attitude matrix information, and calculates a distortion aberration coefficient. Specifically, the aberration coefficient calculation unit 54 first calculates the actual image height of the feature point extracted from the first image information. The actual image height is expressed by the following equation (11).

…（１２）

…（１３）
そして、収差係数算出部５４は、真の像高を以下の式（１４）に基づいて算出する。 Furthermore, the aberration coefficient calculation unit 54 calculates the true image height of the feature point existing in the upper right quarter of FIG. 5 of the first image information (the overlapping area with the second image information). Calculation is performed based on the second posture matrix R.
First, the aberration coefficient calculation unit 54 calculates the true coordinates (coordinates when no distortion occurs) (i2 ′ _k , j2 ′ _k ) of the feature points based on the following equations (12) and (13). To do.

(12)

... (13)
Then, the aberration coefficient calculator 54 calculates the true image height based on the following equation (14).

  Similarly, the aberration coefficient calculation unit 54 calculates the true image height of the feature point existing in the other region of the first image information based on the third to fifth posture matrix R. Thereby, the aberration coefficient calculation unit 54 calculates the true image height and the actual image height for all feature points of the first piece of image information.

Next, the aberration coefficient calculator 54 calculates the distortion aberration coefficient h ₁ based on the true and actual image heights calculated for all the feature points of the first piece of image information and the following equation (5). , H ₂ is calculated.

…（５）

... (5)

  Here, the first term on the right side of Equation (5) represents the Moore Penrose generalized inverse matrix. Equation (5) is derived as follows. That is, the actual image height and the true image height satisfy the relationship of the following formula (15).

…（１５）

... (15)

  When this is made into the form of a matrix, the following equation (16) is obtained.

…（１６）

... (16)

  When 1 to n is substituted for k and written down, the following expression (17) is obtained.

…（１７）

... (17)

When this is transformed using the Moore-Penrose generalized inverse matrix, Equation (5) is obtained.
The aberration coefficient calculation unit 54 generates aberration information related to the calculated distortion aberration coefficient and outputs the aberration information to the aberration information storage unit 60.

  On the other hand, the lens information acquisition unit 30 acquires the lens ID and shooting information from the lens unit 21 and outputs them to the camera control unit 40. The camera control unit 40 outputs the lens ID and imaging information given from the lens information acquisition unit 30 to the aberration information storage unit 60. Next, the aberration information storage unit 60 stores the lens ID and shooting information given from the camera control unit 40 and the aberration information given from the distortion aberration coefficient calculation unit 50 in association with each other. Information stored in the aberration information storage unit 60 is referred to by the aberration correction unit 70. Details will be described later.

  In step S140, the feature point correlation unit 52 notifies the image processing unit 23 that the N-th image information cannot be correlated with the first image information, and the image processing unit 23 determines that the active guide frame is active. To highlight. As such highlighting, for example, a guide frame is displayed with a thick line, a guide frame is blinked, and a guide frame is colored with red (usually black). This prompts the user to shoot again. An audio guide may be used in combination. As the sound at this time, for example, “please shoot so that the overlapping parts of the images match” can be considered.

[Distortion correction processing]
Next, the distortion correction process performed by the photographing apparatus 10 will be described with reference to FIG. In step S <b> 200, the lens unit 21 always takes in external light and makes it incident on the image sensor 22, and the image sensor 22 outputs an electrical signal to the image processing unit 23. The image processing unit 23 generates image information based on the electrical signal. The image processing unit 23 displays the generated image information on the display screen 100. Thereby, the live view is displayed on the display screen 100. In the distortion correction process, the guide frames 101 to 105 are not displayed, and the live view is displayed on the entire display screen 100. In addition, the photographer can perform shooting while arbitrarily setting shooting conditions such as a zoom ratio and focus. When a photographing instruction is issued from the photographer, the image processing unit 23 outputs image information at the time of photographing (when the photographing instruction is given) to the aberration correcting unit 70.

  On the other hand, the lens information acquisition unit 30 acquires the lens ID and shooting information from the lens unit 21 and outputs them to the camera control unit 40. The camera control unit 40 outputs the lens ID and shooting information given from the lens information acquisition unit 30 to the aberration correction unit 70.

  In step S <b> 210, the aberration correction unit 70 searches the aberration information storage unit 60 for aberration information corresponding to the lens ID and the imaging information given from the camera control unit 40. As a result, the aberration correction unit 70 determines whether or not such aberration information has been searched. If the aberration information can be searched, the process proceeds to step S220. The image information is output to the image storage unit 80, and the process proceeds to step S230.

  In step S220, the aberration correction unit 70 corrects the image information given from the image processing unit 23 based on the searched aberration information. Specifically, the aberration correction unit 70 performs correction as follows.

First, the aberration correction unit 70 defines corrected pixel coordinates (i _n , j _n ) as corrected image information. Next, the aberration correction unit 70 calculates which pixel in the image information before correction (image information given from the image processing unit 23) the corrected pixel corresponds to based on the following equation (18). .

…（１８）

... (18)

Here, x _n is the x coordinate of the pixels constituting the image information before the correction, which is the y coordinate of the pixel y _n is constituting the image information before the correction. _xc is the x coordinate of the projection center after correction (same as before correction), and _yc is the y coordinate of the projection center after correction (same as before correction). f (r) is a distortion ratio, that is, (actual image height) / (true image height). The true image height is obtained from Equation (14) described above. The actual image height can be obtained by substituting the true image height into the above equation (15). FIG. 11 conceptually shows the correspondence between the pixel E1 on the corrected image information Pi and the pixel E2 on the uncorrected image information Pr.

Next, the aberration correction unit 70 determines the pixel value of each pixel constituting the corrected image information, that is, the corrected pixel value. The coordinates (x _n , y _n ) calculated by the method described above are given as real values. On the other hand, the coordinates of each pixel constituting the image information before correction are given as integer values. For this reason, the pixel value ν (x _n , y _n ) is not given to the pixel represented by the coordinates (x _n , y _n ). Therefore, the aberration correction unit 70 calculates the pixel value ν (x _n , y _n ) by pixel interpolation, and sets this as the corrected pixel value ν (i _n , j _n ). Specifically, the aberration correction unit 70 calculates the pixel value ν (x _n , y _n ) based on the following equation (19).

…（１９）

... (19)

Here, [x _n ] is a maximum integer not exceeding x _n , and [y _n ] is a maximum integer not exceeding y _n . p and q are arbitrary natural numbers. k is a weighting function, and pixel interpolation is sometimes called “nearest neighbor”, “bilinear”, “bicubic” or the like depending on how the weighting function k is given. FIG. 12 conceptually shows how pixel interpolation is performed. Here, the pixel value of the pixel E2 is calculated from the pixel values of the pixels existing around it.

  Thereby, the aberration correction unit 70 calculates the coordinates and pixel values of each pixel constituting the corrected image information. That is, the aberration correction unit 70 corrects the distortion aberration of the image information.

  In step S <b> 230, the image storage unit 80 stores the image information given from the aberration correction unit 70.

  Therefore, in the present embodiment, the aberration coefficient calculation unit 54 calculates an aberration correction coefficient using an overlapping area between the first image information and the second to fifth image information (see FIGS. 5 to 10). ), The aberration correction unit 70 corrects the image information using the aberration correction coefficient. Note that the imaging device 10 selects image information having the largest number of feature points from the second to fifth image information, and sets the aberration correction coefficient calculated from the image information and the first image information. Based on this, the image information may be corrected. Such correction is possible because distortion is rotationally symmetric with respect to the projection center. Hereinafter, this point will be described.

  FIG. 13 shows the types of distortion, and FIG. 14 is a graph showing the relationship between the distance from the projection center, that is, the actual image height and the distortion ratio (f (r) in Expression (16)) for each distortion. It is. Specifically, FIG. 13A shows barrel aberration, and FIG. 14A shows the relationship between actual image height and distortion ratio in barrel aberration. FIG. 13B shows pincushion aberration, and FIG. 14B shows the relationship between actual image height and distortion ratio in pincushion aberration. FIG. 13C shows the Jin-Kasa aberration, and FIG. 14C shows the relationship between the actual image height and the distortion ratio in the Jin-Kasa aberration. As shown in FIG. 13, since all distortion aberrations are rotationally symmetric with respect to the projection center D, all regions in the image information satisfy the relationship shown in FIG. Therefore, the aberration correction unit 70 can correct the entire image information with high accuracy even if correction is performed as described above. In addition, since the aberration correction unit 70 calculates the distortion aberration coefficient using only the overlapping area between the first image information and the maximum correlation image information, it is possible to reduce the load on the calculation.

  As described above, the photographing apparatus 10 according to the present embodiment extracts feature points from a plurality of pieces of image information obtained by the photographing units 20 performing photographing with different photographing postures, and feature points extracted from one piece of image information. A set of feature points indicating the same point on the subject is searched from the feature points extracted from the other image information. The photographing apparatus 10 estimates a photographing posture when other image information is photographed based on the set of feature points, and calculates a distortion aberration coefficient of the lens based on the estimated photographing posture. Therefore, even if the lens unit 21 or the zoom rate is changed, the imaging device 10 can estimate the imaging posture and calculate the distortion aberration coefficient based on the image information acquired after the change. That is, the photographing apparatus 10 can more reliably acquire the distortion aberration coefficient corresponding to the lens at the time of photographing (that is, the lens unit 21 at the time of photographing) and the zoom rate. Further, since the photographing apparatus 10 can calculate the distortion aberration coefficient when the photographer photographs an actual landscape, the distortion aberration coefficient can be calculated without using a chart or sample whose geometric information is known. it can.

  Furthermore, the imaging device 10 has a case where a predetermined number or more of first feature points in a region where one image information and another image information overlap constitute a set of feature points. When it is determined that one image information and other image information are correlated, and when it is determined that one image information and other image information are correlated, the photographing posture is estimated based on the set of feature points. Therefore, the photographing apparatus 10 can estimate the photographing posture more reliably.

  Further, the photographing apparatus 10 performs singular value decomposition on the left side of the above-described formula (3), thereby obtaining a product U · S · V of the matrices U and V and the diagonal matrix S having singular values as diagonal components. By converting the column components of the matrix V corresponding to the column of the smallest singular value to a00 to a22, the posture matrix R represented by the equation (4) is calculated. Therefore, since the photographing apparatus 10 calculates the photographing posture as the posture matrix R, the photographing posture can be estimated more reliably.

Furthermore, the imaging device 10 calculates the true image height of the feature point extracted from one image information based on the posture matrix R, and the coordinates (i1 _n , j1) of the feature point extracted from the one image information. _n ), the actual image height of the feature point extracted from one image information is calculated. Then, the photographing apparatus 10 calculates the distortion aberration coefficients h ₁ and h ₂ based on the calculated true image height and actual image height and the above-described equation (5). Therefore, even if the lens unit 21 or the zoom rate is changed, the imaging device 10 can estimate the imaging posture based on the image information acquired after the change and calculate the distortion aberration coefficients h ₁ and h _2. .

  Furthermore, since the imaging device 10 acquires the lens ID and imaging information at the time of imaging, and stores the acquired information and the calculated distortion aberration coefficient in the aberration information storage unit 60, the aberration information is stored. It is possible to easily determine under what conditions each distortion aberration coefficient stored in the storage unit 60 is calculated.

  Further, the photographing apparatus 10 acquires the lens ID and photographing information at the time of photographing, searches the aberration information storage unit 60 for a distortion aberration coefficient corresponding to these information, and images based on the searched distortion aberration coefficient. Correct. Therefore, since the photographing apparatus 10 can perform correction using the distortion aberration coefficient corresponding to the currently used lens and zoom ratio, the correction can be performed more accurately.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Image pick-up device 20 Image pick-up part 21 Lens unit 22 Image pick-up element 23 Image processing part 30 Lens information acquisition part 40 Camera control part 50 Distortion aberration coefficient calculation part 51 Feature point extraction part 52 Feature point correlation part 53 Posture estimation part 54 Aberration coefficient calculation part 60 Aberration information storage unit 70 Aberration correction unit 80 Image storage unit 90 Display unit

Claims (6)

A feature point is obtained from each of a plurality of pieces of image information that are obtained by shooting with different shooting postures by a shooting unit having a shooting lens and capable of shooting using the lens. A feature point extraction unit for extracting
A feature point correlator for searching for a set of feature points indicating the same point on the subject from a first feature point extracted from one image information and a second feature point extracted from other image information; ,
A posture estimation unit that estimates a photographing posture when the other image information is photographed based on the set of feature points;
An imaging apparatus comprising: an aberration coefficient calculation unit that calculates a distortion aberration coefficient of the lens based on the imaging attitude estimated by the attitude estimation unit. The feature point correlation unit includes a predetermined number or more of first feature points in a region where the one piece of image information and the other image information overlap, and the set of feature points. And determining that the one image information and the other image information are correlated,
The posture estimation unit determines a shooting posture when the other image information is shot based on the set of feature points when it is determined that the one image information and the other image information are correlated. The photographing apparatus according to claim 1, wherein the photographing apparatus is estimated. The posture estimation unit performs singular value decomposition on the left side of the following equation (1), thereby converting the matrix U, V and the product U · S · V of the diagonal matrix S having singular values as diagonal components. The posture matrix R represented by the following expression (2) is calculated by setting the column components of the matrix V corresponding to the column of the smallest singular value as a00 to a22. Or the imaging device of 2.

... (1)
Here, n is a natural number, i1 _n and j1 _n are the x and y coordinates of the feature points extracted from the one image information, and i2 _n and j2 _n are extracted from the other image information. These are the x coordinate and y coordinate of the feature point.

... (2) The aberration coefficient calculation unit calculates a true image height of a feature point extracted from the one image information based on the posture matrix R, and coordinates (i1) of the feature point extracted from the one image information _n , j1 _n ), the actual image height of the feature point extracted from the one image information is calculated, the calculated true image height and actual image height, and the following equation (3): 4. The photographing apparatus according to claim 3, wherein the distortion aberration coefficients h ₁ and h ₂ are calculated on the basis of.

... (3)

A lens information acquisition unit that acquires identification information for identifying the lens and zoom rate information regarding a zoom rate when the imaging unit captures the subject;
The aberration information storage part which contrasts and memorize | stores the information acquired by the said lens information acquisition part and the distortion aberration coefficient which the said aberration coefficient calculation part calculated is provided, The any one of Claims 1-4 characterized by the above-mentioned. The imaging device according to item 1. An aberration correction unit that searches the aberration information storage unit for a distortion aberration coefficient corresponding to the information acquired by the lens information acquisition unit and corrects the image information acquired by the imaging unit based on the searched distortion aberration coefficient. The imaging device according to claim 5, further comprising:

Images