JP2011114480A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus which accurately corrects an image distortion and creates a high quality image even when an optical center and an imaging region center of an image pickup device are not in agreement while suppressing the system cost at a low level. <P>SOLUTION: The image processing apparatus processes an image focused on the image pickup device through an optical system, and is provided with a control unit which creates an aberration amount and an interpolation parameter of each dividing point according to the optical system when an image height is divided into a plurality of regions, a first coordinate conversion unit 61 which performs conversion to a coordinate system with the optical center as an original point, an aberration amount calculation unit 62 which calculates the aberration amount in respective pixel coordinates after coordinate conversion by the first coordinate conversion unit 61, a coordinate calculation unit 63 which corrects the positions of respective pixel coordinates based on the aberration amount, a second coordinate conversion unit 64 which converts the respective corrected pixel coordinates from the coordinate system with the optical center as the original point to a coordinate system with an arbitrary point in the imaging region of the image pickup device as an original point, and a pixel value calculation unit which calculates pixel values corresponding to respective pixel coordinates after coordinate conversion by the second coordinate conversion unit 64. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus that is used in an imaging apparatus such as a digital camera or a digital video camera and corrects aberrations caused by an optical lens.

  Conventionally, an imaging apparatus such as a digital camera or a digital video camera has a problem due to aberration of an optical lens, and distortion is one of aberrations. This distortion occurs because the imaging magnification is different between the vicinity of the center of the image and the peripheral portion. Although it is possible to reduce the distortion aberration by devising the optical design, it is difficult to completely avoid the influence of the distortion. In view of this, a method for correcting distortion generated in an optical lens by digital image processing in an image processing apparatus has been proposed.

  Patent Document 1 describes an image processing apparatus, an image processing system, and an image processing method for correcting image distortion at low cost and generating a high-quality image in real time. This image processing apparatus divides an image having distortion into a plurality of grids, and corrects an image having distortion in each grid unit according to a correction vector in each grid unit, and a distortion correction target A decoding unit that decodes a correction vector for each lattice, which is encoded in advance in each lattice unit, and supplies the decoded correction vector for each lattice unit to the image correction unit. ing.

  According to this image processing apparatus, a one-dimensional interpolation operation is performed in the horizontal and vertical directions on the captured image with optical distortion, and the correction vector is efficiently used. Therefore, distortion correction for a moving image that requires high-resolution images is realized with a simple configuration, and a high-quality image without distortion can be easily obtained.

  In addition, since image distortion can be corrected in real time by signal processing, the degree of freedom in lens design can be increased, and lens size reduction and lens cost reduction can be easily realized.

JP 2004-80545 A

  However, since the image processing apparatus described in Patent Document 1 needs to include correction vectors corresponding to all lattice units as data, there arises a problem that it is difficult to keep the system cost low.

  In addition, conventional image processing apparatuses are generally designed so that the optical center coincides with the center of the imaging region, and correction is performed on the assumption that both are also coincident when correcting image distortion. It is normal to do. However, when the manufacturing variation is large, there is a product in which the optical center and the imaging region center are deviated in mass-produced image processing apparatuses, so that correction for image distortion may be inaccurate.

  The present invention solves the above-mentioned problems of the prior art, reduces system cost, and corrects image distortion accurately even when the optical center and the imaging area center of the imaging device do not match. It is an object of the present invention to provide an image processing apparatus that generates a high-quality image.

  In order to solve the above problems, an image processing apparatus according to the present invention includes an imaging element that captures light from a subject through an optical system to generate image data, and a region in which the subject is imaged by the optical system. Divide the image height into a plurality of areas, set the dividing point for each area, and generate the aberration amount of each dividing point and the interpolation parameters necessary to calculate the aberration amount of the image height between the dividing points A coordinate system having an optical center of an area where an object is imaged by the optical system as an origin from a coordinate system having an arbitrary point in the imaging area of the imaging device as an origin for each pixel coordinate of the image data A first coordinate conversion unit that converts the image data into the pixel coordinates of the image data after the coordinate conversion by the first coordinate conversion unit based on the aberration amount and the interpolation parameter of each division point generated by the control unit. Calculate the amount of aberration An aberration amount calculation unit, and a coordinate calculation unit that corrects the position of each pixel coordinate of the image data after coordinate conversion by the first coordinate conversion unit based on the aberration amount at each pixel coordinate calculated by the aberration amount calculation unit And for each pixel coordinate of the image data after the coordinate position correction calculated by the coordinate calculation unit, from a coordinate system having the optical center as the origin to a coordinate system having an arbitrary point in the imaging region of the imaging element as the origin Using a second coordinate conversion unit for conversion and a plurality of pixels in the image data generated by the image sensor, pixel values corresponding to the pixel coordinates of the image data after coordinate conversion by the second coordinate conversion unit are obtained. And a pixel value calculation unit for calculating.

  According to the present invention, the system cost can be kept low, and even when the optical center and the imaging region center of the imaging device do not coincide with each other, it is possible to accurately correct image distortion and generate a high-quality image. it can.

It is a block diagram which shows the structure of the image processing apparatus of the form of Example 1 of this invention. It is a block diagram which shows the detailed structure of the aberration correction part of the image processing apparatus of the form of Example 1 of this invention. It is a block diagram which shows the detailed structure of the lead coordinate calculation part of the image processing apparatus of the form of Example 1 of this invention. It is a figure explaining the method to calculate the coordinate before distortion aberration correction in the image processing apparatus of the form of Example 1 of this invention. It is a figure explaining the case where the optical center in the image processing apparatus of the form of Example 1 of this invention differs from the imaging region center of an image pick-up element. It is a figure explaining the coordinate measuring method of the horizontal direction of an optical center, and the vertical direction in the image processing apparatus of the form of Example 1 of this invention. It is a figure explaining one example of the coordinate measuring method of the horizontal direction of an optical center, and the vertical direction in the image processing apparatus of the form of Example 1 of this invention. It is a flowchart figure which shows operation | movement of the image processing apparatus of the form of Example 1 of this invention. It is a block diagram which shows the structure of the image processing apparatus of the form of Example 2 of this invention. It is a figure which shows the relationship between the focal distance and aberration amount in the image processing apparatus of the form of Example 2 of this invention. It is a flowchart figure which shows operation | movement of the image processing apparatus of the form of Example 2 of this invention.

  Embodiments of an image processing apparatus according to the present invention will be described below in detail with reference to the drawings.

  Embodiments of the present invention will be described below with reference to the drawings. First, the configuration of the present embodiment will be described. FIG. 1 is a block diagram illustrating the configuration of the image processing apparatus according to the first embodiment of the present invention. This image processing apparatus is an apparatus that processes an image formed on an image sensor 10 via an optical system 5 as shown in FIG. 1, and includes an image sensor 10, a signal processing unit 20, an aberration correction unit 30, and a control unit. 40.

  The optical system 5 is constituted by a plurality of interchangeable lenses, for example, and collects reflected light from the subject and forms an image on the image sensor 10. The image sensor 10 converts the imaged image into image data and outputs the image data. Here, for example, the image data is color data such as Red, Green, and Blue for each pixel.

  The signal processing unit 20 converts color data such as Red, Green, and Blue into Y (luminance) and C (color difference) data based on the image data (color data) output from the image sensor 10 and outputs the data.

  The signal processing unit 20 may be configured to output color data such as Red, Green, and Blue as they are instead of converting into Y (luminance) and C (color difference) data. In that case, aberration correction is performed based on the output color data by an aberration correction unit 30 described later. However, when the signal processing unit 20 is configured to convert to the Y (luminance) and C (color difference) data format, there is an advantage that the required memory can be reduced by reducing the data amount.

  When the image height is divided into a plurality of dividing points, the control unit 40 calculates the aberration amount at each dividing point corresponding to the optical system 5 and the interpolation parameter necessary for obtaining the aberration amount of the image height between the dividing points. Generated and written in an interpolation parameter register 31 in the aberration correction unit 30 described later. Here, the aberration amount is obtained by dividing the image height before distortion aberration correction by the image height after distortion aberration correction.

  Note that any method may be used to generate the aberration amount and the interpolation parameter at each division point according to the optical system 5. For example, the control unit 40 stores information such as an aberration amount and an interpolation parameter according to the type of the optical system in advance as a table in a storage unit such as a flash memory or a ROM (Read On Memory) and is mounted on the optical system 5. The mounted optical system 5 is identified by reading the read ROM, and an aberration amount and an interpolation parameter corresponding to the current optical system 5 can be generated by referring to the table. In addition, when the ROM is not mounted on the optical system 5 or when the control unit 40 does not have a means for reading the ROM mounted on the optical system 5, the control unit 40 performs an optical operation based on the user's external operation or the like. A configuration for specifying the type of the system 5 may be used.

  The aberration correction unit 30 corrects distortion aberration of the input image data, and outputs the corrected image data. FIG. 2 is a block diagram illustrating a detailed configuration of the aberration correction unit 30 of the image processing apparatus according to the present exemplary embodiment. As shown in FIG. 2, the aberration correction unit 30 includes an interpolation parameter register 31, a read coordinate calculation unit 32, a data write unit 33, an image memory 34, a data read unit 35, and a pixel interpolation unit 36.

  The interpolation parameter register 31 stores the aberration amount and the interpolation parameter at each division point output by the control unit 40.

  The lead coordinate calculation unit 32 reads out aberration amount information and interpolation parameter information stored in the interpolation parameter register 31, and calculates image coordinates before distortion aberration correction from image coordinates after distortion aberration correction based on these information. FIG. 3 is a block diagram illustrating a detailed configuration of the lead coordinate calculation unit 32 of the image processing apparatus according to the present exemplary embodiment. As shown in FIG. 3, the lead coordinate calculation unit 32 includes a first coordinate conversion unit 61, an aberration amount calculation unit 62, a coordinate calculation unit 63, and a second coordinate conversion unit 64.

  The first coordinate conversion unit 61 forms an object by the optical system 5 from a coordinate system with an arbitrary point in the imaging region of the imaging element 10 as the origin for each pixel coordinate of the image data input to the aberration correction unit 30. Convert to a coordinate system with the optical center of the region as the origin.

  The aberration amount calculation unit 62 calculates the aberration amount at each pixel coordinate of the image data after the coordinate conversion by the first coordinate conversion unit 61 based on the aberration amount and the interpolation parameter of each division point generated by the control unit 40. To do.

  The coordinate calculation unit 63 corrects the position of each pixel coordinate of the image data after the coordinate conversion by the first coordinate conversion unit 61 based on the aberration amount at each pixel coordinate of the image data calculated by the aberration amount calculation unit 62. .

  The second coordinate conversion unit 64 calculates an arbitrary point in the imaging region of the imaging element 10 from the coordinate system with the optical center as the origin for each pixel coordinate of the image data in which the position of each pixel coordinate is corrected by the coordinate calculation unit 63. Convert to the origin coordinate system.

  The data write unit 33 writes the image data (Y (luminance), C (color difference) data) output by the signal processing unit 20 in the image memory 34. The image memory 34 stores the image data written by the data write unit 33.

  The data read unit 35 calculates a memory address of a plurality of pixels necessary for pixel interpolation based on the image coordinates calculated by the read coordinate calculation unit 32 and reads out a plurality of pixels necessary for pixel interpolation from the image memory 34.

  The pixel interpolation unit 36 corresponds to the pixel value calculation unit of the present invention, and performs pixel interpolation using a plurality of pixels in the image data generated by the image sensor 10, so that the image data after the coordinate conversion by the second coordinate conversion unit 64 is performed. A pixel value corresponding to each pixel coordinate is calculated. Specifically, the pixel interpolation unit 36 uses a plurality of pixels to be used for pixel interpolation based on each pixel coordinate of the image data after coordinate conversion output by the second coordinate conversion unit 64 in the lead coordinate calculation unit 32. Weighting is calculated, weighted interpolation is performed on a plurality of pixels of the image data read by the data read unit 35, and the interpolated pixels are output as pixel values corresponding to the pixel coordinates. The pixel interpolation unit 36 can output a final output image by calculating pixel values corresponding to the pixel coordinates of all the image data.

  Next, the operation of the present embodiment configured as described above will be described. FIG. 4 is a diagram illustrating a method for calculating image coordinates before distortion correction from image coordinates after distortion correction in the image processing apparatus of the present embodiment. In FIG. 4, A′B′C′D ′ is an effective area where light from a subject forms an image by the optical system 5, and a point P represents an optical center. PA 'is the maximum image height.

  First, the designer of the image processing apparatus of the present embodiment divides the maximum image height into a plurality of regions PQ ′, Q′R ′, R ′S ′, and S′A ′ as shown in FIG. The aberration amounts at the corrected points P, Q ′, R ′, S ′, and A ′ are obtained in advance. Points P, Q ', R', S ', A' correspond to the dividing points of the present invention. The obtained aberration amount at each division point is stored by the control unit 40 as described above, and written to the interpolation parameter register 31 in the aberration correction unit 30 as necessary.

  For example, when obtaining the aberration amount at the point E ′ after distortion correction, the aberration correction unit 30 included in the image processing apparatus of the present embodiment interpolates the aberration amounts at the points R ′ and S ′ to interpolate at the point E ′. Obtain the amount of aberration.

  Here, the aberration amount interpolation method uses interpolation by an n-order polynomial (n is a natural number) such as linear interpolation or spline interpolation. In addition, it is not always necessary to make the interval between the dividing points equal, and the image processing apparatus of the present invention increases the interpolation accuracy by narrowing the dividing interval or increasing the number of divisions in a region where the change in the amount of aberration is large. Can do.

  When the aberration amount at an arbitrary point after distortion correction is calculated by spline interpolation, the spline interpolation formula is as follows.

In the case of x [i] ≦ x <x [i + 1],
xx = xx- [i] (1)
y = y [i] + xx * (q [i] + xx * (r [i] + s [i] * xx)) (2)
Here, x is the square of the image height at an arbitrary point, and i is an integer value. Further, y is an aberration amount at an arbitrary point. Although the image height at an arbitrary point may be x, the amount of calculation can be reduced by setting x to the square of the image height at an arbitrary point. When the point P is the origin, the horizontal coordinate of an arbitrary point is cx ′, and the vertical coordinate is cy ′, the square x of the image height at the arbitrary point is expressed by the following equation.

x = cx ′ * cx ′ + cy ′ * cy ′ (3)
If the image height at the point E ′ is x, it is necessary to further calculate the square root.

  Here, x [0], y [0], q [0], r [0], and s [0] are interpolation parameters at the point P. x [1], y [1], q [1], r [1], and s [1] are interpolation parameters at the point Q '. x [2], y [2], q [2], r [2], and s [2] are interpolation parameters at the point R ′. x [3], y [3], q [3], r [3], and s [3] are interpolation parameters at the point S ′. x [4], y [4], q [4], r [4], and s [4] are interpolation parameters at the point A ′.

  Also, x [0] is the point P, x [1] is the point Q ′, x [2] is the point R ′, x [3] is the point S ′, and x [4] is the image height 2 at the point A ′. It is a power. Further, y [0] is the point P, y [1] is the point Q ', y [2] is the point R', y [3] is the point S ', and y [4] is the aberration amount at the point A'. .

  For example, consider the image height at point E ′ shown in FIG. Since the image height at the point E ′ can be expressed as x [2] ≦ x <x [3], the equation (1) is xx = xx−2 [2]. The equation (2) is expressed by y = y [2] + xx * (q [2] + xx * (r [2] + s [2] * xx)).

  Assuming that the horizontal coordinate of the point E after the coordinate position correction of the point E ′ shown in FIG. 4 is cx, cx = y * cx ′. Further, if the vertical coordinate of the point E is cy, cy = y * cy ′.

  FIG. 5 is a diagram illustrating a case where the optical center and the imaging area center of the imaging device 10 are different in the image processing apparatus of the present embodiment. In FIG. 5, A′B′C′D ′ indicates the upper left end, the lower left end, the lower right end, and the upper right end of the effective region where the subject forms an image by the optical system 5, and the point P is the effective of the optical system 5. Represents the optical center of the region. On the other hand, A ″ B ″ C ″ D ″ is an imaging region of the image sensor 10, and a point P ″ represents the center of the imaging region.

  In FIG. 5, xsize represents the number of horizontal pixels of the image sensor 10, and ysize represents the number of vertical pixels of the image sensor 10. Further, when the horizontal coordinate of the optical center P is xp, the vertical coordinate is yp, the horizontal coordinate of the imaging region center P ″ is xp ″, and the vertical coordinate is yp ″, the imaging element with respect to the optical center P The offsets xoff and yoff of the ten imaging region centers P ″ are expressed by the following equations.

xoff = xp ″ −xp (4)
yoff = yp ″ −yp (5)
The offset information is written into the interpolation parameter register 31 by the control unit 40 as a kind of interpolation parameter. The control unit 40 may acquire the offset information indicating the shift amount between the optical center P and the imaging region center P ″ of the imaging device 10 by any method. An example is shown below. FIG. 6 is a diagram for explaining a method of measuring the horizontal coordinate xp and the vertical coordinate yp of the optical center P in the image processing apparatus of the present embodiment.

  As shown in FIG. 6, the image sensor 10 photographs the chart 1 in which a black circle is drawn on a white background via the optical system 5. The black circle is large enough to shoot. The center of the black circle is arranged so as to coincide with the optical center P on the optical axis center 3 of the optical system 5.

  Due to the influence of the spatial frequency characteristics of the optical system 5, the black circle formed on the image sensor 10 becomes a blurred image as shown in FIG. Here, the maximum luminance of the image generated by the imaging center 10 is 255. When binarization is performed with respect to the image formed on the image sensor 10 with 50% or more of the luminance set to the maximum luminance 255 and the other luminance set to the minimum luminance 0, an image as shown in FIG. 7B is obtained.

  Assuming that the minimum value in the horizontal direction of the portion of brightness 0 is xmin, the maximum value is xmax, the minimum value in the vertical direction is ymin, and the maximum value is ymax, the horizontal coordinate xp of the optical center P shown in FIG. The vertical coordinate yp is expressed by the following equation.

xp = (xmin + xmax) / 2 (6)
yp = (ymin + ymax) / 2 (7)
By substituting the values of xp and yp determined in this way into the equations (4) and (5), the offsets xoff and yoff are obtained. The control unit 40 acquires offset information xoff and yoff by the method described above, for example, in the manufacturing stage, and uses it as an interpolation parameter.

  Next, the operation of the aberration correction unit 30 will be described in detail. FIG. 8 is a flowchart showing the operation of the image processing apparatus of this embodiment. The aberration amount interpolation method will be described on the assumption that spline interpolation is used.

  First, the control unit 40 generates an aberration amount and an interpolation parameter for each of the division points P, Q ′, R ′, S ′, and A ′. In the present embodiment, since the control unit 40 has information on the aberration amount and the interpolation parameter in advance, it is only necessary to output such information. The interpolation parameters include the horizontal pixel number xsize, the vertical pixel number ysize, the horizontal offset xoff of the imaging area center P ″ of the imaging element 10 with respect to the optical center P, and the vertical offset yoff. The aberration amounts and the interpolation parameters of the points P, Q ′, R ′, S ′, and A ′ are written in the interpolation parameter register 31 (step S1).

  Next, the first coordinate conversion unit 61 in the lead coordinate calculation unit 32 and the horizontal coordinate xp ″ of the imaging region center P ″ in the coordinate system with the point A ″ shown in FIG. 5 as the origin and the vertical coordinate yp ″ is calculated based on xsize and ysize (step S2).

  Next, the first coordinate conversion unit 61 sets the horizontal coordinate sx ″ and the vertical coordinate sy ″ of the image data in the coordinate system with the point A ″ shown in FIG. 5 as the origin as the origin (step S3). The 1-coordinate conversion unit 61 covers the entire imaging region by sequentially changing the horizontal coordinate sx ″ and the vertical coordinate sy ″ as will be described later.

  Next, the first coordinate conversion unit 61 compares sy ″ with ysize−1. If sy ″ ≦ ysize−1, the process proceeds to step S5 to be described later. If not, the process ends ( Step S4). That is, the first coordinate conversion unit 61 determines whether or not the vertical coordinate sy ″ is in the imaging region of the imaging device 10.

  Next, the first coordinate conversion unit 61 compares sx ″ with xsize−1 (step S5). If sx ″ ≦ xsize−1, the process proceeds to step S7 to be described later. “1” is added to “1” and “0” is substituted for “sx”, and the process returns to Step S4 (Step S6). That is, the first coordinate conversion unit 61 determines whether or not the vertical coordinate sx ″ is within the imaging region of the imaging device 10, and when sx ″ exceeds the imaging region, sx ″ is returned to 0 and is vertical. The coordinates are lowered one step.

  Next, for each pixel coordinate of the image data input to the aberration correction unit 30, the first coordinate conversion unit 61 uses the optical center P as the origin from the coordinate system having the point A ″ in the imaging region of the imaging element 10 as the origin. (Step S7) The cx ′ generated by the coordinate conversion is the horizontal coordinate with the point P as the origin, as described above, and cy ′ The vertical coordinate.

  Next, the aberration amount calculation unit 62 performs coordinate conversion by the first coordinate conversion unit 61 on the basis of the aberration amount and the interpolation parameter of each division point generated by the control unit 40 in steps S8 to S16. The amount of aberration at each pixel coordinate of the subsequent image data is calculated.

  Specifically, the aberration amount calculation unit 62 first calculates the square x of the image height based on the coordinates cx ′ and cy ′ output from the first coordinate conversion unit 61 (step S8).

  Next, the aberration amount calculation unit 62 compares x with x [1] (step S9). If x <x [1], the aberration amount calculation unit 62 sets i to 0 (step S10), and proceeds to step S16 described later.

  When x ≧ x [1], the aberration amount calculation unit 62 compares x with x [2] (step S11). In the case of x <x [2], the aberration amount calculation unit 62 sets 1 to i (step S12), and proceeds to step S16 described later.

  When x ≧ x [2], the aberration amount calculation unit 62 compares x with x [3] (step S13). When x <x [3], the aberration amount calculation unit 62 sets i to 2 (step S14), and proceeds to step S16 described later.

  In the case of x ≧ x [3], the aberration amount calculation unit 62 sets 3 to i (step S15).

  That is, the aberration amount calculation unit 62, in steps S8 to S15, selects any one of the plurality of regions in which the image height of each pixel coordinate after the coordinate conversion by the first coordinate conversion unit 61 is divided by the control unit 40. It is determined whether it belongs to the area.

  Thereafter, the aberration amount calculation unit 62 uses the interpolation parameters to interpolate each pixel in the image data by interpolating from the aberration amount at the division point dividing the region to which the region divided by the control unit 40 belongs. The aberration amount y at the coordinates is calculated (step S16). In this embodiment, the aberration amount calculation unit 62 calculates the aberration amount y by spline interpolation.

  For example, when x [2] ≦ x <x [3] is satisfied, the aberration amount calculation unit 62 sets i to 2 in step S14, and performs interpolation parameters (x [2], y [2], q [2], r [2], s [2]) is used to interpolate from the aberration amount y [2] at the dividing point R ′ to obtain the aberration amount y at the pixel coordinates. calculate.

  Next, the coordinate calculation unit 63 determines the position of each pixel coordinate of the image data after the coordinate conversion by the first coordinate conversion unit 61 based on the aberration amount at each pixel coordinate of the image data calculated by the aberration amount calculation unit 62. Is corrected (step S17). Accordingly, the coordinate calculation unit 63 calculates the horizontal coordinate cx and the vertical coordinate cy of the image data before distortion correction.

  Next, the second coordinate conversion unit 64 captures an image from the coordinate system having the optical center P as the origin for each pixel coordinate cx, cy of the image data in which the position of each pixel coordinate of the image data is corrected by the coordinate calculation unit 63. The coordinate is converted into a coordinate system having an arbitrary point in the imaging region of the element 10 (point A ″ in this embodiment) as the origin (step S18). The second coordinate conversion unit 64 performs horizontal coordinate by this coordinate conversion. sx and vertical coordinate sy are output.

  Next, the data read unit 35 calculates memory addresses of a plurality of pixels necessary for pixel interpolation based on sx and sy output from the second coordinate conversion unit 64 (step S19).

  Thereafter, the data read unit 35 reads the pixel value (Y (luminance) and C (color difference) of the pixel) of the pixel corresponding to the calculated memory address from the image memory 34 (step S20).

  Next, the pixel interpolation unit 36 calculates weights of a plurality of pixels used for pixel interpolation based on the pixel coordinates sx and sy after the coordinate conversion output from the second coordinate conversion unit 64 (step S21). Pixel interpolation using weighting is performed on the pixel values of the plurality of pixels read by the data read unit 35 in S20 (step S22). Thereafter, the pixel interpolation unit 36 outputs the interpolated pixel as a pixel value corresponding to the pixel coordinate.

  Next, the first coordinate conversion unit 61 adds 1 to sx ″ (step S23). In this way, the first coordinate conversion unit 61 shifts the pixel coordinates in the horizontal direction after aberration correction to the right by one. After that, the process returns to step S5.

  As described above, according to the image processing apparatus according to the first embodiment of the present invention, the system cost is kept low, and even when the optical center of the optical system and the imaging region center of the imaging device do not coincide with each other. High-quality images can be generated by accurately correcting image distortion.

  That is, the image processing apparatus according to the present embodiment includes the first coordinate conversion unit 61, so that the pixel coordinates of the image data input to the aberration correction unit 30 or the image data after the aberration correction by the aberration correction unit 30 is performed. Is converted from a coordinate system having an arbitrary point (point A ″ in the present embodiment) in the imaging area of the image sensor 10 to a coordinate system having the optical center P as the origin, and the aberration amount calculation unit 62 and coordinate calculation are performed. By providing the unit 63, the pixel coordinates of the image data before aberration correction are calculated in the coordinate system having the optical center P as the origin, and by providing the second coordinate conversion unit 64, the pixel coordinates of the image data before aberration correction are optically calculated. A pixel of the image data is provided by converting the coordinate system having the center P as the origin into a coordinate system having an arbitrary point (in the present embodiment, the point A ″) in the imaging region of the image sensor as the origin, and including the pixel interpolation unit 36. Display pixel values corresponding to coordinates Since obtained by interpolation, it is possible to perform accurate correction even when the imaging region center P of the optical center P and the image pickup device "does not match, it has the advantage of contributing to the generation of high quality images.

  Furthermore, unlike the image processing apparatus described in Patent Document 1, the image processing apparatus according to the present embodiment does not need to include correction vectors corresponding to all lattice units as data, and divides the image height into a plurality of regions. By providing only the aberration amount and the interpolation parameter at each division point in this case, the amount of aberration at the image height between the division points can be interpolated from the aberration amount at the division point, thus reducing the system cost. It also has the advantage of being able to

  FIG. 9 is a block diagram illustrating the configuration of the image processing apparatus according to the second embodiment of the present invention. This image processing apparatus is an apparatus that processes an image formed on the image sensor 10 via the optical system 5 as shown in FIG. 9, and includes a signal processing unit 20, an aberration correction unit 30, a control unit 40, and zoom drive. Part 50 is provided. That is, the difference from the configuration of the image processing apparatus according to the first embodiment described with reference to FIG. 1 is that the zoom driving unit 50 is further provided.

  The zoom drive unit 50 adjusts the focal length of the optical system 5. The optical system 5 has a structure in which the lens moves to adjust the zoom magnification, that is, the focal length. The zoom drive unit 50 drives the corresponding lens of the optical system 5 based on the value set by the control unit 40.

  The image sensor 10, the signal processing unit 20, and the aberration correction unit 30 are the same as those in the first embodiment, and a duplicate description is omitted.

  The control unit 40 has the same function as that of the first embodiment, determines the focal length of the optical system 5, and generates a driving signal for driving the zoom driving unit 50 according to the focal length. Further, when the focal length by the optical system 5 is divided into a plurality of regions, the control unit 40 interpolates from “aberration amount of each division point of image height” corresponding to “each division point of focal length”, “Aberration amount at each division point of image height” at the current focal length (determined focal length) is calculated.

  Note that any method may be used for generating the aberration amount and the interpolation parameter of “each division point of image height” corresponding to “each division point of focal length”. For example, the control unit 40 stores in advance information such as aberration amounts and interpolation parameters at “divided points of image height” corresponding to “divided points of focal length” as a table in a storage unit such as a flash memory or a ROM. It may be.

  Next, the operation of the present embodiment configured as described above will be described. FIG. 10 is a diagram illustrating the relationship between the focal length and the aberration amount in the image processing apparatus according to the present embodiment. In FIG. 10, the horizontal axis represents the focal length, and the vertical axis represents the aberration amount. When the focal length is a, the lens of the optical system 5 is at the wide end (when the zoom lens is moved to the widest side (wide angle side)). When the focal length is e, the lens of the optical system 5 is at the telephoto end (the zoom lens is moved to the most telephoto side and the magnification is maximum).

  As shown in FIG. 10, in the zoom optical system, the amount of aberration varies according to the focal length. Therefore, the control unit 40 of the image processing apparatus according to the present embodiment divides the focal length into a plurality of regions ab, bc, cd, and de, and corresponds to the division points a, b, c, d, and e of the focal length. The amount of aberration is obtained in advance and stored. In this case, the aberration amount in this case needs to be obtained in advance as in Example 1, “aberration amount at each division point of image height”. That is, the designer of the image processing apparatus according to the present embodiment can determine the image height division points P, Q ′, R ′, S ′, and the like for each of the focal length division points a, b, c, d, and e. The amount of aberration in each of A ′ is obtained in advance.

  The obtained aberration amount is stored by the control unit 40 as described above, and written to the interpolation parameter register 31 in the aberration correction unit 30 as necessary.

  The aberration amount interpolation method uses interpolation by an nth order polynomial (n is a natural number) such as linear interpolation or spline interpolation. In addition, it is not always necessary to make the interval between the dividing points equal. The image processing apparatus of the present invention can increase the interpolation accuracy by narrowing the dividing interval or increasing the number of divisions in a region where the change in the amount of aberration is large. .

  When the aberration amount at the focal length f is calculated by spline interpolation, the spline interpolation formula is as follows.

In the case of x_ [j] ≦ f <x_ [j + 1],
xx_ = fx− [j] (8)
y_ = y_ [j] + xx _ * (q_ [j] + xx _ * (r_ [j] + s_ [j] * xx_)) (9)
Here, y_ is an aberration amount at the focal length f. Further, x_ [0], y_ [0], q_ [0], r_ [0], and s_ [0] are interpolation parameters at the focal length a. x_ [1], y_ [1], q_ [1], r_ [1], and s_ [1] are interpolation parameters at the focal length b. x_ [2], y_ [2], q_ [2], r_ [2], and s_ [2] are interpolation parameters at the focal length c. x_ [3], y_ [3], q_ [3], r_ [3], and s_ [3] are interpolation parameters at the focal length d. x_ [4], y_ [4], q_ [4], r_ [4], and s_ [4] are interpolation parameters at the focal length e.

  X_ [0] is the focal length a, x_ [1] is the focal length b, x_ [2] is the focal length c, x_ [3] is the focal length d, and x_ [4] is the focal length e. Furthermore, y_ [0] is the focal length a, y_ [1] is the focal length b, y_ [2] is the focal length c, y_ [3] is the focal length d, and y_ [4] is the amount of aberration at the focal length e. Show.

  Therefore, since x_ [0] ≦ f <x_ [1] can be expressed based on the focal length f shown in FIG. 10, the equation (8) is xx_ = fx− [0]. Moreover, (9) Formula is represented by y_ = y_ [0] + xx _ * (q_ [0] + xx _ * (r_ [0] + s_ [0] * xx_)).

  Next, the operation of the control unit 40 will be described in detail. FIG. 11 is a flowchart illustrating the operation of the image processing apparatus according to the present exemplary embodiment. The aberration amount interpolation method will be described on the assumption that spline interpolation is used.

  First, the control unit 40 determines the focal length f of the optical system 5 by, for example, a user's zoom operation, and generates a drive signal for driving the zoom drive unit 50 according to the focal length f. The zoom drive unit 50 adjusts the focal length of the optical system 5 to f based on the drive signal generated by the control unit 40.

  Next, the control unit 40 compares the focal length f with x_ [1] (step S1). In the case of f <x_ [1], the control unit 40 sets 0 to j (step S2), and proceeds to step S8 described later.

  When f ≧ x_ [1], the control unit 40 compares f with x_ [2] (step S3). In the case of f <x_ [2], the control unit 40 sets 1 to j (step S4), and proceeds to step S8 described later.

  If f ≧ x_ [2], the control unit 40 compares f with x_ [3] (step S5). In the case of f <x_ [3], the control unit 40 sets j to 2 (step S6), and proceeds to step S8 described later.

  When f ≧ x_ [3], the control unit 40 sets 3 to j (step S7).

  In other words, the control unit 40 determines which region the focal length f belongs to from among a plurality of regions divided in steps S1 to S7.

  Next, the control unit 40 interpolates from “aberration amount at each division point of image height” corresponding to “each division point of focal length” to obtain “image height at the current focal length (determined focal length) f”. Is calculated "(step S8). That is, by performing spline interpolation, the control unit 40 performs the aberration amount y_p at the point P, the aberration amount y_q ′ at the point Q ′, the aberration amount y_r ′ at the point R ′, and the point S ′ for the focal length f. The aberration amount y_s ′ and the aberration amount y_a ′ at the point A ′ are calculated.

  Next, the control unit 40 calculates interpolation parameters at points P, Q ′, R ′, S ′, and A ′ based on y_p, y_q ′, y_r ′, y_s ′, and y_a ′ (step S9). .

  Finally, using the obtained aberration amount and interpolation parameter, the image processing apparatus of the present embodiment performs the process shown in FIG. 8 (step S10), and ends the process.

  As described above, according to the image processing apparatus according to the second embodiment of the present invention, in addition to the effects of the first embodiment, the optical system 5 has a structure in which the lens moves to adjust the focal length. Even when the amount of aberration changes in accordance with the focal length, the control unit 40 can calculate an appropriate interpolation parameter in accordance with the focal length f and generate a high-quality image.

  The image processing apparatus according to the present invention is used in an imaging apparatus such as a digital camera or a digital video camera, and can be used in an image processing apparatus that corrects aberrations caused by an optical system.

DESCRIPTION OF SYMBOLS 1 Chart 3 Optical center 5 Optical system 10 Image pick-up element 20 Signal processing part 30 Aberration correction part 31 Interpolation parameter register 32 Read coordinate calculation part 33 Data write part 34 Image memory 35 Data read part 36 Pixel interpolation part 40 Control part 50 Zoom drive part 61 First coordinate conversion unit 62 Aberration amount calculation unit 63 Coordinate calculation unit 64 Second coordinate conversion unit

Claims (2)

An image sensor that captures light from a subject via an optical system and generates image data;
The image height of the area where the subject is imaged by the optical system is divided into a plurality of areas, division points are set for each area, and the aberration amount of each division point and the image height between the division points are set. A control unit for generating interpolation parameters necessary for obtaining
First, each pixel coordinate of the image data is converted from a coordinate system having an arbitrary point in the imaging area of the image sensor as an origin to a coordinate system having an optical center of an area where a subject is imaged by the optical system as an origin. A coordinate transformation unit;
An aberration amount calculation unit that calculates an aberration amount at each pixel coordinate of the image data after the coordinate conversion by the first coordinate conversion unit, based on the aberration amount and the interpolation parameter of each division point generated by the control unit; ,
A coordinate calculation unit that corrects the position of each pixel coordinate of the image data after coordinate conversion by the first coordinate conversion unit, based on the aberration amount at each pixel coordinate calculated by the aberration amount calculation unit;
For each pixel coordinate of the image data after the coordinate position correction calculated by the coordinate calculation unit, the coordinate system having the optical center as an origin is converted into a coordinate system having an arbitrary point in the imaging region of the imaging element as the origin. A second coordinate conversion unit;
A pixel value calculation unit that calculates a pixel value corresponding to each pixel coordinate of the image data after the coordinate conversion by the second coordinate conversion unit, using a plurality of pixels in the image data generated by the image sensor;
An image processing apparatus comprising: A zoom driving unit for adjusting a focal length of the optical system;
The control unit determines a focal length of the optical system, generates a driving signal for driving the zoom driving unit according to the focal length, and divides the focal length by the optical system into a plurality of regions. 2. The division point is set for each of the regions, and the aberration amount of the image height at the current focal length is calculated from the aberration amount of the image height corresponding to each division point of the region. Image processing apparatus.