JP2015216576A - Image processing apparatus, image processing method, imaging apparatus, electronic apparatus, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suitably correct image blur.SOLUTION: An image processing apparatus includes an acquisition unit for acquiring a filter for performing blur correction respectively set for a plurality of image heights and a correction unit for correcting a pixel value of a pixel in an image height to be corrected by using the filter acquired by the acquisition unit. The correction unit applies a filter set to an image height adjacent to the image height to be corrected to a pixel value of a pixel to be corrected, calculates a coefficient from positional relation between the image height to be corrected and the adjacent image height and calculates a corrected pixel value by using the filter-applied pixel value and the coefficient. This technique can be applied to an image processing apparatus, e.g. a camera, for processing a photographed image.

Description

  The present technology relates to an image processing device, an image processing method, an imaging device, an electronic device, and a program. Specifically, the present invention relates to an image processing apparatus, an image processing method, an imaging apparatus, an electronic apparatus, and a program that perform image blur correction.

  Blur may occur in an image taken by an imaging device such as a digital camera. In particular, in a camera using an inexpensive lens, image blur due to lens aberration increases as the image height (distance from the optical center) increases.

  For example, Patent Documents 1 to 3 are disclosed as conventional techniques that disclose image noise and blur correction processing based on such various factors. Patent Document 1 discloses a correction processing configuration using different sharpening parameters for each region of a captured image. Specifically, the image correction is performed by increasing the sharpening parameter as the image height (distance from the optical center) increases.

  However, the process of Patent Document 1 merely performs a process of increasing the intensity of the high-frequency component in accordance with the image height, and does not take into account, for example, a change in the frequency characteristic of blur, and appropriate correction is not performed. there is a possibility.

  Patent Document 2 discloses a configuration in which a blur correction process is performed by selecting a filter corresponding to each lens characteristic in order to cope with a blur that varies depending on a lens difference or a manufacturing error. However, if this correction processing is applied to blur in which the blur method changes continuously according to the image height, such as the above-described image height-dependent blur, the number of types of filters for blur correction is enormous. Could be a number.

  Furthermore, Patent Document 3 applies an appropriate correction according to the image height by applying a filter in which the position of the center of gravity is shifted for each image height with respect to so-called one-sided blur indicating how the blur occurs depending on the image height. A configuration to execute is disclosed. More specifically, correction corresponding to blurring is performed by correcting the position of the center of gravity of the filter. However, the focus is on the correction of the center of gravity, the control of the correction strength is not shown, and individual processing corresponding to the strong or weak part is not executed, and appropriate correction is not made Occurs.

JP 2006-246080 A JP 2009-159603 A JP 2010-081263 A

  In the conventional blur correction, in addition to the above, for example, pixel value correction based on filtering by convolution calculation is performed, but in order to perform various corrections, a number of filters for performing this filtering are used. In some cases, a large-capacity memory is required for storage, which may hinder the realization of a small size and low cost.

  This technique is made in view of such a situation, and makes it possible to perform an appropriate correction according to the image height.

  An image processing apparatus according to an aspect of the present technology is an object to be corrected using an acquisition unit that acquires a filter that performs blur correction that is set for each of a plurality of image heights, and the filter acquired by the acquisition unit. A correction unit that corrects the pixel value of the pixel at the obtained image height.

  The correction unit applies the filter that is set to an image height adjacent to the image height that is the correction target to a pixel value of the pixel that is the correction target, and the image height that is the correction target. A coefficient can be calculated from the positional relationship with the adjacent image height, and a corrected pixel value can be calculated using the pixel value after applying the filter and the coefficient.

  The correction unit calculates a coefficient from a positional relationship between the image height that is the correction target and the adjacent image height, and the filter that is set to an image height adjacent to the image height that is the correction target; A filter to be applied to the pixel value of the pixel to be corrected is generated using the coefficient, and a corrected pixel value is calculated using the generated filter and the pixel value of the pixel to be corrected To be able to.

  As the filter coefficient, PSF (Point Spread Function) data is calculated from a plurality of image points on the first image height, the PSF data is averaged, and the averaged PSF data is used. Thus, the coefficient is approximated by a predetermined function and calculated from the approximated PSF data.

  The coefficient of the filter can be calculated using a Wiener filter.

  The image height for which the filter is set may be an image height at which the shape of the PSF changes greatly.

  As the filter coefficient, MTF (Modulation Transfer Function) data is calculated from a plurality of image points on the first image height, the MTF data is averaged, and the averaged MTF data is used. Thus, the coefficient is approximated by a predetermined function and calculated from the approximated MTF data.

  The coefficient of the filter can be calculated using a Wiener filter.

  The image height for which the filter is set can be an inflection point of the MTF data.

  An image processing method according to an aspect of the present technology acquires a filter that performs blur correction that is set for each of a plurality of image heights, and uses the acquired filter to obtain a pixel of a pixel at an image height that is a correction target. A step of correcting the value.

  A program according to an aspect of the present technology acquires a filter that performs blur correction that is set for each of a plurality of image heights, and uses the acquired filter to calculate a pixel value of a pixel at an image height that is a correction target. A process including a correction step is executed.

  An imaging apparatus according to an aspect of the present technology uses the imaging device, an acquisition unit that acquires a filter that performs blur correction that is set for each of a plurality of image heights, and the filter acquired by the acquisition unit, A correction unit that corrects a pixel value of a pixel at an image height that is a correction target among pixels picked up by the image pickup device;

  An electronic apparatus according to an aspect of the present technology is subjected to correction using an acquisition unit that acquires a filter that performs blur correction that is set for each of a plurality of image heights, and the filter acquired by the acquisition unit. A correction unit that corrects a pixel value of a pixel at an image height and an image processing unit that performs predetermined image processing on the pixel value after correction from the correction unit.

  In the image processing apparatus, the image processing method, and the program according to one aspect of the present technology, a filter that performs blur correction that is set for each of a plurality of image heights is acquired, and the acquired filter is used to correct the correction target. The pixel value of the pixel at the set image height is corrected.

  An imaging device according to one aspect of the present technology includes the image processing device, and includes an imaging element for capturing an image.

  An electronic apparatus according to an aspect of the present technology includes the image processing device, and includes a processing unit that further processes a signal processed by the image processing device.

  According to one aspect of the present technology, it is possible to perform appropriate correction according to the image height.

  Note that the effects described here are not necessarily limited, and may be any of the effects described in the present disclosure.

It is a figure showing composition of an embodiment of an image processing device to which this art is applied. It is a figure which shows the structure of a signal processing part. It is a figure which shows an example of a lens aberration correction filter coefficient. It is a figure which shows an example of a lens aberration correction filter coefficient. It is a figure which shows an example of a lens aberration correction filter coefficient. It is a figure for demonstrating lens PSF data. It is a figure for demonstrating a coma aberration. It is a figure for demonstrating how to obtain | require a lens aberration correction filter coefficient. It is a figure for demonstrating how to obtain | require a lens aberration correction filter coefficient. It is a figure for demonstrating selection of the image height which calculates a coefficient. It is a figure for demonstrating the position of the image height from which the coefficient was calculated. It is a figure for demonstrating a blend coefficient. It is a figure for demonstrating a recording medium.

Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. 1. Configuration of image processing apparatus 2. Configuration of signal processing unit 3. Lens aberration correction filter coefficient 4. Determination of first lens aberration correction filter coefficient 5. Method of obtaining second lens aberration correction filter coefficient 6. Image height for calculating lens aberration correction filter coefficient 7. Interpolation of lens aberration correction filter coefficient About recording media

<Configuration of image processing apparatus>
The present technology described below can be applied to an apparatus that corrects a blur that occurs in a captured image. First, the characteristics of blur occurring in a captured image will be described.

  In an image taken by an imaging device such as a digital camera, blurring occurs according to lens characteristics and characteristics of an imaging element such as a CCD or CMOS. In particular, in a camera using an inexpensive lens, image blur due to lens aberration increases as the image height (distance from the optical center) increases.

  Of the blurs that generally occur according to the image height, the center region of the image has a small blur. That is, the closer to the optical center (the smaller the image height), the smaller the blur. On the other hand, the greater the distance from the optical center, that is, the greater the image height, the greater the blur.

  As described above, the blur generated in the image taken by the camera tends to be smaller at the center of the image area near the optical center of the lens of the camera and larger at the peripheral area of the image far from the optical center of the lens. An imaging apparatus to which the present technology that can correct such blur will be described.

<Configuration of signal processing unit>
FIG. 1 is a diagram illustrating a configuration of an embodiment of an imaging apparatus to which the present technology is applied. The imaging apparatus 100 includes an optical lens 111, an imaging element (image sensor) 112, a signal processing unit 113, a control unit 114, and a memory 115. Note that the imaging device is an aspect of an image processing device.

  Note that the image processing apparatus according to the present disclosure includes an apparatus such as a personal computer (PC). An image processing apparatus such as a PC does not include the optical lens 111 and the image sensor 112 of the image capturing apparatus 100 illustrated in FIG. 1, and includes other components, and includes an acquisition data input unit or a storage unit of the image capturing apparatus 100. It becomes composition.

  Hereinafter, the imaging apparatus 100 illustrated in FIG. 1 will be described as a representative example of the image processing apparatus of the present disclosure. Note that the imaging apparatus 100 illustrated in FIG. 1 is, for example, a still camera or a video camera.

  The imaging device 112 of the imaging apparatus 100 shown in FIG. 1 has a configuration including a color filter having a Bayer array composed of RGB arrays. Here, the configuration and processing using the image sensor 112 having an RGB array will be described, but the present technology can also be applied to a configuration using an image sensor having an RGBW array other than the RGB array. Further, for example, the present invention can be applied to a configuration using an image sensor having an array of Y (yellow), C (cyan), M (magenta), and the like. The present technology can be applied to color arrangements other than the arrangements exemplified here.

The imaging device 112 of the imaging apparatus 100 shown in FIG. That is,
Red (R) that transmits wavelengths near red,
Green (G) that transmits wavelengths in the vicinity of green,
Blue (B) that transmits wavelengths near blue,
It is an image sensor provided with a filter having these three types of spectral characteristics.

  The image sensor 112 having the RGB array 121 receives RGB light in units of pixels via the optical lens 111, and generates and outputs an electrical signal corresponding to the received light signal intensity by photoelectric conversion. A mosaic image composed of three types of RGB spectra is obtained by the image sensor 112.

  The output signal of the image sensor 112 is input to the image signal correction unit 131 of the signal processing unit 113.

  The image signal correction unit 131 performs a blur correction process in consideration of, for example, a blur method that changes according to the image height (distance from the optical center) and the focus position. The configuration of the image signal correction unit 131 will be described later with reference to FIG.

  The mosaic image subjected to the blur correction in the image signal correction unit 131 is output to the RGB signal processing unit 132. The output of the image signal correction unit 131 is data having a Bayer array, similar to the output from the image sensor 112.

  The RGB signal processing unit 132 generates a color image 116 by performing demosaic processing, white balance adjustment processing, γ correction processing, and the like. The generated color image 116 is recorded in the memory 115.

  The control unit 114 controls these series of processes. For example, a program for executing a series of processes is stored in the memory 115, and the control unit 114 executes the program read from the memory 115 and controls the series of processes. The memory 115 can be configured by various recording media such as a magnetic disk, an optical disk, and a flash memory.

  A detailed configuration of the image signal correction unit 131 will be described with reference to FIG. As shown in FIG. 2, the image signal correction unit 131 includes an image height calculation unit 201, a blend coefficient calculation unit 202, a line memory 203, a lens aberration correction processing unit 204, an edge detection unit 205, and an edge / flat portion blend processing unit. 206. The lens aberration correction processing unit 204 includes a lens aberration correction unit 221 and a blend processing unit 222.

  The pixel value signal corresponding to each pixel output from the image sensor 112 is temporarily stored in the line memory 203. An xy address indicating the coordinate position of each pixel associated with the pixel value of each pixel is output to the image height calculation unit 201.

  The line memory 203 includes a line memory for the horizontal 7 lines of the image sensor 112, for example. From the line memory 203, data for seven horizontal lines are sequentially output in parallel. The output destinations are the lens aberration correction unit 221, the edge detection unit 205, and the edge / flat portion blend processing unit 206. The imaging data of the RGB array 121 is output in units of 7 lines to each of these processing units.

  Here, the description will be continued on the assumption that data is output in units of 7 lines. However, this is because a filter described later is described by taking a filter for 7 × 7 pixels as an example, and this filter is a filter for other than 7 × 7 pixels. If so, the number of lines corresponding to the filter is set, and the line memory 203 accumulates and outputs pixel values for the set number of lines.

  The edge detection unit 205 verifies the output signal from the line memory 203, generates edge information included in the image, for example, edge information including the edge direction and the edge strength, and outputs the edge information to the edge / flat part blend processing unit 206. . Specifically, for example, the edge / flat part blend processing unit 206 calculates the flatness (weightFlat) calculated from the pixel information of 7 × 7 pixels with the processing target pixel (the center pixel of 7 × 7 pixels) as the center. Output to. This flatness (weightFlat) calculation process can be executed as a process similar to the process described in Japanese Patent Application Laid-Open No. 2011-55038, which is an earlier application of the present applicant.

  The lens aberration correction processing unit 204 verifies the input signal (Rin, Gin, Bin) from the line memory 203, performs a process for correcting image blur, and obtains a blur correction signal (Rd, Gd, Bd) is calculated and output to the edge / flat portion blend processing unit 206.

  The lens aberration correction processing unit 204 corrects image blur, particularly in this embodiment, image blur that depends on the image height. The lens aberration correction unit 221 of the lens aberration correction processing unit 204 acquires lens aberration correction filter coefficients at each image height. The lens aberration correction filter coefficient is a coefficient calculated using, for example, a Wiener filter in order to correct lens aberrations that differ for each image height, as will be described later.

  The correction result by the lens aberration correction unit 221 is supplied to the blend processing unit 222. The coefficient α from the blend coefficient calculating unit 202 is also supplied to the blend processing unit 222. Based on the coefficient α from the blend coefficient calculation unit 202, the blend processing unit 222 generates a blur correction signal (Rd, Gd, Bd) after lens aberration correction at an image height for which no lens aberration correction filter coefficient is set, This is supplied to the edge / flat portion blend processing unit 206.

  The edge / flat portion blend processing unit 206 is output from the RGB signal (Rin, Gin, Bin) in the output signal from the line memory 203, the edge information output from the edge detection unit 205, and the lens aberration correction processing unit 204. A blur correction signal (Rd, Gd, Bd) is input. The edge / flat part blend processing unit 206 uses these pieces of information to generate an output signal RGB that has been subjected to blur correction in consideration of edges, and outputs the output signal RGB to the RGB signal processing unit 132.

  The edge / flat part blend processing unit 206 is based on the edge information of the processing target pixel calculated by the edge detection unit 205, that is, the pixel information of 7 × 7 pixels centering on the processing target pixel (center pixel of 7 × 7 pixels). In accordance with the calculated flatness (weightFlat), the weighted average processing of the blur correction signal (Rd, Gd, Bd) output from the lens aberration correction processing unit 204 is executed, and the RGB pixel values of the RGB array 121 are obtained. calculate. Specifically, RGB pixel values are determined according to the following formula.

R = (weightFlat) × (Rd) + (1-weightFlat) × Rin
G = (weightFlat) × (Gd) + (1−weightFlat) × Gin
B = (weightFlat) × (Bd) + (1−weightFlat) × Bin

  R, G, and B obtained as a calculation result of this expression are output to the RGB signal processing unit 132. The RGB signal processing unit 132 performs signal processing on the RGB array (Bayer array) signal output from the edge / flat part blend processing unit 206 as signal processing on the RGB array (Bayer array) signal, and performs the color image 116 (FIG. 1) is generated. The RGB signal processing unit 132 generates, for example, a color image 116 by executing white balance adjustment processing, demosaic processing, shading processing, RGB color matrix processing, γ correction processing, and the like.

<Lens aberration correction filter coefficient>
The lens aberration correction filter coefficient at a predetermined image height supplied to the lens aberration correction unit 221 will be described.

The lens aberration correction filter coefficient is calculated using, for example, a Wiener filter. Here, the winner filter will be briefly described.
(1) Ideal image (original image) without blur,
(2) Blurred shot image,
(3) a restored image restored by filter application processing on the captured image;
Assume these three images.

here,
(1) An ideal image (original image) without blur,
(3) a restored image restored by filter application processing on the captured image;
A filter that minimizes the square error between these two images is called a least square filter or a Wiener filter.

f (x, y) is an ideal image (original image) without blur,
g (x, y) is a captured image including blur,
h (x, y) is a degradation function due to lens aberration or camera shake,
n (x, y) is a noise component,
And
(X, y) is the pixel position of each image, and f (x, y) to n (x, y) may indicate the pixel value of the coordinate position (x, y) of each image. Here, assuming that the degradation function h (x, y) due to lens aberration and camera shake is a fixed value, the relationship of the following formula (1) is established.

When both sides of the above equation (1) are Fourier transformed, the following equation (2) is obtained.
G (u, v) = H (u, v) × F (u, v) + N (u, v) (2)

  In Equation (2), G (u, v), H (u, v), F (u, v), and N (u, v) are g (x, y), h (x, y), and f, respectively. Represents the Fourier transform of (x, y), n (x, y). From this equation, when there is no zero point in the degradation function due to lens aberration or camera shake, and the noise component is known, F (u, v) can be obtained from the following equation (3).

  However, since the noise component is generally unknown, the above equation (3) cannot be solved exactly. Therefore, blur correction is performed using the Wiener filter K (u, v) of the following equation (4) that minimizes the error between the ideal image (F) and the restored image (F ′) corrected for blur.

However, in the above formula (4),
(Sn (u, v) / Sf (u, v)): Indicates the power spectral density (Γ) between the ideal image (original image) F and noise N.
(Γ = Sn (u, v) / Sf (u, v))

  As shown in the following equation (5), the image obtained by performing inverse Fourier transform of this filter k (x, y) and the image f ′ (x, y) corrected for blur are obtained by convolving the observed image in real space. Can do.

An example of the blur correction process by the Wiener filter according to the above equation (5) is shown in FIG. In FIG.
(A) Captured image G
(B) Wiener filter K
Is shown.

  The captured image is a RAW image composed of RGB pixels. The correction target pixel is the center G pixel of 7 × 7 pixels shown on the left side of FIG. The Wiener filter K shown on the right side of FIG. 3 shows a multiplication coefficient for each of a plurality of G pixels included in 7 × 7 pixels. A convolution operation is performed in which the pixel values of the G pixels included in the 7 × 7 pixels are multiplied by each coefficient and added to calculate a correction pixel value of the central G pixel.

This is expressed by the following equation (6).

  Similarly, R pixels are processed. FIG. 4 shows an example of an image and a filter when processing R pixels. The correction target pixel is an R pixel at the center of 7 × 7 pixels shown on the left side of FIG. The Wiener filter K shown on the right side of FIG. 4 shows a multiplication coefficient for each of a plurality of R pixels included in a 7 × 7 pixel. A convolution operation for multiplying and adding each pixel value to the pixel value of the R pixel included in the 7 × 7 pixel is executed to calculate a correction pixel value of the center R pixel.

This is expressed by the following equation (7).

  Similarly, the B pixel is processed. FIG. 5 shows an example of an image and a filter when processing R pixels. The correction target pixel is the B pixel at the center of the 7 × 7 pixels shown on the left side of FIG. The Wiener filter K shown on the right side of FIG. 5 shows a multiplication coefficient for each of a plurality of B pixels included in a 7 × 7 pixel. A convolution operation for multiplying and adding each pixel value to the pixel value of the B pixel included in the 7 × 7 pixel is executed to calculate a correction pixel value of the central B pixel.

This is expressed by the following equation (8).

  In order to generate such a lens aberration correction filter coefficient, lens PSF (Point Spread Function) data is required. PSF (point spread function) is a function representing the response of the optical system to a point light source. FIG. 6 shows an example of lens PSF data. The diagram shown on the left side of FIG. 6 is a diagram showing the position of the image point, and the graph on the right side shows the PSF data value obtained from each image point.

  6 shows one image height 302 existing in the effective area 301 of the image sensor (the image sensor 112 (FIG. 2)). In FIG. 6, the horizontal direction is the X axis and the vertical direction is the Y axis. Intersections between the image height 302 and the Y axis are image points A and C, and intersections between the image height 302 and the X axis are image points B and D.

  On the right side of FIG. 6, graphs obtained from the image points A to D are shown. In addition, the graphs obtained in the sagittal direction and the tangential direction are shown for the image points A to D, respectively.

  For example, when a graph obtained from the sagittal direction of the image point A is compared with a graph obtained from the tangential direction, it can be seen that the shapes of the graphs obtained from the same image point A are different. From this, it can be seen that the lens PSF has anisotropy and, for example, the degree of blur is different between the sagittal direction and the tangential direction.

  In addition, a graph obtained from the sagittal direction of the image point A, a graph obtained from the sagittal direction of the image point B, a graph obtained from the sagittal direction of the image point C, and a graph obtained from the sagittal direction of the image point D When each is compared, it can be seen that even if the graphs are obtained from the same sagittal direction, the shapes are different. From this, it can be seen that, even if the lens PSF has the same image height, the degree of blur differs depending on the position in the lens. This is the same in the tangential direction.

  Thus, the degree of blur is different even within the same lens. Factors that vary the degree of blur include, for example, variations in manufacturing. Therefore, even if the lenses are manufactured in the same process, the degree of blur may be different.

  For example, when correcting coma aberration off-axis as shown in FIG. 7, it is necessary to hold a plurality of anisotropic lens aberration correction filter coefficients according to the angle.

  If the lens aberration correction filter coefficients are held for each angle, such as sagittal direction and tangential direction, for each image height, the number of lens aberration correction filter coefficients to be held increases. There is a high possibility that the capacity of the memory for holding it will increase.

  The description of the configuration that effectively reduces the number of lens aberration correction filter coefficients to be held will be continued. Here, by approximating the lens PSF using a Gaussian function, a Bessel function or the like, the number of lens aberration correction filter coefficients can be reduced, the memory can be reduced, and as a result, the circuit scale can be reduced. explain.

<Determining First Lens Aberration Correction Filter Coefficient>
With reference to FIG. 8, the process of calculating lens aberration correction filter coefficients using a Gaussian function will be described. In step S1, lens PSF data is acquired at a predetermined image point. Here, as described with reference to FIG. 6, from the four image points A, image points B, C, and D on the X-axis and Y-axis on the image height 302 of the lens, The description will be continued assuming that PSF data in the sagittal direction and the tangential direction are acquired.

  In this case, the case where the lens aberration correction filter coefficient at this image height is obtained from four image points located on the same image height at the end of the optical lens 111 will be described as an example.

  As shown in step S1 of FIG. 8, here, eight PSF data are acquired. Note that how to determine the image point (image height) for acquiring the PSF data will be described later.

  In step S2, averaging is performed using all of the obtained PSF data. In this case, an average of 8 pieces of PSF data is obtained.

  In step S3, approximation is performed using the averaged PSF data and one type of Gaussian function. The approximation coefficient of the Gaussian function is obtained by solving the least square method of the PSF data that has been averaged and the Gaussian function.

  As a result of such calculation, in step S4, PSF data approximated to isotropic PSF data is obtained.

  In step S5, lens aberration correction filter coefficients are obtained by solving a Wiener filter from PSF data approximated to isotropic PSF data.

  Although an example of approximation using one type of function is shown here, approximation using a Bessel function is also possible. As long as the result of combining the functions is isotropic, it is possible to improve the approximation accuracy of the lens PSF data by combining a plurality of functions. By improving the lens aberration correction performance, it is possible to improve the lens aberration correction performance.

<Determining Second Lens Aberration Correction Filter Coefficient>
Next, with reference to FIG. 9, another process for calculating lens aberration correction filter coefficients using a Gaussian function will be described. In step S11, MTF (Modulation Transfer Function) data is acquired at a predetermined image point. The MTF data is one index for evaluating the lens performance, and is data representing how faithfully the contrast of the subject can be reproduced by frequency characteristics.

  The graph of MTF data is a graph in which the horizontal axis is the spatial frequency and the vertical axis is MTF data of an image formed by the lens, as shown in step S11 of FIG. The graph of the MTF data may be a graph in which the contrast reproduction rate (%) and the distance (mm) from the center of the screen are taken on the horizontal axis.

  The graph shown at step S11 in FIG. 9 is also on the lens image height 302 as described with reference to FIG. 6, and the four image points A and B on the X and Y axes. The description will be continued assuming that MTF data in the sagittal direction and the tangential direction are acquired from the image point C and the image point D, respectively.

  Similar to PSF data, MTF data also has anisotropy at the same image point, and therefore differs depending on the direction and also at different image points. Therefore, the MTF data is acquired from a plurality of directions at the same image point as the PSF data, and the MTF data is acquired from different image points within the same lens.

  As shown in step S11 of FIG. 9, the description is continued here assuming that eight MTF data are acquired. A method for determining the image point from which the MTF data is acquired will be described later.

  In step S12, averaging is performed using all of the obtained MTF data. In this case, an average of 8 pieces of MTF data is obtained.

  In step S13, approximation is performed using the averaged MTF data and one type of Gaussian function. The approximation coefficient of the Gaussian function is obtained by solving the least square method of the averaged MTF data and the Gaussian function.

  As a result of such calculation, in step S14, MTF data approximated to isotropic MTF data is obtained.

  In step S15, lens aberration correction filter coefficients are obtained by solving a Wiener filter from MTF data approximated to isotropic MTF data.

  Although an example of approximation using one type of function is shown here, approximation using a Bessel function is also possible. Further, as long as the result of combining the functions is isotropic, it is possible to improve the approximation accuracy of the MTF data by combining a plurality of functions, thereby improving the approximation accuracy of the MTF data. As a result, the lens aberration correction performance can be improved.

  In this way, PSF data and MTF data obtained from a plurality of image points on the lens and from different angles of one image point are averaged, and the averaged data is used to calculate An approximate value of PSF data or MTF data at a predetermined image height is obtained. Then, the lens aberration correction filter coefficient is calculated using the obtained approximate value and the Wiener filter.

  The lens aberration correction filter coefficient calculated from the predetermined image height and the calculation using the lens aberration correction filter coefficient at the predetermined image height have been described with reference to FIGS. Omitted.

<Image height for calculating lens aberration correction filter coefficient>
As described above, at a predetermined image height, a lens aberration correction filter coefficient to be applied to a pixel located at the image height is calculated. A plurality of image heights are selected for one lens, and a lens aberration correction filter coefficient is calculated at each image height.

  Increasing the number of image heights selected from one lens, in other words, reducing the interval between selected image heights, will increase the possibility of performing fine blur correction using lens aberration correction filter coefficients. However, the number of lens aberration correction filter coefficients to be retained increases, and there is a high possibility that a large amount of memory capacity is required.

  However, as will be described later with reference to FIGS. 11 and 12, image height pixels for which lens aberration correction filter coefficients are not set can be corrected by interpolation. Also, with reference to FIG. Since the image height for calculating the lens aberration correction filter coefficient is determined by a method as will be described, the interval between the selected image heights can be widened to some extent.

  The intervals between the selected image heights can be equal intervals, and the intervals may be constant regardless of the lens. However, the number of lens aberration filter coefficients can be effectively reduced by detecting a characteristic image height as described with reference to FIG. An example of calculating the lens aberration correction filter coefficient of the image height will be described.

  FIG. 10 is a diagram illustrating a graph of MTF data. The vertical axis of the graph of MTF data illustrated in FIG. 10 represents MTF data, and the horizontal axis represents image height. Referring to the graph shown in FIG. 10, the MTF data changes depending on the image height and is not a constant value.

  As shown in FIG. 10, the MTF data may change depending on the image height. Therefore, the image height that changes drastically is selected as the image height for calculating the lens aberration correction filter coefficient.

  In FIG. 10, the image height 0 is selected because it is the center position of the lens. The image height 2 is selected because it is the image height at which the MTF data falls. Since the distance between the image height 0 and the image height 2 is separated, the image height 1 is selected as the image height at the approximate center.

  Image height 3 is selected because it is a turning point where the MTF data once lowered at image height 2 rises again and falls again. The image height 4 is selected because it is the end of the lens. For example, image points A to D in FIG. 6 are image points at the position of the image height 4, MTF data is acquired from a plurality of positions on the lens corresponding to the image height 4, and the lens is processed by the above-described processing. An aberration correction filter coefficient is calculated.

  In FIG. 10, although MTF data is used, PSF data may be used. Similarly, in the case of PSF data, the image height at which the shape of the PSF greatly changes may be selected.

  As described above, as the image height for calculating the lens aberration correction filter coefficient, an inflection point at which the MTF data changes rapidly is selected. In addition, since the center and the end of the lens are also characteristic positions in the lens, the image height at that position is also selected as the image height for calculating the lens aberration correction filter coefficient.

  Further, when the number of image heights selected by such processing is small, or when the interval between the selected image heights is wide, an additional image height may be selected so as to be an appropriate interval. .

  A plurality of image points are selected from the image height selected in this manner, and MTF data or PSF data is acquired from the selected image points, whereby lens aberration correction filter coefficients are calculated.

  The graph of MTF data and the graph of PSF data may differ depending on the lens. Therefore, by acquiring data for each lens, selecting an image height, and calculating a lens aberration correction filter coefficient, it is possible to calculate a lens aberration correction filter coefficient corresponding to a different characteristic for each lens. Therefore, the blur correction can be performed with higher accuracy.

  The lens aberration correction filter coefficient calculated in this way is calculated, for example, when the lens is manufactured, and is stored in association with the lens. For example, in the case of the main body of the imaging apparatus 100 (FIG. 1) and a detachable lens (optical lens 111), the housing portion of the optical lens 111 is provided with a holding function such as a memory, and the lens aberration correction filter coefficient is obtained by the holding function. Is retained.

  In this case, as shown in FIG. 2, the lens aberration correction filter coefficient is supplied from the side of the external optical lens 111 different from the signal processing unit 113.

  When the optical lens 111 and the imaging apparatus 100 are integrated, the lens aberration correction unit 221 (FIG. 2) may be configured to hold the lens aberration correction filter coefficient.

  Alternatively, the lens aberration correction filter coefficient may be calculated by the user when the imaging apparatus 100 is being used, instead of being calculated when the optical lens 111 is manufactured.

  For example, a test pattern or the like is captured by the user with the imaging apparatus 100, PSF data or MTF data is acquired from the captured image, and the lens aberration correction filter coefficient is calculated by the method described above. It is also possible to configure.

  As described above, when the lens aberration correction filter coefficient is calculated at the time of use, it is possible to appropriately update the lens aberration correction filter coefficient in accordance with a change in the use environment or a secular change. Further, when the optical lens 111 is a replaceable imaging device, the processing for calculating the lens aberration correction filter coefficient may be executed and held when the optical lens 111 is replaced. Is possible.

  Alternatively, when the optical lens 111 is manufactured, a lens aberration correction filter coefficient is calculated and associated with the IP address and managed on the side where the optical lens 111 is manufactured. If IPv6 (Internet Protocol Version 6) technology is used, an IP address can be assigned to the optical lens 111. Moreover, even if it is not an IP address, the optical lens 111 can be uniquely identified, for example, a manufacturing number may be used.

  In the case of such a configuration, as an example, when the optical lens 111 is attached to the imaging apparatus 100 and the imaging apparatus 100 is connected to the network, lens aberration correction associated with an IP address, a manufacturing number, or the like is performed. The filter coefficient can be downloaded and stored on the imaging apparatus 100 side.

  As described above, the lens aberration correction filter coefficient is calculated for each optical lens 111 and is supplied or held to the lens aberration correction unit 221 of the imaging apparatus 100.

<Lens aberration correction filter coefficient interpolation>
Next, processing for correcting blur using lens aberration correction filter coefficients respectively corresponding to a plurality of image heights will be described with reference to FIGS. 11 and 12.

  FIG. 11 is a diagram for explaining the position of the image height selected for calculating the lens aberration correction filter coefficient. In the example shown in FIG. 11, an example in which five image heights are selected is shown. These five locations correspond to the image heights 0 to 4 shown in FIG. A graph of MTF data is shown on the lower side of FIG. 11, and this graph is the same as the graph shown in FIG.

  The image height 0 in the graph shown in FIG. 10 is shown as the image height ih0 in the graph shown in FIG. Similarly, the image height 1 is the image height ih1, the image height 2 is the image height ih2, the image height 3 is the image height ih3, and the image height 4 is the image height ih4.

  In the example shown in FIG. 11, the image height ih4 is located on the image circle 303 of the optical lens 111. The lens aberration correction filter coefficients for the image heights ih0 to 4 are calculated and held.

  When such an image height and a lens aberration correction filter coefficient are set, blur correction at the coordinates (x, y) of the optical lens 111 is performed based on the following equation (9). Since the lens correction filter coefficient is not set in the image height of the pixel located at the coordinates (x, y) to be corrected, the position is located at the coordinates (x, y) based on the following equation (9). The pixel value of the pixel to be calculated is calculated.

In Expression (9), deblur_r (x, y) represents a pixel value after correction at coordinates (x, y). α represents a blending coefficient. ih _n indicates an image height position, and is, for example, the image heights ih0 to ih4 in FIG.

deblur_ih _n is an image height ihn in which an adjacent lens correction filter coefficient is set closer to the center of the lens than the coordinates (x, y), and the lens correction filter coefficient Filter set to the image height ihn. The pixel value calculated by applying ihn (i, j) to the pixel value Pr (i, j) of the pixel located at the coordinates (x, y) is represented.

deblur_ih _{n + 1} is an image height ihn + 1 in which an adjacent lens correction filter coefficient is set on the side farther from the center of the lens than the coordinates (x, y), and is set to the image height ihn + 1. Represents the pixel value calculated by applying the lens correction filter coefficient Filter ihn + 1 (i, j) to the pixel value Pr (i, j) of the pixel located at the coordinates (x, y).

The blend coefficient α is described as α ₁₀ , α ₂₁ , α ₃₂ , and α ₄₃ in the upper diagram of FIG. That is, when the image height between the image height ih0 to image height ih1 are is used blending coefficient alpha _10, when the image height between the image height ih1 to image height ih2, the blend coefficient alpha ₂₁ is used, when the image height between the image height ih2 to image height ih3, the blend coefficient alpha ₃₂ is used when the image height between the image height ih3 to image height ih4, the blend coefficient alpha ₄₃ is used.

The blend coefficient α is calculated according to the image height, as shown in FIG. 12, instead of using the same value between adjacent image heights for which the lens aberration correction filter coefficients are set. Position of the pixel to be corrected, the distance r from the center of the optical lens 111, the distance r is is a image height ih0 above, is smaller than the image height IH1, blending coefficient alpha ₁₀ is calculated. The blending coefficient alpha ₁₀ from a distance r, a value obtained by subtracting the image height Ih0, are divided by the value obtained by subtracting the image height Ih0 from the image height IH1.

When the position of the pixel to be corrected is equal to or higher than the image height ih1 and smaller than the image height ih2, the blend coefficient α ₂₁ is calculated. The blending coefficient alpha ₂₁ from a distance r, a value obtained by subtracting the image height IH1, is a value obtained by dividing the value obtained by subtracting the image height IH1 from the image height ih2.

Position of the pixel to be corrected, and the image height ih2 above, is smaller than the image height IH3, blending coefficient alpha ₃₂ is calculated. The blend coefficient α ₃₂ is a value obtained by dividing the value obtained by subtracting the image height ih2 from the distance r by the value obtained by subtracting the image height ih2 from the image height ih3.

Position of the pixel to be corrected, and the image height ih3 above, when the image height IH4, blending coefficient alpha ₄₃ is calculated. The blend coefficient α ₄₃ is a value obtained by dividing the value obtained by subtracting the image height ih3 from the distance r by the value obtained by subtracting the image height ih3 from the image height ih4.

  As described above, the blend coefficient α is a value calculated by using the image height in which two lens aberration correction filter coefficients adjacent to the position are set according to the position of the pixel to be corrected. That is, the blend coefficient α is a value calculated by performing a linear calculation according to the distance r (x, y) from the center of the optical lens 111, as shown in FIG.

Referring to Expression (9), the pixel value deblur_r (x, y) after the correction of the distance r (x, y) from the center of the optical lens 111 is a lens aberration correction filter coefficient that fluctuates around the position of the distance r. Is applied to the pixel value Pr to be corrected, and the pixel value deblur_ih after the filter application is applied to the image height ih _n and the image height ih _{n + 1. n} is calculated using pixel value deblur_ih _{n + 1} .

  As shown in Expression (9), the lens correction filter coefficient that is set to the image height adjacent to the image height of the pixel that is the correction target is applied to the pixel value of the pixel that is the correction target. Lens aberration correction is performed by blending the corrected pixel value calculated in accordance with the image height position of the pixel for which no lens aberration correction filter coefficient is set.

  Reference is now made to FIG. 2 again. An xy address indicating the coordinate position of each pixel associated with the pixel value of each pixel is supplied from the image sensor 112 to the image height calculation unit 201. The image height calculation unit 201 derives the distance r (x, y) from the center of the optical lens 111, that is, the image height, from the supplied xy address.

  The blend coefficient calculation unit 202 sets the blend coefficient α corresponding to the image height calculated by the image height calculation unit 201 to the distance r (x, y from the center of the optical lens 111 as described with reference to FIG. ) Is calculated by a linear calculation according to). The calculated blend coefficient α is supplied to the blend processing unit 222 of the lens aberration correction processing unit 204.

While processing related to the calculation of the blend coefficient α is performed, the lens aberration correction unit 221 of the lens aberration correction processing unit 204 acquires the supplied lens aberration correction filter coefficient, and the acquired lens The blur correction of the pixel value to be processed is performed using the aberration correction filter coefficient. The lens aberration correction unit 221 calculates the pixel value deblur ih _n and the pixel value deblur ih _{n + 1} in Equation (9).

  As described with reference to FIGS. 3 to 5, the lens aberration correction unit 221 performs a convolution operation that multiplies the pixel values of the pixels included in the 7 × 7 pixels by each coefficient of the lens aberration correction filter coefficient and adds them. Then, the correction pixel value of the center pixel is calculated.

  The blend processing unit 222 uses the two corrected pixel values supplied from the lens aberration correction unit 221 and the blend coefficient α supplied from the blend coefficient calculation unit 202 to perform a calculation based on Expression (9), The pixel value after final lens aberration correction is calculated and output to the edge / flat portion blend processing unit 206.

  In this way, blur correction according to the image height is performed.

  Here, after the lens aberration is corrected by the lens aberration correction unit 221, blend processing using the corrected pixel value is performed, and the final lens aberration corrected pixel value for the pixel to be corrected An example in which is calculated is shown.

  As another configuration, the lens correction filter coefficient set at each image height position is blended using the blend coefficient α, the lens correction filter coefficient after blending is calculated, and the lens correction filter coefficient after blending is used. The lens aberration correction corresponding to the image height may be performed by performing the filtering process.

In this case, correction is performed by calculation based on the following equation (10).

  In Expression (10), Filter blend (i, j) represents a lens correction filter coefficient to be applied to a pixel that is a correction target. Filter ihn represents a lens correction filter coefficient set to the image height ihn, and Filter ihn + 1 represents a lens correction filter coefficient set to the image height ihn + 1. The image height ihn and the image height ihn + 1 are image heights adjacent to the image height of the pixel to be corrected.

  As shown in Expression (10), by blending two lens correction filter coefficients set to an image height adjacent to the image height of the pixel to be corrected, the pixel to be corrected A lens correction filter coefficient to be applied is calculated. The blend coefficient α used for the blend is a coefficient calculated as described with reference to FIG. 12, and is calculated and used in the same manner as described above.

  deblur r (x, y) represents a pixel value after correction of a pixel to be corrected. This is calculated by using the blended lens correction filter coefficient Filter blend (i, j) as the pixel value of the pixel to be corrected.

  In this way, when the lens correction filter coefficients are blended and then applied to the pixel value to be corrected, the configuration in the lens aberration correction processing unit 204 of the signal processing unit 113 shown in FIG. The lens aberration correction unit 221 performs correction after the filter coefficient is blended in the blend processing unit 222.

  According to the present technology, it is possible to perform lens aberration correction processing optimized for the MTF characteristics of the lens. Further, lens aberration correction corresponding to the image height can be performed without using a large-scale system. As a result, it is possible to correct lens aberration corresponding to image height in an image sensor (imaging device) that cannot use a large-scale system.

  In addition, by selecting the image height from the inflection points of the MTF curve that depends on the image height, it is possible to perform highly accurate lens aberration correction even with a small number of image height points, and by reducing the capacity of the memory that holds the lens correction filter coefficient, etc. It can be realized with a large-scale circuit.

  By approximating the PSF data with a function, it is possible to reduce the circuit scale, and it is possible to create a lens aberration correction filter coefficient that suppresses the influence of manufacturing variations.

<About recording media>
The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

  FIG. 13 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processing by a program. In a computer, a CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are connected to each other by a bus 1004. An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a storage unit 1008, a communication unit 1009, and a drive 1110 are connected to the input / output interface 1005.

  The input unit 1006 includes a keyboard, a mouse, a microphone, and the like. The output unit 1007 includes a display, a speaker, and the like. The storage unit 1008 includes a hard disk, a nonvolatile memory, and the like. The communication unit 1009 includes a network interface. The drive 1110 drives a removable medium 1111 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

  In the computer configured as described above, the CPU 1001 loads, for example, the program stored in the storage unit 1008 to the RAM 1003 via the input / output interface 1005 and the bus 1004 and executes the program. Is performed.

  The program executed by the computer (CPU 1001) can be provided by being recorded on a removable medium 1111 as a package medium, for example. The program can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  In the computer, the program can be installed in the storage unit 1008 via the input / output interface 1005 by attaching the removable medium 1111 to the drive 1110. Further, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. In addition, the program can be installed in advance in the ROM 1002 or the storage unit 1008.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  In addition, the effect described in this specification is an illustration to the last, and is not limited, Moreover, there may exist another effect.

  The embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present technology.

  In addition, this technique can also take the following structures.

(1)
An acquisition unit that acquires a filter that performs blur correction set to each of a plurality of image heights;
An image processing apparatus comprising: a correction unit that corrects a pixel value of a pixel at an image height to be corrected using the filter acquired by the acquisition unit.
(2)
The correction unit is
Applying the filter set to an image height adjacent to the image height that is the correction target to a pixel value of the pixel that is the correction target;
A coefficient is calculated from the positional relationship between the image height that is the correction target and the adjacent image height,
The image processing apparatus according to (1), wherein a pixel value after correction is calculated using the pixel value after application of the filter and the coefficient.
(3)
The correction unit is
A coefficient is calculated from the positional relationship between the image height that is the correction target and the adjacent image height,
Using the filter set to an image height adjacent to the image height that is the correction target and the coefficient, a filter that is applied to the pixel value of the pixel that is the correction target is generated,
The image processing apparatus according to (1), wherein a corrected pixel value is calculated using the generated filter and a pixel value of the pixel to be corrected.
(4)
The filter coefficients are:
PSF (Point Spread Function) data is calculated from a plurality of image points on the first image height,
The PSF data is averaged;
The averaged PSF data is used and approximated by a predetermined function,
The image processing apparatus according to any one of (1) to (3), wherein the coefficient is a coefficient calculated from the approximated PSF data.
(5)
The calculation of the coefficient of the filter uses a Wiener filter. The image processing apparatus according to (4).
(6)
The image processing apparatus according to (4) or (5), wherein the image height for which the filter is set is an image height at which the shape of the PSF changes greatly.
(7)
The filter coefficients are:
MTF (Modulation Transfer Function) data is calculated from a plurality of image points on the first image height,
The MTF data is averaged;
The averaged MTF data is used and approximated by a predetermined function,
The image processing apparatus according to any one of (1) to (3), wherein the coefficient is a coefficient calculated from the approximated MTF data.
(8)
The image processing apparatus according to (7), wherein the filter coefficient is calculated using a Wiener filter.
(9)
The image processing apparatus according to (7) or (8), wherein the image height for which the filter is set is an inflection point of the MTF data.
(10)
Get a filter that performs blur correction set to multiple image heights,
An image processing method including a step of correcting a pixel value of a pixel at an image height to be corrected using the acquired filter.
(11)
Get a filter that performs blur correction set to multiple image heights,
A computer-readable program for executing a process including a step of correcting a pixel value of a pixel at an image height to be corrected using the acquired filter.
(12)
An image sensor;
An acquisition unit that acquires a filter that performs blur correction set to each of a plurality of image heights;
An imaging apparatus comprising: a correction unit that corrects a pixel value of a pixel at an image height that is a correction target among pixels captured by the imaging device using the filter acquired by the acquisition unit.
(13)
An acquisition unit that acquires a filter that performs blur correction set to each of a plurality of image heights;
Using the filter acquired by the acquisition unit, a correction unit that corrects the pixel value of the pixel at the image height that is the correction target;
An electronic device comprising: an image processing unit that performs predetermined image processing on the pixel value after correction from the correction unit.

  DESCRIPTION OF SYMBOLS 100 Imaging device, 111 Optical lens, 112 Image sensor, 113 Signal processing part, 131 Image signal correction part, 132 RGB signal processing part, 201 Image height calculation part, 202 Blend coefficient calculation part, 203 Line memory, 204 Lens aberration correction process Section, 205 edge detection section, 206 edge / flat section blend processing section

Claims (13)

An acquisition unit that acquires a filter that performs blur correction set to each of a plurality of image heights;
An image processing apparatus comprising: a correction unit that corrects a pixel value of a pixel at an image height to be corrected using the filter acquired by the acquisition unit. The correction unit is
Applying the filter set to an image height adjacent to the image height that is the correction target to a pixel value of the pixel that is the correction target;
A coefficient is calculated from the positional relationship between the image height that is the correction target and the adjacent image height,
The image processing apparatus according to claim 1, wherein a pixel value after correction is calculated using the pixel value after application of the filter and the coefficient. The correction unit is
A coefficient is calculated from the positional relationship between the image height that is the correction target and the adjacent image height,
Using the filter set to an image height adjacent to the image height that is the correction target and the coefficient, a filter that is applied to the pixel value of the pixel that is the correction target is generated,
The image processing apparatus according to claim 1, wherein a corrected pixel value is calculated using the generated filter and a pixel value of the pixel to be corrected. The filter coefficients are:
PSF (Point Spread Function) data is calculated from a plurality of image points on the first image height,
The PSF data is averaged;
The averaged PSF data is used and approximated by a predetermined function,
The image processing apparatus according to claim 1, wherein the coefficient is calculated from the approximated PSF data. The image processing apparatus according to claim 4, wherein the filter coefficient is calculated using a Wiener filter. The image processing apparatus according to claim 4, wherein the image height for which the filter is set is an image height at which the shape of the PSF changes greatly. The filter coefficients are:
MTF (Modulation Transfer Function) data is calculated from a plurality of image points on the first image height,
The MTF data is averaged;
The averaged MTF data is used and approximated by a predetermined function,
The image processing apparatus according to claim 1, wherein the coefficient is calculated from the approximated MTF data. The image processing apparatus according to claim 7, wherein the filter coefficient is calculated using a Wiener filter. The image processing apparatus according to claim 7, wherein the image height for which the filter is set is an inflection point of the MTF data. Get a filter that performs blur correction set to multiple image heights,
An image processing method including a step of correcting a pixel value of a pixel at an image height to be corrected using the acquired filter. Get a filter that performs blur correction set to multiple image heights,
A computer-readable program for executing a process including a step of correcting a pixel value of a pixel at an image height to be corrected using the acquired filter. An image sensor;
An acquisition unit that acquires a filter that performs blur correction set to each of a plurality of image heights;
An imaging apparatus comprising: a correction unit that corrects a pixel value of a pixel at an image height that is a correction target among pixels captured by the imaging device using the filter acquired by the acquisition unit. An acquisition unit that acquires a filter that performs blur correction set to each of a plurality of image heights;
Using the filter acquired by the acquisition unit, a correction unit that corrects the pixel value of the pixel at the image height that is the correction target;
An electronic device comprising: an image processing unit that performs predetermined image processing on the pixel value after correction from the correction unit.

Images