JP2017183775A - Image processing apparatus, image processing method, and image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an image processing method, and an image pickup device capable of suppressing deterioration of image quality.SOLUTION: The image processing apparatus comprises: an acquisition part which acquires, from among plural correction coefficients associated respectively to a first axis and a second axis intersecting each other, two or more correction coefficients in accordance with the positional coordinates of a correction target pixel; and a correction part which corrects an image by using the correction coefficient acquired by the acquisition part.SELECTED DRAWING: Figure 1

Description

  The present disclosure relates to an image processing apparatus that performs image correction, an image processing method, and an imaging device.

  In the field of imaging devices such as digital cameras, techniques for correcting images (correcting image blur) according to the image height dependence of lenses and MTF characteristics have been proposed (for example, Patent Documents 1 and 2).

JP2015-216576A JP 2011-193277 A

  However, in the above-described method of Patent Document 1, correction processing is performed using the same correction coefficient (filter coefficient) for pixel values at the same image height. For this reason, it is difficult to obtain a sufficient correction effect when the resolution varies due to a lens manufacturing error or the like. Patent Document 2 discloses a technique in which a user performs correction using a parameter set prepared in advance using software. However, even in this technique, the above-described variation in resolution occurs. In some cases, sufficient correction cannot be performed. This leads to image quality degradation.

  It is desired to realize an image processing apparatus, an image processing method, and an image sensor that can suppress deterioration in image quality even when the lens resolution varies.

  An image processing apparatus according to an embodiment of the present disclosure includes two or more correction coefficients corresponding to the position coordinates of a correction target pixel among a plurality of correction coefficients associated with each of a first axis and a second axis that intersect each other. An acquisition unit that acquires a correction coefficient and a correction unit that performs an image correction process using the correction coefficient acquired by the acquisition unit are provided.

  According to an image processing method of an embodiment of the present disclosure, two or more correction coefficients corresponding to each of the first axis and the second axis intersecting each other are selected according to the position coordinates of the correction target pixel. A correction coefficient is acquired, and an image correction process is performed using the acquired correction coefficient.

  An imaging device according to an embodiment of the present disclosure includes a pixel unit including a plurality of pixels and a signal processing unit that performs signal processing on an image obtained from the pixel unit. A signal processing unit that acquires two or more correction coefficients according to the position coordinates of the correction target pixel from a plurality of correction coefficients associated with the first axis and the second axis that intersect with each other; A correction unit that performs an image correction process using the correction coefficient acquired by the acquisition unit.

  In the image processing device, the image processing method, and the imaging device according to the embodiment of the present disclosure, the position of the correction target pixel from among a plurality of correction coefficients associated with the first axis and the second axis that intersect each other. Two or more correction coefficients are acquired according to the coordinates, and image correction processing is performed using the acquired correction coefficients. Thus, correction can be performed using a correction coefficient optimized according to the position coordinates of the pixel.

  In the image processing device, the image processing method, and the imaging device according to the embodiment of the present disclosure, the position of the correction target pixel from among a plurality of correction coefficients associated with the first axis and the second axis that intersect each other. Two or more correction coefficients are acquired according to the coordinates, and image correction processing is performed using the acquired correction coefficients. As a result, a sufficient correction effect can be obtained even when resolution variations occur due to lens manufacturing errors or the like using the optimized correction coefficient. Therefore, even when the lens resolution varies, the image quality deterioration can be suppressed.

  The above content is an example of the present disclosure. The effects of the present disclosure are not limited to those described above, and may be other different effects or may include other effects.

It is a figure showing functional composition of an image sensor concerning a 1 embodiment of this indication. It is a schematic diagram for demonstrating the arrangement | positioning relationship between a pixel part and a signal processing part. It is a schematic diagram showing an example of a correction target pixel (G pixel). It is a schematic diagram showing an example of the correction coefficient of the correction target pixel shown in FIG. 3A. It is a schematic diagram showing an example of a correction target pixel (R pixel). FIG. 4B is a schematic diagram illustrating an example of a correction coefficient of the correction target pixel illustrated in FIG. 4A. It is a schematic diagram showing an example of a correction target pixel (B pixel). FIG. 5B is a schematic diagram illustrating an example of a correction coefficient of the correction target pixel illustrated in FIG. 5A. It is a schematic diagram showing a correspondence with a correction coefficient, an image height, and an effective pixel area. It is a characteristic view showing MTF data for explaining setting of an image height position. It is a schematic diagram for demonstrating the MTF data shown in FIG. It is a schematic diagram for demonstrating an example of a correction coefficient blend process (interpolation process). It is a figure showing the example of functional composition in the case of performing friend processing after pixel value amendment. It is a schematic diagram showing the correspondence between the correction coefficient according to Modification 1 and the image height and effective pixel area. It is a figure showing the flow of the correction coefficient calculation which concerns on the modification 2, and a signal processing.

Hereinafter, modes for carrying out the present disclosure will be described in detail with reference to the drawings. The description will be given in the following order.
1. Embodiments: Configuration: Configuration of imaging device, configuration of signal processing unit (image processing apparatus), correction coefficient Modifications 1 and 2

<Embodiment>
[Constitution]
FIG. 1 illustrates a functional configuration of an imaging device (imaging device 1) according to an embodiment of the present disclosure together with a lens (lens 30). The imaging device 1 is an image sensor such as a charge coupled device image sensor (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor, and is applied to, for example, a still camera or a video camera. The image sensor 1 is a signal that performs various signal processing including image correction processing on the pixel unit 10 that receives the passing light L of the lens 30 and the image signal (imaging signal D0) output from the pixel unit 10, for example. And a processing unit 20. In addition, although not shown, the image pickup device 1 includes a control unit that drives and controls each unit of the pixel unit and the signal processing unit, and a memory.

  The pixel unit 10 includes, for example, a plurality of pixels, for example, R (red), G (green), and B (blue) pixels (sub-pixels) arranged two-dimensionally in the effective pixel region. Each of the R, G, and B pixels has an RGB array 10a made of, for example, a Bayer array. Specifically, the pixel unit 10 is provided with a color filter having an RGB array 10a. In the present embodiment, the case where the pixel unit 10 has the RGB array 10a will be described. However, the present disclosure is configured to have an array other than RGB, for example, an RGBW array including R, G, B, and W (white). It can also be applied to. Alternatively, the pixel unit 10 may have, for example, an array of Y (yellow), C (cyan), M (magenta), and the like. The present disclosure can be applied to color arrangements other than the arrangements exemplified here.

  With the above configuration, in the pixel unit 10, any of the RGB color lights is received in units of pixels via the lens 30, and an electrical signal corresponding to the received light signal intensity is generated by photoelectric conversion. As a result, a mosaic image composed of three types of RGB spectra is obtained by the pixel unit 10.

  The pixel unit 10 and the signal processing unit 20 are stacked, for example, as schematically shown in FIG. Specifically, the imaging element 1 has a structure (so-called stacked type element) in which a substrate 1A on which a pixel unit 10 is formed and a substrate 2A on which a signal processing unit 20 is stacked (bonded) one above the other. Structure). Note that the “signal processing unit 20” in the present embodiment corresponds to a specific example of the “image processing apparatus” in the present disclosure. Also, the “image processing method” of the present disclosure is embodied by the configuration and operation of the image processing apparatus (signal processing unit 20), and thus description thereof is omitted.

  In the present embodiment, the configuration in which the signal processing unit 20 is provided in the image sensor 1 will be described as an example. However, the image processing apparatus of the present disclosure is applied to such an image sensor 1. The apparatus is not limited to the lens 30 and the pixel unit 10, and may be a personal computer (PC) or the like. In this case, in addition to the configuration of the signal processing unit 20 illustrated in FIG. 1, the configuration further includes an interface unit to which imaging data is input and a memory in which the input imaging data is stored.

(Configuration of the signal processing unit 20)
The signal processing unit 20 includes, for example, a position coordinate calculation unit 21 (a distance calculation unit 21A and an angle calculation unit 21B), a correction coefficient acquisition unit 22, an image height coefficient calculation unit 23A, a quadrant coefficient calculation unit 23B, and a correction A coefficient blend processing unit 24, a lens aberration correction processing unit 25, a line memory 26, an edge detection unit 27, an edge / flat part blend processing unit 28, and an RGB signal processing unit 29 are provided. The correction coefficient acquisition unit 22 of the present embodiment corresponds to a specific example of the “acquisition unit” of the present disclosure, and the correction coefficient blend processing unit 24 and the lens aberration correction processing unit 25 correspond to the “correction unit of the present disclosure. To a specific example. The correction coefficient blend processing unit 24 corresponds to a specific example of the “correction coefficient calculation unit” of the present disclosure, and the lens aberration correction processing unit 25 corresponds to a specific example of the “correction processing unit” of the present disclosure.

  The position coordinate calculation unit 21 includes, for example, a distance calculation unit 21A and an angle calculation unit 21B. Based on the xy address indicating the coordinate position of each pixel associated with the pixel value of each pixel of the pixel unit 10, The position coordinates of the correction target pixel (hereinafter referred to as pixel Pt) are detected. The distance calculation unit 21A calculates the distance of the pixel Pt (the distance r from the optical center, the distance corresponding to the image height). Information on the calculated distance r is supplied to the correction coefficient acquisition unit 22 and the image height coefficient calculation unit 23A, respectively. The angle calculation unit 21B calculates a quadrant position (angle d) in the image area of the pixel Pt. Information about the calculated angle d is supplied to the correction coefficient acquisition unit 22 and the quadrant coefficient calculation unit 23B, respectively.

  The correction coefficient acquisition unit 22 selects two or more correction coefficients (Fr, Fg, Fb) from among a plurality of correction coefficients (Fr, Fg, Fb) according to the position coordinates (distance r, angle d) of the pixel Pt calculated by the position coordinate calculation unit 21. A correction coefficient (correction coefficient group Fgrp) is selected (acquired). Each of the correction coefficients Fr, Fg, and Fb is composed of, for example, a plurality of filter coefficients. Information on the acquired correction coefficient group Fgrp is supplied to the correction coefficient blend processing unit 24. The correction coefficients Fr, Fg, and Fb are set for each color of R, G, and B, respectively, and may be held in the imaging device 1 in advance or may be input from the outside. Details of these correction coefficients Fr, Fg, and Fb will be described later.

  The inter-image height coefficient calculation unit 23A calculates a coefficient (Ar) used in blend processing (interpolation processing) between image heights based on the distance r calculated by the distance calculation unit 21A. Information regarding the calculated coefficient Ar is supplied to the correction coefficient blend processing unit 24.

  The quadrant coefficient calculation unit 23B calculates a coefficient (Ad) used in the blending process (interpolation process) between quadrants based on the angle d calculated by the angle calculation unit 21B. Information about the calculated coefficient Ad is supplied to the correction coefficient blend processing unit 24.

  The correction coefficient blend processing unit 24 performs processing for generating a correction coefficient (correction coefficient F1) for the pixel Pt using the correction coefficient group Fgrp acquired by the correction coefficient acquisition unit 22. In other words, the processing unit generates the correction coefficient F1 for the pixel Pt at the position (image point) where the correction coefficients Fr, Fg, and Fb are not associated with each other by interpolation. Specifically, an interpolation process using each correction coefficient constituting the correction coefficient group Fgrp, the coefficient Ar calculated by the inter-image height coefficient calculation unit 23A, and the coefficient Ad calculated by the inter-quadrant coefficient calculation unit 23B. The correction coefficient F1 of the pixel Pt is calculated by (blending processing). The correction coefficient F1 is calculated for each RGB color. Information about the calculated correction coefficient F1 is supplied to the lens aberration correction processing unit 25.

  The lens aberration correction processing unit 25 uses correction coefficients F1 (specifically, correction coefficients F1r, F1g, and F1b for each color) for each color signal (Rin, Gin, Bin) input from the line memory 26. Correction processing is performed, and the corrected image signal D1 (Rd, Gd, Bd) is calculated. The calculated image signal D1 is supplied to the edge / flat portion blend processing unit. The lens aberration correction processing unit 25 can perform, for example, a process for correcting image blur, particularly a correction process in consideration of aberration variations depending on the position coordinates (image height and quadrant position) of the pixel Pt.

  The line memory 26 temporarily holds the pixel signal output from the pixel unit 10. For example, the line memory 26 is configured to store pixel values for seven horizontal lines of the pixel unit 10 and sequentially output seven horizontal line data in parallel. The output destinations are the lens aberration correction processing unit 25, the edge detection unit 27, and the edge / flat portion blend processing unit. Imaging data having the RGB array 10a is output in units of 7 lines to each of these processing units.

  Here, the line memory 26 is described as outputting in units of 7 lines, but this is not particularly limited, but a correction coefficient (filter) described later corresponds to, for example, 7 × 7 pixels. This is an example of a case. For this reason, when the correction coefficient corresponds to the number of pixels other than 7 × 7 pixels, the line memory 26 can store and output in units of lines corresponding to the number of pixels of the correction coefficient. It is good to be done.

  Based on the output signal from the line memory 26, the edge detection unit 27 generates information related to edges in the image (including, for example, the edge direction and edge strength), and outputs the information to the edge / flat part blend processing unit 28. Specifically, for example, the flatness (Weightflat) calculated from the pixel information of 7 × 7 pixels is calculated based on the processing target pixel (the center pixel of the 7 × 7 pixels), and the edge / flat portion blending process is performed. To the unit 28. In addition, as this flatness calculation process, the process similar to the process described in Unexamined-Japanese-Patent No. 2011-55038 can be used, for example.

  The edge / flat portion blend processing unit 28 includes RGB signals (Rin, Gin, Bin) in the input signal from the line memory 26, edge information output from the edge detection unit 27, and output from the lens aberration correction processing unit 25. Using the image signal D1 (Rd, Gd, Bd) to be generated, the image signal D2 after the correction process considering the edge information is generated and output to the RGB signal processing unit 29. Specifically, the edge / flat portion blend processing unit 28 is configured to perform lens aberration correction processing according to the edge information calculated by the edge detection unit 27 (for example, flatness calculated from pixel information of 7 × 7 pixels). The weighted average processing of the corrected image signal D1 (Rd, Gd, Bd) output from 25 is executed to calculate the RGB pixel values of the RGB array 10a. Specifically, RGB pixel values are determined according to the following three equations.

R = (Weightflat) * (Rd) + (1-Weightflat) * Rin
G = (Weightflat) * (Gd) + (1-Weightflat) * Gin
B = (Weightflat) * (Bd) + (1-Weightflat) * Bin

  The R, G, and B color signals obtained as the calculation result of this expression are output to the RGB signal processing unit 29. The RGB signal processing unit 29 performs signal processing on the signals of the RGB array 11a output from the edge / flat portion blend processing unit 28 as signal processing on the signals of the RGB array (Bayer array) 11a to generate a color image. . The RGB signal processing unit 29 performs, for example, white balance adjustment processing, demosaic processing, shading processing, RGB color matrix processing, gamma correction processing, and the like to generate a color image.

  The image signal Dout for the color image generated by the RGB signal processing unit 29 is output to the outside or recorded in a memory inside the apparatus.

  A control unit (not shown) executes control of a series of processes in each of the above processing units. For example, a program for executing a series of processes is stored in a memory (not shown), and the control unit executes the program read from the memory and controls the series of processes. The memory can be composed of various recording media such as a magnetic disk, an optical disk, and a flash memory.

(About correction factor)
The correction coefficients Fr, Fg, and Fb are either stored in advance as lens aberration correction filter coefficients or input from the outside. These correction coefficients Fr, Fg, and Fb can be calculated using, for example, a Wiener filter. The winner filter will be briefly described. In the following, three images, an ideal image without blur (original image A1), a blurred photographed image (captured image A2), and a restored image (restored image A3) restored by filter correction processing for the photographed image are represented. Assuming this will be explained.

  Of the above three images, a filter that minimizes the square error of two images of the original image A1 and the restored image A3 is called a least square filter or a Wiener filter. Here, the original image A1 is f (x, y), the photographed image A2 is g (x, y), the degradation function due to lens aberration or camera shake is h (x, y), and the noise component is n (x, y). Then, the following formula (1) is established. Note that (x, y) is the pixel position of each image, and f (x, y) to n (x, y) may indicate the pixel value of the coordinate position (x, y) of each image. Further, it is assumed that the deterioration function h (x, y) due to lens aberration and camera shake is a fixed value.

  When both sides of the above formula (1) are Fourier transformed, the following formula (2) is obtained. However, G (u, v), H (u, v), F (u, v), and N (u, v) are g (x, y), h (x, y), and f (x, y) represents the Fourier transform of n (x, y).

   G (u, v) = H (u, v) × F (u, v) + N (u, v) (2)

  Assuming that there is no zero point in the degradation function due to lens aberration or camera shake, and the noise component is known, the above equations (2) to F (u, v) can be expressed as the following equation (3).

  However, since the noise component is generally unknown, the above equation (3) cannot be solved exactly. Therefore, correction is performed using a Wiener filter K (u, v) of the following equation (4) that minimizes the error between the original image A1 and the restored image A3. Note that (Sn (u, v) / Sf (u, v)) corresponds to the power spectral density of the original image A1 and the noise component.

  As shown in the following formula (5), the result of inverse Fourier transform of the Wiener filter represented by formula (4) (k (x, y)) and the captured image A2 are convolved in real space. F ′ (x, y) of the restored image A3 that has been corrected can be obtained.

  An example of the Wiener filter corresponding to the above equation (5) is shown in FIGS. 3A and 3B. FIG. 3A shows 7 × 7 pixels centered on the G pixel in the photographed image, and FIG. 3B shows an example of a Wiener filter in which the G pixel at the center of the pixel region in FIG. 3A is the correction target pixel Pt (pixel Pgt). It is shown.

  Here, the captured image is a RAW image composed of RGB pixels. The correction target pixel Pgt is the center G pixel of the 7 × 7 pixels shown in FIG. 3A. The Wiener filter shown in FIG. 3B shows a multiplication coefficient for each of a plurality of G pixels included in a 7 × 7 pixel. A correction pixel value of the central G pixel is calculated by executing a convolution operation that multiplies each coefficient by the pixel value of the G pixel included in the 7 × 7 pixels and adds the result. Specifically, the pixel value of the G pixel can be corrected by the following formula (6).

  Similarly, FIG. 4A shows a 7 × 7 pixel area centered on the R pixel in the captured image, and FIG. 4B shows the R pixel at the center of the pixel area in FIG. 4A as the correction target pixel Pt (pixel Prt). An example of the Wiener filter is shown. FIG. 5A shows a 7 × 7 pixel area centered on the B pixel in the captured image, and FIG. 5B shows a Wiener filter in which the B pixel at the center of the pixel area in FIG. 5A is the correction target pixel Pt (pixel Pbt). An example is shown. The correction value is calculated for each of the R pixel and the B pixel in the same manner as in the case of the G pixel.

  When calculating the correction coefficient, lens PSF (Point Spread Function) data or MTF (Modulation Transfer Function) data is used. Correction coefficients calculated according to the characteristics of the lens 30 are stored in advance as correction coefficients Fr, Fg, and Fb, or are input from the outside.

(Detailed configuration example of correction coefficient)
In the present embodiment, a plurality of correction coefficients Fr, Fg, and Fb are associated (set) with each of the first axis and the second axis that intersect each other. The following description will be made using the correction coefficient Fg for correcting the G pixel as a representative. A plurality of correction coefficients Fg are set for one lens 30. An example is shown in FIG. FIG. 6 shows a correction filter corresponding to 7 × 7 pixels as an example of the correction coefficient.

  For example, a plurality of correction coefficients Fg are associated with each of the X axis and the Y axis orthogonal to each other. For example, four correction coefficients f11 to f14 (coefficient sequence q1) are set in the positive direction on the X axis, and for example, four correction coefficients f31 to f34 (coefficient sequence q3) are set in the negative direction on the X axis. Is set. For example, four correction coefficients f21 to f24 (coefficient sequence q2) are set in the positive direction on the Y axis, and for example, four correction coefficients f41 to f44 (coefficient sequence q4) are set in the negative direction on the Y axis. Is set. Note that the X-axis (first axis) and the Y-axis (second axis) intersect (here, orthogonal) at the optical axis (optical center). A correction coefficient f0 is set at the optical center (point where the image height is 0).

  The correction coefficient f11 is set, for example, to the image height ih1 (image point at the position of the image height ih1) closest to the optical axis (close to the optical axis) in the coefficient sequence q1. The correction coefficient f12 is set to the image height ih2 next to (one outside) the image height ih1 in the coefficient sequence q1. The correction coefficient f13 is set to the image height ih3 next to (one outside) the image height ih2 in the coefficient sequence q1. The correction coefficient f14 is set to the image height ih4 next to (one outside) the image height ih3 in the coefficient sequence q1.

  Similarly, the correction coefficient f21 is set to the image height ih1 in the coefficient sequence q2. The correction coefficient f22 is set to the image height ih2 in the coefficient sequence q2. The correction coefficient f23 is set to the image height ih3 in the coefficient sequence q2. The correction coefficient f24 is set to the image height ih4 in the coefficient sequence q2. The correction coefficient f31 is set to the image height ih1 in the coefficient sequence q3. The correction coefficient f32 is set to the image height ih2 in the coefficient sequence q3. The correction coefficient f33 is set to the image height ih3 in the coefficient sequence q3. The correction coefficient f34 is set to the image height ih4 in the coefficient sequence q3. The correction coefficient f41 is set to the image height ih1 in the coefficient sequence q4. The correction coefficient f42 is set to the image height ih2 in the coefficient sequence q4. The correction coefficient f43 is set to the image height ih3 in the coefficient sequence q4. The correction coefficient f44 is set to the image height ih4 in the coefficient sequence q4.

  Here, as an example, a case where the image height ih4 corresponds to the image circle (outermost end) of the lens 30 will be described as an example.

  Thus, in the present embodiment, a plurality of correction coefficients Fg are set for each of a plurality of image heights and a plurality of angles (rows on the X axis and the Y axis). In the example shown in FIG. 6, a plurality of correction coefficients Fg (specifically, correction coefficients f11 to f14, f21 to f24, and f31 to 34) are obtained at the position coordinates of the four image heights and the four angular directions. , F41 to f44) are set. These correction coefficients Fg can be calculated by a method using a Wiener filter as described above. In the present embodiment, two or more correction coefficients (correction coefficient group Fgrp) are acquired according to the position coordinates of the pixel Pt from the plurality of correction coefficients Fg associated on two axes in this way. It has become.

  The same applies to the correction coefficients Fr and Fb, and a plurality of correction coefficients Fr and Fb are associated with each other on two axes (first axis and second axis). Note that, for each of R, G, and B, a plurality of correction coefficients need only be associated with two intersecting axes, and these two axes do not necessarily have to be orthogonal (may be oblique). . For example, two axes may be arranged along the diagonal line of the effective pixel region 301.

  These correction coefficients Fr, Fg, and Fb are calculated, for example, in the manufacturing process of the lens 30 or the manufacturing process of the camera module, and are held in association with the lens 30. For example, when the lens 30 is detachable from the camera body, the housing portion of the lens 30 has a holding function such as a memory, and correction coefficient information is held by the holding function. In this case, the correction coefficients Fr, Fg, and Fb are supplied from the lens 30 side outside the signal processing unit 20. Alternatively, when the lens 30 is integrated with the camera, the correction coefficients Fr, Fg, and Fb may be held in the signal processing unit 20 in advance. As described above, the correction coefficient is calculated for each lens 30 and is stored in the signal processing unit 20 in advance or input from the outside.

(Image height setting)
As described above, the correction coefficient is calculated for a single lens 30 at a selective number of image heights (four image heights ih1 to ih4 in FIG. 6), but this number of image heights is increased. In this case, the correction accuracy using the correction coefficient is improved by narrowing the image height interval. However, the number of correction coefficients to be retained increases and the memory capacity increases. For this reason, it is desirable that the number of image heights is set based on a balance between correction accuracy and memory capacity. The region between the image heights can be calculated by interpolation as will be described later.

  The set image height intervals can be set to be equal intervals, and the intervals may be constant irrespective of lens characteristics. Alternatively, a characteristic image height (image height 0 to 4) may be detected as shown in FIGS. 7 and 8 without being equally spaced. Thereby, the number of correction coefficients can be effectively reduced. 7 and 8 are diagrams showing MTF data with respect to image height. FIG. 8 also shows the image circle 303 and the effective pixel region 301. As described above, the MTF data changes depending on the image height. Therefore, an image height at an image point with a relatively large degree of change is selected, and a correction coefficient is calculated at the selected image height.

  Specifically, image height 0 is selected as the center position of the lens. The image height 2 is selected because it is the image height at the point where the MTF data falls. Since the distance between the image height 0 and the image height 2 is separated, the image height 1 is selected as the image height at the approximate center. Image height 3 is selected because it is a turning point where the MTF data once lowered at image height 2 rises again and falls again. Image height 4 is selected as the end of the lens. For example, MTF data or PSF data may be acquired from a plurality of positions on the lens 30 corresponding to these image heights 0 to 4, and the correction coefficient may be calculated by the processing as described above.

  As described above, the image height at which the correction coefficient is calculated may be the image height at the image point that is the inflection point in the MTF data. In addition, since the lens center and the edge are also characteristic positions in the lens, the image height at the position may be selected. Furthermore, when the number of image heights serving as feature points is small, or when the interval between selected image heights is wide, the image height may be added so as to be an appropriate interval.

  MTF data for image height may vary from lens to lens. Therefore, it is desirable that data is acquired for each lens, an image height is selected, and a lens aberration correction filter coefficient is calculated. Lens aberration correction filter coefficients corresponding to different characteristics for each lens can be calculated, and correction processing can be performed with higher accuracy.

[Action, effect]
In the imaging device 1 described above, the light beam that has passed through the lens 30 enters the pixel unit 10, whereby a pixel signal is obtained for each pixel by photoelectric conversion. At this time, the pixel unit 10 acquires R, G, B color RAW data corresponding to the RGB array 10a as the imaging signal D0. The obtained imaging signal D0 is output to the signal processing unit 20.

  The signal processing unit 20 performs image correction processing based on the input imaging signal D0. Specifically, first, the position coordinate calculation unit 21 calculates the position coordinate of the pixel Pt to be corrected based on the xy address indicating the coordinate position of each pixel associated with the pixel value of each pixel of the pixel unit 10. To detect. Specifically, the distance calculation unit 21A calculates the distance r of the pixel Pt. Information on the calculated distance r is output to each of the correction coefficient acquisition unit 22 and the image height coefficient calculation unit 23A. Further, the angle calculation unit 21B calculates the quadrant position (angle d) in the image area of the pixel Pt. Information about the calculated angle d is output to the correction coefficient acquisition unit 22 and the quadrant coefficient calculation unit 23B, respectively.

  The correction coefficient acquisition unit 22 selects two or more correction coefficients from a plurality of correction coefficients (here, the correction coefficient Fg is taken as an example) according to the input position coordinates (distance r, angle d) of the pixel Pt. (Correction coefficient group Fgrp) is selected (obtained). Information on the acquired correction coefficient group Fgrp is supplied to the correction coefficient blend processing unit 24.

  The image height coefficient calculation unit 23A calculates a coefficient Ar to be used in the blend processing (interpolation processing) between image heights based on the input distance r. Information about the calculated coefficient Ar is output to the correction coefficient blend processing unit 24. The quadrant coefficient calculation unit 23B calculates a coefficient Ad used in the blend process (interpolation process) between quadrants based on the input angle d. Information about the calculated coefficient Ad is output to the correction coefficient blend processing unit 24.

  The correction coefficient blend processing unit 24 generates a correction coefficient F1 for the pixel Pt by interpolation processing using the input correction coefficient group Fgrp. Specifically, the correction coefficient F1 (first correction coefficient) of the pixel Pt is calculated by interpolation processing (blend processing) using the correction coefficients constituting the correction coefficient group Fgrp and the coefficients Ar and Ad.

FIG. 9 shows an example of correction coefficient blending processing. In this way, interpolation processing is performed between angles (between quadrants and axes) and between image heights in the XY plane. Specifically, the correction coefficient F1 of the center pixel Pt of 7 × 7 pixels is calculated by interpolation processing using a correction coefficient group Fgrp including two or more correction coefficients Fg and the coefficients Ar and Ad. The image height ih _n corresponds to any one of the image heights ih1 to ih4 (image height position) shown in FIG. 6, for example, and the image height ih _{n + 1} is ₁ of the image height ih _n . It corresponds to the outer image height. Here, the pixel Pt will be described as a G pixel located in the first quadrant in the XY plane.

In this example, the correction coefficient acquisition unit 22 uses a plurality of correction coefficients Fg (correction coefficients f11 to f14, f21 to f24, f31 to 34, and f41 to f44 shown in FIG. 6) to indicate the position coordinates ( Two or more appropriate correction coefficients Fg (correction coefficient group Fgrp) are selected (acquired) according to the distance r and the angle d). The correction coefficient group Fgrp includes, for example, two correction coefficients F0 _n and F0 _{n + 1} (any two correction coefficients in the coefficient sequence q1 illustrated in FIG. 6) set in the positive direction on the X axis, and, for example, And two correction coefficients F90 _n and F90 _{n + 1} (any two correction coefficients in the coefficient sequence q2 shown in FIG. 6) set in the positive direction on the Y-axis.

Of these correction coefficients _{F0 n, F0 n + 1,} F90 n, F90 n + 1, for example, the image height ih _n correction coefficients associated with the on F0 _n, F90 _n, calculated based on the angle d Using the coefficient Ad (0 ≦ Ad <1), the correction coefficient blend process is performed from the following equation (6) by linear calculation. Thereby, the blending process between quadrants (between axes) is performed on the image height ih _n , and the correction coefficient F2 (second correction coefficient) is calculated.

F2 = (1−Ad) × F0 _n + Ad × F90 _n (6)

Similarly, linearity is calculated using correction coefficients F0 _{n + 1} and F90 _{n + 1 associated} with the image height ih _{n + 1} and a coefficient Ad (0 ≦ Ad <1) calculated based on the angle d. The correction coefficient blending process is performed from the following equation (7) by calculation. Thereby, a blending process between quadrants (between axes) is performed on the image height ih _{n + 1} , and a correction coefficient F3 (third correction coefficient) is calculated.
F3 = (1-Ad) · F0 _{n + 1} + Ad × F90 _{n + 1} (7)

Subsequently, these correction coefficients F2 and F3 are obtained from the following equation (8) by linear calculation using a coefficient Ar (0 ≦ Ar <1) calculated based on the distance r (distance from the optical center). The correction coefficient blending process is performed. Thereby, the blending process between the image heights is performed, and the correction coefficient F1 of the pixel Pt at the position of the distance r and the angle d is calculated. That is, the correction coefficient of the pixel Pt is optimized.
F1 = (1-Ar) × F2 + Ad × F3 (8)

  Information about the correction coefficient F1 calculated by the interpolation process (blend process) as described above is output to the lens aberration correction processing unit 25.

  The lens aberration correction processing unit 25 corrects the pixel value in the pixel Pt using the correction coefficient F1 (as an example) for each color signal (Rin, Gin, Bin) input from the line memory 26. Lens aberration correction processing). Specifically, when the correction coefficient F1 corresponds to 7 × 7 pixels, a calculation process of multiplying and adding the 7 × 7 pixels is performed, and the pixel of the center pixel Pt is executed. Correct the value. Assuming that the pixel value of the pixel Pt at the distance r is Pr, the pixel value Pr1 after lens aberration correction is calculated by the following equation (9).

  In this way, the pixel value of the pixel Pt is corrected. Similarly, for the R and B pixels, after blending correction coefficients between quadrants and between image heights to calculate a correction coefficient F1, correction processing using the correction coefficient F1 (lens aberration correction) Process).

  In this embodiment, the correction processing is performed on the pixel values after blending (interpolating) correction coefficients between quadrants and between image heights. However, the blend processing (interpolation processing) It may be performed after the correction. FIG. 10 shows a functional block configuration in that case. In other words, the lens aberration correction processing unit 25a corrects the pixel value of the pixel Pt to be corrected using a plurality of correction coefficients associated on two axes. Thereafter, the blend processing unit 26a may perform interpolation processing between quadrants and between image heights based on the corrected pixel values. Further, the order in which the blending process and the correction process are performed may be determined according to the data amount of the correction coefficient.

  Thereby, it is possible to perform an image correction process, particularly a correction process taking into account aberration variations depending on the position coordinates (image height and quadrant position) of the pixel Pt. The image signal D1 (Rd, Gd, Bd) after the correction processing by the lens aberration correction processing unit 25 is output to the edge / flat portion blend processing unit.

  The edge / flat portion blend processing unit 28 receives RGB signals (Rin, Gin, Bin) input from the line memory 26, edge information input from the edge detection unit 27, and lens aberration correction processing unit 25. Using the image signal D1 (Rd, Gd, Bd), an image signal D2 after correction processing in consideration of edge information is generated. Specifically, the edge / flat portion blend processing unit 28 is configured to perform lens aberration correction processing according to the edge information calculated by the edge detection unit 27 (for example, flatness calculated from pixel information of 7 × 7 pixels). The weighted average processing of the corrected image signal D1 (Rd, Gd, Bd) output from 25 is executed to calculate the RGB pixel values in the RGB array 10a. Specifically, RGB pixel values are determined according to the following three equations.

R = (Weightflat) * (Rd) + (1-Weightflat) * Rin
G = (Weightflat) * (Gd) + (1-Weightflat) * Gin
B = (Weightflat) * (Bd) + (1-Weightflat) * Bin

  The R, G, and B color signals obtained as the calculation result of this expression are output to the RGB signal processing unit 29. The generated image signal D2 is output to the RGB signal processing unit 29.

  The RGB signal processing unit 29 performs signal processing on the input RGB array (Bayer array) signal, for example, white balance adjustment processing, demosaic processing, shading processing, RGB color matrix processing, gamma correction processing, and the like to generate a color image. Generate. An image signal Dout for the generated color image is output to the outside or recorded in a memory inside the apparatus.

  As described above, in the present embodiment, the position coordinates of the pixel Pt to be corrected are selected from among a plurality of correction coefficients associated with the first axis and the second axis (X axis, Y axis) that intersect each other. Two or more correction coefficients are acquired according to (distance r, angle d), and image correction processing is performed using the acquired correction coefficients. Thereby, the correction coefficient can be optimized according to the position coordinates of the pixel Pt.

  Here, in the photographed image based on the light beam passing through the lens 30, the blurring that occurs varies depending on the characteristics of the lens 30. For example, as the image height (distance from the optical center) increases, the blur of the image due to lens aberration increases. Further, due to a manufacturing error of the lens and the like, not only the image height of the lens but also a variation in aberration occurs in each region (quadrant) in the XY plane orthogonal to the optical axis, resulting in a variation in resolution.

  As described above, even image points located at the same image height have different degrees of blur of the lens 30 and are often not isotropic. This is caused by, for example, a lens manufacturing error. If the resolution of the lens 30 varies, even if correction processing is performed, a location where correction is insufficient or an overcorrected location occurs depending on the pixel position, leading to image quality degradation.

  On the other hand, in the present embodiment, when the blur condition of the lens 30 is not isotropic, particularly when blur occurs only on one side of the lens 30 in the vertical direction or the horizontal direction (when so-called blur occurs). Particularly effective signal processing is realized.

  As described above, in the imaging device 1 according to the present embodiment, a pixel to be corrected is selected from a plurality of correction coefficients associated with the first axis and the second axis (X axis and Y axis) that intersect each other. Two or more correction coefficients are acquired according to the position coordinates (distance r, angle d) of Pt, and image correction processing is performed using the acquired correction coefficients. As a result, the correction coefficient can be optimized, and a sufficient correction effect can be obtained even when the resolution varies due to a lens manufacturing error or the like. Therefore, even when the lens resolution varies, the image quality deterioration can be suppressed.

  In addition, overcorrection of the pixel value can be suppressed particularly in a region where the resolution of the lens 30 is high, which is advantageous in suppressing image quality deterioration. Furthermore, the manufacturing yield of the camera module can be improved. In addition, since correction processing is performed in consideration of the resolution variation of the lens 30 inside the image sensor 1, adjustment work by the user is also unnecessary.

  Next, a modification of the above embodiment will be described. Below, the same code | symbol is attached | subjected about the component similar to the said embodiment, and the description is abbreviate | omitted.

<Modification 1>
FIG. 11 is a schematic diagram illustrating a correspondence between a correction coefficient (any one of Fg, Fr, and Fb) according to the first modification, an image height, and an effective pixel area. In the above embodiment, as illustrated in FIG. 6, the configuration in which the coefficient sequences q1 to q4 are arranged along the X axis and the Y axis is illustrated. However, as in the present modification, the coefficient sequences q1 to q4 are arranged. q4 may be associated with two axes rotated 45 ° counterclockwise from the X axis and the Y axis, respectively. In this case, it is effective when the lens blur occurs along an oblique direction. Note that the rotation angle of the shaft is not limited to 45 °, and can be set in a range from 0 ° to 90 ° depending on a region where a lens blur occurs.

<Modification 2>
FIG. 12 shows a flow of correction coefficient calculation and signal processing according to the second modification. In the above embodiment, the correction coefficients Fr, Fg, and Fb are calculated in the manufacturing process of the lens 30 or the manufacturing process of the camera module, and are stored in the circuit in advance as calibration information, or are externally (lens 30 ). However, the signal processing (image processing) of the present disclosure can also be applied when there is no calibration information regarding such a lens 30 (for example, when the lens 30 is an old lens).

  For example, the correction coefficient may be input by the user. Specifically, as shown in FIG. 12, after the lens 30 is attached to the camera body (step S21), a test chart (test pattern) is photographed by the image sensor 1 (step S22). Subsequently, lens aberration characteristics (for example, PSF data or MTF data) are acquired from the captured image (step S23). A correction coefficient may be calculated based on the acquired lens aberration characteristic (step S24), and the lens aberration may be corrected using this correction coefficient.

  As described above, the correction coefficient may be calculated not when the lens 30 is manufactured but when the camera is used. In this case, an old lens having no calibration information can be used as the lens 30. In addition, it is possible to appropriately update the correction coefficient in response to a change in the usage environment or aging of the lens 30. Further, when the lens 30 is replaceable with respect to the camera body, it is possible to cope with the case where the lens 30 is replaced with another lens.

  Although the present disclosure has been described with reference to the embodiment and its modifications, the present disclosure is not limited to the above-described embodiment and the like, and various modifications can be made. For example, the series of signal processing described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, for example, a program constituting the software is installed in the computer. Here, the computer includes, for example, a general-purpose personal computer capable of executing various functions by installing various programs by installing a computer incorporated in dedicated hardware.

  Moreover, the effect described in this specification is an example, The other effect may be sufficient and the other effect may be included.

In addition, this indication can also take the following structures.
(1)
An acquisition unit that acquires two or more correction coefficients according to the position coordinates of the correction target pixel from among a plurality of correction coefficients associated with each of the first axis and the second axis that intersect each other;
An image processing apparatus comprising: a correction unit that performs an image correction process using the correction coefficient acquired by the acquisition unit.
(2)
The correction unit is
A correction coefficient calculation unit that calculates a first correction coefficient at the position coordinates by an interpolation process using the correction coefficient acquired by the acquisition unit;
The image processing apparatus according to (1), further including: a correction processing unit that corrects a pixel value corresponding to the position coordinates using the first correction coefficient calculated by the correction coefficient calculation unit.
(3)
The image processing apparatus according to (2), wherein the correction unit performs a first interpolation process based on an angle of the position coordinates with respect to the first axis or the second axis.
(4)
The image processing apparatus according to (3), wherein the correction unit performs a second interpolation process based on an image height of the position coordinates before or after the first interpolation process.
(5)
In the first interpolation process,
By interpolating the correction coefficient associated with the first image height on the first axis and the correction coefficient associated with the first image height on the second axis according to the angle, 2 correction factor is calculated,
A correction coefficient associated with the second image height adjacent to the first image height on the first axis and a correction coefficient associated with the second image height on the second axis are the angles. The image processing apparatus according to (4), wherein the third correction coefficient is calculated by performing interpolation according to the above.
(6)
In the second interpolation process, after the first interpolation process,
The image processing apparatus according to (5), wherein the first correction coefficient is calculated by interpolating the second correction coefficient and the third correction coefficient in accordance with an image height of the position coordinates.
(7)
The correction unit is
Correct the pixel value using the correction coefficient acquired by the acquisition unit;
The image processing apparatus according to (1), wherein the image is corrected by an interpolation process based on the corrected pixel value.
(8)
The acquisition unit acquires a plurality of correction coefficients associated with the first axis and the second axis for each color,
The image processing apparatus according to any one of (1) to (7), wherein the correction unit performs an image correction process using the acquired correction coefficient for each color.
(9)
From the plurality of correction coefficients associated with each of the first axis and the second axis intersecting each other, two or more correction coefficients are acquired according to the position coordinates of the correction target pixel,
An image processing method for correcting an image using an acquired correction coefficient.
(10)
When performing the correction process of the image,
A first correction coefficient at the position coordinates is calculated by an interpolation process using the acquired correction coefficient,
The image processing method according to (9), wherein the pixel value corresponding to the position coordinate is corrected using the calculated first correction coefficient.
(11)
The image processing method according to (10), wherein the first interpolation processing is performed based on an angle of the position coordinates with respect to the first axis or the second axis.
(12)
The image processing method according to (11), wherein the second interpolation processing is performed based on the image height of the position coordinates before or after the first interpolation processing.
(13)
In the first interpolation process,
By interpolating the correction coefficient associated with the first image height on the first axis and the correction coefficient associated with the first image height on the second axis according to the angle, 2 correction factor is calculated,
A correction coefficient associated with the second image height adjacent to the first image height on the first axis and a correction coefficient associated with the second image height on the second axis are the angles. The image processing method according to (12), wherein the third correction coefficient is calculated by performing interpolation according to the method.
(14)
In the second interpolation process, after the first interpolation process,
The image processing method according to (13), wherein the first correction coefficient is calculated by interpolating the second correction coefficient and the third correction coefficient in accordance with an image height of the position coordinates.
(15)
Correct the pixel value using the acquired correction coefficient,
The image processing method according to (9), wherein the image is corrected by an interpolation process based on the corrected pixel value.
(16)
Obtaining a plurality of correction coefficients associated with the first axis and the second axis for each color;
The image processing method according to any one of (9) to (15), wherein image correction processing is performed using the acquired correction coefficient for each color.
(17)
According to the data amount of a plurality of correction coefficients associated with the first axis and the second axis,
After calculating the first correction coefficient at the position coordinate by the interpolation process using the acquired correction coefficient, the pixel value corresponding to the position coordinate is corrected by using the first correction coefficient. Perform corrections, or
The pixel value is corrected using the acquired correction coefficient, and it is determined whether the correction processing of the image is performed by the interpolation processing based on the corrected pixel value. Any one of the above (9) to (16) The image processing method as described.
(18)
A pixel portion including a plurality of pixels;
A signal processing unit that performs signal processing on an image obtained from the pixel unit,
The signal processing unit
An acquisition unit that acquires two or more correction coefficients according to the position coordinates of the correction target pixel from among a plurality of correction coefficients associated with each of the first axis and the second axis that intersect each other;
An image sensor comprising: a correction unit that performs correction processing of the image using the correction coefficient acquired by the acquisition unit.
(19)
The image sensor according to (18), wherein the pixel unit and the signal processing unit are stacked.

  DESCRIPTION OF SYMBOLS 1 ... Image pick-up element, 10 ... Pixel part, 20 ... Signal processing part, 21 ... Position coordinate calculation part, 21A ... Distance calculation part, 21B ... Angle calculation part, 22 ... Correction coefficient acquisition part, 23A ... Inter-image height coefficient calculation part , 23B ... quadrant coefficient calculation unit, 24 ... correction coefficient blend processing unit, 25 ... lens aberration correction processing unit, 26 ... line memory, 27 ... edge detection unit, 28 ... edge / flat part blend processing unit, 29 ... RGB signal Processing part.

Claims (19)

An acquisition unit that acquires two or more correction coefficients according to the position coordinates of the correction target pixel from among a plurality of correction coefficients associated with each of the first axis and the second axis that intersect each other;
An image processing apparatus comprising: a correction unit that performs an image correction process using the correction coefficient acquired by the acquisition unit. The correction unit is
A correction coefficient calculation unit that calculates a first correction coefficient at the position coordinates by an interpolation process using the correction coefficient acquired by the acquisition unit;
The image processing apparatus according to claim 1, further comprising: a correction processing unit that corrects a pixel value corresponding to the position coordinates using the first correction coefficient calculated by the correction coefficient calculation unit. The image processing apparatus according to claim 2, wherein the correction unit performs a first interpolation process based on an angle of the position coordinates with respect to the first axis or the second axis. The image processing apparatus according to claim 3, wherein the correction unit performs a second interpolation process based on an image height of the position coordinates before or after the first interpolation process. In the first interpolation process,
By interpolating the correction coefficient associated with the first image height on the first axis and the correction coefficient associated with the first image height on the second axis according to the angle, 2 correction factor is calculated,
A correction coefficient associated with the second image height adjacent to the first image height on the first axis and a correction coefficient associated with the second image height on the second axis are the angles. The image processing apparatus according to claim 4, wherein the third correction coefficient is calculated by performing interpolation according to. In the second interpolation process, after the first interpolation process,
The image processing apparatus according to claim 5, wherein the first correction coefficient is calculated by interpolating the second correction coefficient and the third correction coefficient in accordance with an image height of the position coordinates. The correction unit is
Correct the pixel value of the correction target pixel using the correction coefficient acquired by the acquisition unit,
The image processing apparatus according to claim 1, wherein the image correction process is performed by an interpolation process based on the corrected pixel value. The acquisition unit acquires a plurality of correction coefficients associated with the first axis and the second axis for each color,
The image processing apparatus according to claim 1, wherein the correction unit performs an image correction process using the acquired correction coefficient for each color. From the plurality of correction coefficients associated with each of the first axis and the second axis intersecting each other, two or more correction coefficients are acquired according to the position coordinates of the correction target pixel,
An image processing method for correcting an image using an acquired correction coefficient. When performing the correction process of the image,
A first correction coefficient at the position coordinates is calculated by an interpolation process using the acquired correction coefficient,
The image processing method according to claim 9, wherein a pixel value corresponding to the position coordinate is corrected using the calculated first correction coefficient. The image processing method according to claim 10, wherein a first interpolation process is performed based on an angle of the position coordinates with respect to the first axis or the second axis. The image processing method according to claim 11, wherein a second interpolation process is performed based on an image height of the position coordinates before or after the first interpolation process. In the first interpolation process,
By interpolating the correction coefficient associated with the first image height on the first axis and the correction coefficient associated with the first image height on the second axis according to the angle, 2 correction factor is calculated,
A correction coefficient associated with the second image height adjacent to the first image height on the first axis and a correction coefficient associated with the second image height on the second axis are the angles. The image processing method according to claim 12, wherein the third correction coefficient is calculated by performing interpolation according to. In the second interpolation process, after the first interpolation process,
The image processing method according to claim 13, wherein the first correction coefficient is calculated by interpolating the second correction coefficient and the third correction coefficient in accordance with an image height of the position coordinates. Correct the pixel value of the correction target pixel using the acquired correction coefficient,
The image processing method according to claim 9, wherein the image is corrected by an interpolation process based on the corrected pixel value. Obtaining a plurality of correction coefficients associated with the first axis and the second axis for each color;
The image processing method according to claim 9, wherein an image correction process is performed using the acquired correction coefficient for each color. According to the data amount of a plurality of correction coefficients associated with the first axis and the second axis,
After calculating the first correction coefficient at the position coordinate by the interpolation process using the acquired correction coefficient, the pixel value corresponding to the position coordinate is corrected by using the first correction coefficient. Perform corrections, or
The image processing method according to claim 9, wherein a pixel value of the correction target pixel is corrected using the acquired correction coefficient, and it is determined whether the correction process of the image is performed by an interpolation process based on the corrected pixel value. . A pixel portion including a plurality of pixels;
A signal processing unit that performs signal processing on an image obtained from the pixel unit,
The signal processing unit
An acquisition unit that acquires two or more correction coefficients according to the position coordinates of the correction target pixel from among a plurality of correction coefficients associated with each of the first axis and the second axis that intersect each other;
An image sensor comprising: a correction unit that performs correction processing of the image using the correction coefficient acquired by the acquisition unit. The imaging device according to claim 18, wherein the pixel unit and the signal processing unit are stacked.

Images