JP5769790B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image correction technique.

  An image obtained by imaging with an imaging apparatus such as a digital camera is affected by lateral chromatic aberration and distortion aberration of an optical system such as an imaging lens. When attention is focused on the pixels in the captured image, the image that should originally be focused on the target pixel is moved and formed on a line connecting the target pixel and the center of the image due to each aberration.

  In recent years, multi-pixel imaging devices are used in imaging devices, or optical systems with wide-angle or high zoom magnification are used, and image quality degradation due to aberrations becomes obvious. Some processes correct the influence of aberrations. It is known that the correction amount for correcting the information of the pixel moved by the aberration to the pixel that should be originally obtained is determined by the image height representing the distance from the center of the image and the characteristics of the optical system.

  In addition, the movement of the image due to aberration is not necessarily the pixel unit of the image sensor, but may be moved between pixels of the image sensor (missing pixel), and interpolation calculation is necessary to correct the influence of aberration. . Patent Documents 1 and 2 disclose a technique for deriving a pixel value of a pixel of interest after correction by multiplying a plurality of pixels by an interpolation function in order to correct lateral chromatic aberration and distortion in a captured image. Has been.

JP 2001-186533 A JP 2005-057605 A

However, in the above-described technique, a processing method for correcting the influence of aberration has been disclosed, but the correction limit limited by the capacity of the storage area when performing the correction processing has not been mentioned so far. In other words, in an imaging apparatus including a multi-pixel imaging element as described above, an image obtained by imaging is also increased in capacity. For this reason, in order to apply image processing such as correcting the influence of aberration to the obtained image, it is necessary to divide the image into a number of regions and read out each region in turn into a storage region for correction processing. There is. In addition, in order to shorten the time taken from imaging to recording, particularly during video recording, it is necessary to avoid reading of overlapping areas as much as possible, and image processing can be completed with one reading for one area. Is required. That is, since the aberration correction amount is limited by the capacity of the storage area, it is necessary to apply a suitable correction in consideration of the correction limit.
The present invention has been made in view of the above-described problems, and an object thereof is to suitably correct image quality deterioration.

In order to achieve the above object, an image processing apparatus of the present invention comprises the following arrangement.
Storage means for reading out a partial area of the first image and storing it as a second image, and a signal of a peripheral pixel centered on a position different from the target pixel in the signal value of the target pixel in the second image And an arithmetic means that calculates using an interpolation coefficient that multiplies the signal value of the peripheral pixel and the signal value of the peripheral pixel, and the arithmetic means is at least one of a predetermined number of peripheral pixels centered at a position different from the target pixel. The interpolation coefficient is set to a different value between the case where it is not stored as the second image and the case where all of a predetermined number of peripheral pixels centered at positions different from the target pixel are stored as the second image. It is characterized by that.

  With this configuration, according to the present invention, it is possible to suitably correct image quality degradation.

The figure showing the functional composition of the digital camera concerning an embodiment The figure for demonstrating the pixel coordinate generation circuit which concerns on embodiment. Approximate function derivation method for calculating correction amount The figure showing the circuit composition of the coefficient generation circuit concerning an embodiment The figure for demonstrating the deriving method of the pixel value after correction | amendment Diagram for explaining frequency response characteristics of interpolation function The figure showing the circuit composition of the frequency characteristic control circuit concerning an embodiment Diagram for explaining the suppression coefficient of the interpolation function The figure for demonstrating the deriving method of the pixel value after correction | amendment which concerns on embodiment. The figure showing the circuit composition of the optical correction circuit concerning an embodiment A diagram for explaining reference pixels on a buffer memory used for interpolation calculation The figure showing the relationship between the image height and correction amount which concerns on embodiment The figure for demonstrating the relationship between the correction limit amount and each parameter which concerns on embodiment The figure showing the circuit composition of the amendment characteristic information generation circuit concerning Embodiment 3. The figure showing the circuit composition of the feature detection circuit concerning Embodiment 4.

(Embodiment 1)
Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings. In one embodiment described below, the present invention is applied to a digital camera that can correct lateral chromatic aberration or distortion caused by an optical system of a captured image as an example of an image processing apparatus. An example will be described. However, the present invention can be applied to any device that can correct image quality degradation caused by the aberration of the optical system that captured the image in the input image.
Although the relationship between the correction amount and the image height in the lateral chromatic aberration and the distortion aberration is different, the correction amount of the aberration depends on the image height. Therefore, in the present embodiment, either the lateral chromatic aberration or the distortion aberration of each RGB color is used. The correction amount will be described below.

  FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. In the present embodiment, in the digital camera 100, in order to correct the influence due to the aberration generated in the captured image, at least a necessary functional configuration is shown and described, and components related to image recording and display are not illustrated. .

  The microcomputer 101 is a block that controls the overall processing of the digital camera 100, and exchanges data such as computations and computation results to each block. The microcomputer 101 includes a non-volatile memory that stores an operation program related to the entire processing of the digital camera 100, and a work memory that temporarily stores the execution of the operation program, the calculation result, and input data.

  The optical system 102 is composed of a lens group such as a zoom lens and a focus lens, and forms a subject image on reflected light from the subject via an optical system 102 and a diaphragm 106 on an image sensor 108 described later. The optical system 102 and the aperture 106 are driven by the zoom lens actuator 103, the focus lens actuator 104, and the aperture actuator 105 according to the drive position information supplied from the microcomputer 101, and optically adjust the subject image. The focal length, focus position, and aperture value, which are information for calculating the drive position information supplied to the optical system 102 and the aperture 106, are factors that affect the lateral chromatic aberration and distortion aberration. These corrections will be collectively referred to as optical parameters.

  The reflected light from the subject that has entered the digital camera 100 via the optical system 102 and the aperture 106 is color-separated into RGB by the color separation prism 107, and forms an image on the image sensor 108 for each color of RGB. The image sensor 108 is a CCD, a CMOS sensor, or the like. An optical image formed on each image sensor 108 is photoelectrically converted, and an obtained analog image signal is in accordance with a drive waveform input by an analog front end (AFE) 109. Are output to the AFE 109 in the scan order. The AFE 109 is a block that converts an input analog image signal into a digital image. The AFE 109 converts the analog image signal input from the RGB image pickup elements 108 into digital images Sr, Sg, and Sb, respectively. Output to the correction circuit 115. The drive waveform of the image sensor 108 generated by the AFE 109 to output the analog image signal from the image sensor 108 is based on the horizontal synchronization signal HD and the vertical synchronization signal VD output from the timing generator (TG) 110 to the AFE 109. The TG 110 receives information related to driving of the image sensor 108 included in the digital camera 100 from the microcomputer 101, generates a horizontal synchronization signal HD and a vertical synchronization signal VD, and outputs them to the AFE 109 and the pixel coordinate generation circuit 111. The information related to the driving of the image sensor 108 includes, for example, information about the time required for the image sensor 108 to output an analog image signal for one horizontal line, or an analog image signal required for outputting all analog image signals. This is information on the time required to read out the image signal.

  The pixel coordinate generation circuit 111 is a circuit that polarizes information on each pixel position in order to correct the influence of aberration. Since the lateral chromatic aberration and distortion depend on the image height, which is the distance from the optical center, the pixel coordinate generation circuit 111 converts the information of each pixel position into polar coordinates with the optical center as a pole. Here, the pixel coordinate generation circuit 111 will be further described with reference to FIG.

  FIG. 2A shows an example of the circuit configuration of the pixel coordinate generation circuit 111. By counting the horizontal synchronization signal HD and the vertical synchronization signal VD input from the TG 110, pixel position information Xt and Xy currently output to the AFE 109 on the image sensor 108 is obtained and input to the polar coordinate conversion circuit 201. The Further, information Cx and Cy of the pixel position at the optical center of the captured image supplied from the microcomputer 101 is input to the polar coordinate conversion circuit 201. In the polar coordinate conversion circuit 201, based on these input information, as shown in FIG. 2B, information on the radius Rt and the deflection angle θt, which are polar coordinates of the pixel currently output to the AFE 109, is Calculate with the formula.

  The polar coordinate information thus calculated is output to a coefficient generation circuit 113 described later. Note that the square root calculation represents a known method such as a bisection method or a square root method with a finite word length accuracy, and the atan calculation represents a high-order function approximation or a segmented low-order that divides the XY ratio into a predetermined range. It can also be realized in hardware by function approximation.

  The optical correction database 112 is a storage device such as a flash memory that stores optical correction characteristic information for aberrations of the optical system, that is, the optical parameters at the time of imaging described above. Since the storage capacity of the optical correction database 112 is finite, the optical correction characteristic information includes, for example, correction amounts of magnification chromatic aberration and distortion aberration of each color at discrete image height positions with respect to a preset combination of optical parameters. It is stored as the plot data shown. That is, when calculating a combination of optical parameters and a correction amount at an image height position that are not stored as optical correction characteristic information, the correction amount is stored from the optical correction characteristic information stored in the optical correction database 112. It is necessary to obtain an approximate function of the image height.

Here, a method for the microcomputer 101 to calculate plot data Cm1 to Cm4 of the approximate function of the correction amount and the image height corresponding to the optical parameters at the time of imaging from the optical correction characteristic information stored in the optical correction database 112 is illustrated. 3 will be described.
The microcomputer 101 acquires, for example, optical parameters at the time of imaging when the imaging is performed according to a user instruction, and two optical correction characteristic information Ca, which has a high similarity to the combination of optical parameters at the time of imaging, from the optical correction database 112. Read Cb. The read optical correction characteristic information is plot data (Ca1 to Ca4, Cb1 to Cb4) indicating correction amounts at four image height positions in two different approximate functions as shown in FIG. In FIG. 3 (a), each plot data is shown connected by a curve, but is connected to identify the respective optical correction characteristic information Ca and Cb. In the actual plot data, There is no data between plots. The microcomputer 101 performs weighting according to the degree of divergence between the optical parameters at the time of imaging and the read optical parameters of the two optical correction characteristic information Ca and Cb, and based on the two optical characteristic information as shown in FIG. Optical correction characteristic information in the optical parameters at the time of imaging is calculated. The microcomputer 101 outputs plot data Cm1 to Cm4 calculated from the two pieces of optical correction characteristic information Ca and Cb to the coefficient generation circuit 113 as optical correction characteristic information of optical parameters at the time of imaging.

  The coefficient generation circuit 113 derives an approximate function of the correction amount and the image height in the optical parameter at the time of imaging between the plots from the input plot data Cm of the optical correction characteristic information of the optical parameter at the time of imaging. Further, the coefficient generation circuit 113 calculates the correction amount of the influence due to the aberration of the optical system 102 at the input pixel position, using the derived correction amount and the approximate function of the image height.

Here, the circuit configuration of the coefficient generation circuit 113 will be described with reference to FIG. The coefficient generation circuit 113 includes a function coefficient calculation circuit 401, a correction value calculation circuit 402, and an XY vector coefficient calculation circuit 403.
The function coefficient calculation circuit 401 calculates the coefficient of the approximate function of the correction amount and the image height for each plot section from the plot data of the optical correction characteristic information of the optical parameter at the time of imaging. In this embodiment, an approximate function between plots is derived from the plot data of the optical correction characteristic information as a quadratic function of the image height Rt as follows.

Zt = aRt ² + bRt + c
That is, the function coefficient calculation circuit 401 outputs the three coefficients a, b, and c of the derived approximate function to the correction value calculation circuit 402.

  In the present embodiment, a method for deriving the approximate function of the correction amount and the image height as a quadratic function will be described. However, the approximate function may be approximated by a linear function or a high-order function. For example, when the approximate function is a cubic function, four coefficients are output to the correction value calculation circuit 402. In this embodiment, a method for deriving the approximate function of the correction amount and the image height from the plot data in the function coefficient calculation circuit 401 will be described. However, the optical correction database 112 stores the coefficients of the approximate function in each optical parameter. It may be. In that case, the coefficient generation circuit 113 does not need to have the function coefficient calculation circuit 401, and if the coefficient of the approximate function in the optical parameters at the time of imaging input to the coefficient generation circuit 113 is directly input to the correction value calculation circuit 402. Good.

  The correction value calculation circuit 402 is based on the coefficients of the approximate function of each plot section input from the function coefficient calculation circuit 401 and the image height information of the pixel of interest input from the pixel coordinate generation circuit 111 to the coefficient generation circuit 113. As shown in FIG. 3C, the correction amount Zt of the target pixel is calculated. The correction amount Zt calculated by the correction value calculation circuit 402 is a correction amount on a line segment connecting the pixel of interest to the pole in the polar coordinate system, and an XY vector coefficient is used for conversion from the polar coordinate system to the XY coordinate system. It is necessary to multiply the vector coefficient output from the calculation circuit 403.

  The XY vector coefficient calculation circuit 403 calculates a vector coefficient for converting the correction amount calculated in the polar coordinate system into the correction amount in the XY coordinate system. Specifically, the XY vector coefficient calculation circuit 403 has the following vector coefficients in accordance with the information on the deflection angle θt of the pixel of interest input from the pixel coordinate generation circuit 111 to the coefficient generation circuit 113.

Vx = cos θt
Vy = sin θt
Note that the sine function and cosine function calculation can also be realized in hardware by high-order function approximation or by piecewise low-order function approximation in which the argument θt is divided into predetermined values.

  Using the vector coefficients calculated by the XY vector coefficient calculation circuit 403 in this way, the coefficient generation circuit 113 can express the X component ZtH and the Y component ZtV of the correction amount in the XY coordinate system as follows.

ZtH = Zt · Vx
ZtV = Zt · Vy
The coefficient generation circuit 113 outputs the XY components ZtH and ZtV of the calculated correction amount to the frequency characteristic control circuit 114 and the optical correction circuit 115 described later. Note that the correction amount in the target pixel represents the phase shift amount of the optical image that should originally exist in the target pixel, and the XY component of the correction amount represents the horizontal and vertical phase shift components. Further, the correction amount of lateral chromatic aberration is different in the digital images Sr, Sg, and Sb because the angles at which RGB colors are refracted by the optical system 102 are different. Value.

  The optical correction circuit 115 performs the interpolation operation from the pixel values of the peripheral pixels centering on the position away from the target pixel by the correction amount calculated by the coefficient generation circuit 113 in the captured image, thereby affecting the influence of the aberration. The pixel value of the target pixel after correction is calculated. Specifically, it is obtained from the interpolation function with respect to a predetermined number of peripheral pixels centering on the positions separated by the XY components ZtH and ZtV of the correction amount of the target pixel calculated by the coefficient generation circuit 113 from the target pixel. By multiplying the interpolation coefficient, the pixel value at the position of the target pixel after correction is derived. Thereby, the optical correction circuit 115 can perform an interpolation operation using the interpolation function for all the pixels of the input digital image, and output an image in which the influence of the aberration is corrected.

Here, the interpolation function and the aberration correction method of the present embodiment using the interpolation function will be described in detail with reference to the drawings.
The interpolation function is a finite function obtained by superimposing a window function on a sinc function, as used in Bi-Cubic interpolation. When correcting the influence of aberration, pixels in a range defined by the window function ( It is possible to derive an interpolation coefficient to be multiplied for each of the reference pixels). The pixel value s ′ of the virtual pixel S ′, which is a pixel that should be imaged on the target pixel S, is the pixel value of _n reference pixels S _{0 to} S _n−1 centered on the virtual pixel S ′. It can be obtained by multiplying the interpolation coefficient and taking the sum. Pixel value s of the virtual pixel ', using the interpolation coefficients c ₀ to c _n-1 is multiplied to n pixel value s of the reference pixel ₀ ~s _n-1, and the reference pixel, respectively, by the following expression Can do.

  In this embodiment, like the virtual pixel S ′ and the pixel value s ′ of the virtual pixel, the information specifying the pixel is expressed using uppercase alphabets, and the pixel value of the pixel is expressed using the same lowercase alphabet. .

For example, when the positional relationship between the pixel of interest S and the virtual pixel S ′ is as shown in FIG. 5, the pixel value s ′ of the virtual pixel has four peripheral values centered on the virtual pixel S ′ when the number of reference pixels is four. Derived using the reference pixels S _{1 to} S ₄ . At this time, the interpolation coefficient derived by the interpolation function is a value that is inversely proportional to the distance of the reference pixel with respect to the virtual pixel S ′, for example, and is a coefficient that gives the maximum weight to the pixel S4 closest to S ′.

FIGS. 6A, 6B, and 6C are examples showing the time characteristics of the interpolation function of 8-point interpolation, and the window functions are narrowed in the order of (a), (b), and (c). The interval between the eight pixels being referred to is narrow. In the figure, the values of interpolation coefficients c _{0 to} c ₇ corresponding to pixels whose phases are shifted back and forth with respect to the interpolation phase indicated by time 0 are shown.

  6D, 6E, and 6F are examples showing the frequency characteristics of the respective interpolation functions, and the frequency response characteristics in the vicinity of the Nyquist frequency are different. When the window function is wide as shown in FIG. 6D, the frequency response characteristics in the vicinity of the Nyquist frequency are steep, so that the influence of aberration is corrected using the interpolation coefficient obtained from such an interpolation function. As a result, a pixel value with high reproducibility can be obtained. When the window function is narrow as shown in FIG. 6 (f), the frequency response characteristic near the Nyquist frequency is gentle. Therefore, when the influence of aberration is corrected using the interpolation coefficient obtained from such an interpolation function. The pixel value reproduction accuracy is low.

In this embodiment, the frequency characteristic control circuit 114 is provided in order to control the frequency characteristic of the interpolation function used for the interpolation calculation in the optical correction circuit 115 as described above. Here, the circuit configuration of the frequency characteristic control circuit 114 will be described with reference to FIG.
When the XY components ZtH and ZtV of the aberration correction amount input from the coefficient generation circuit 113 are input, the frequency characteristic control circuit 114 controls the frequency characteristics of the interpolation function for each of the horizontal direction and the vertical direction. The suppression coefficient is output to the optical correction circuit 115. Specifically, when an aberration correction amount is input to the frequency characteristic control circuit 114, the coefficient coefficient calculation circuits 701 to 703 are used for the horizontal and vertical correction amounts, and the suppression coefficients are set to ItHa, ItHb, ItHc. And ItVa, ItVb, ItVc are determined. Since the suppression coefficient is a value determined by the correction limit regardless of the direction, the suppression coefficient of the interpolation function in the horizontal direction will be described below in the present embodiment.

  The suppression coefficients ItHa, ItHb, and ItHc are functions of the aberration correction amount ZtH whose behavior is changed by the threshold values th_a, th_b, and th_c as shown in FIG. Each of the coefficient function calculation circuits 701 to 703 is supplied with, for example, threshold information and gradients grad_a, grad_b, and grad_c with respect to the correction amount ZtH exceeding the threshold from the microcomputer 101, and the suppression coefficient is determined according to the value of the correction amount.

For example, when an image of a partial area of a captured image is read out to a finite storage area of the optical correction circuit 115, the smaller correction amount of aberration is generally referred to the virtual pixel in the correction process. It is possible to take many pixels in the storage area. That is, the smaller the correction amount, the wider the window function of the interpolation function, and the larger the correction amount, the narrower the window function of the interpolation function. When correcting the eight reference pixels in the horizontal direction around the virtual pixel S ′ as shown in FIG. 6, the interpolation coefficient ch ′ obtained by controlling the frequency characteristic of the interpolation function controls the frequency characteristic. By using the interpolation coefficient ch and the suppression coefficient obtained from no interpolation function,
ch'j = chj (j = 3, 4)
ch'j = chj × ItHa (j = 2, 5)
ch′j = chj × ItHb (j = 1, 6)
ch′j = chj × ItHc (j = 0, 7)
It is expressed.

At this time, in order for the interpolation coefficient ch ′ obtained by controlling the frequency characteristic of the interpolation function to control the width of the window function as shown in FIG. 6 according to the correction amount ZtH,
0 ≦ th_c ≦ th_b ≦ th_a
And x intercepts itc_a, itc_b, itc_c calculated by the gradients grad_a, grad_b, grad_c
0 ≦ itc_c ≦ itc_b ≦ itc_a
The microcomputer 101 controls parameters supplied to the frequency characteristic control circuit 114 so that In this way, the frequency characteristic control circuit 114 outputs the interpolation coefficient suppression coefficient determined by the aberration correction amount to the optical correction circuit 115.

Next, a method for deriving a corrected pixel value performed by the optical correction circuit 115 using the suppression coefficient determined by the frequency characteristic control circuit 114 will be described in detail with reference to the drawings. The corrected pixel value derivation method described below is based on 8 × 8 pixels centered on the position of the virtual pixel S ′ that is separated from the target pixel S by the correction amount calculated by the coefficient generation circuit 113. In the following description, it is assumed that the pixel value of the virtual pixel S ′ is derived using the peripheral pixels as reference pixels. Specifically, as shown in FIG. 9, in order to derive the pixel value of the virtual pixel S ′ existing at the pixel position separated from the target pixel S by the correction amount ZtH in the horizontal coordinate and the correction amount ZtV in the vertical coordinate, 64 pixels from S _{00 to} S ₇₇ centered on the pixel S ′ are used as reference pixels. However, the implementation of the present invention is not limited to this, and any method may be used as long as it derives the pixel value of the virtual pixel S ′ using any number of peripheral pixels of 2 or more as reference pixels.

Here, the circuit configuration of the optical correction circuit 115 will be described with reference to FIGS. 10A, 10B, and 10C.
The aberration correction amounts ZtH and ZtV input from the coefficient generation circuit 113, and the suppression coefficients ItHa to ItHc and ItVa to ItVc input from the frequency characteristic control circuit 114 are the interpolation control circuit 1001 or the interpolation control for each vertical and horizontal component. Input to the circuit 1002. The interpolation control circuit 1001 calculates a vertical component interpolation coefficient, and outputs interpolation coefficients cv _{0 to} cv ₇ to the interpolation circuit 1012 using the interpolation function and the input suppression coefficient. Further, the interpolation control circuit 1001 transmits the input information on the vertical correction amount ZtV to the buffer memory 1003. The interpolation control circuit 1002 is a circuit that generates a horizontal component interpolation coefficient, and outputs the interpolation coefficients ch _{0 to} ch ₇ to each of the interpolation circuits 1004 to 1011 using the interpolation coefficient and the input suppression coefficient. Further, the interpolation control circuit 1002 transmits the input information of the horizontal correction amount ZtH to the buffer memory 1003.

The buffer memory 1003 is a storage area in which some of the images of the digital images Sr, Sg, and Sb converted by the AFE 109 are stored for correction processing, and the images sequentially converted by the AFE 109 are stored. Is done. When horizontal and vertical correction amounts ZtH and ZtV are input to the buffer memory 1003, the microcomputer 101 positions the virtual pixel S ′ that is separated from the target pixel S of the image stored in the buffer memory 1003 by the correction amount. Is calculated. Further, the microcomputer 101 determines a predetermined number of peripheral pixels centered on the pixel position of the calculated virtual pixel S ′, and sets the pixel values of the determined peripheral pixels so that they are processed by different interpolation circuits for each vertical coordinate. Output to the interpolation circuits 1004 to 1011. For example, the pixel values s _{00 to} s ₀₇ of the reference pixels in FIG. 9 are output to the interpolation circuit 1004.

The configuration of each of the interpolation circuits 1004 to 1011 is shown in FIG. 10B, and the configuration of the interpolation circuit 1012 is shown in FIG. As shown in FIG. 10B, the sum value circuit 1013 is multiplied by interpolation coefficients ch ′ _{0 to} ch ′ ₇ corresponding to the respective pixel positions in the horizontal direction to the pixel values of the reference pixels having the same vertical coordinates. Output. The summation circuit 1013 calculates the sum of the pixel values of the reference pixels multiplied by the interpolation coefficient and outputs a horizontal interpolation value. The horizontal interpolation values s ′ _{0 to} s ′ ₇ obtained from the pixel values that have been subjected to the horizontal interpolation operation having the same vertical coordinates and output from each of the interpolation circuits 1004 to 1011 are output to the interpolation circuit 1012. Perform vertical interpolation. As shown in FIG. 10C, the interpolation circuit 1012 applies interpolation coefficients cv ′ _{0 to} cv ′ ₇ corresponding to the respective pixel positions in the vertical direction to the horizontal interpolation values of the pixel values subjected to the horizontal interpolation calculation. Multiply and output to summation circuit 1014. Then, the summation circuit 1014 calculates and outputs an interpolation value s ′ that is the summation of the pixel values of all the reference pixels multiplied by the interpolation coefficient. The interpolation calculation for calculating the average value s ′ of the interpolation values via the above-described two-stage interpolation circuit can also be expressed by the following equation.

By doing this, it is possible to derive the pixel value of the optical image that should have been focused on the pixel of interest, and therefore it is preferable to apply the correction processing in the optical correction circuit 115 to all the pixels. It is possible to correct image quality degradation caused by various aberrations.
When the correction processing by the optical correction circuit 115 is applied to all pixels, the following problems occur depending on the capacity of the buffer memory 1003, the amount of correction, and the position of the pixel of interest.

  In the correction process, a predetermined number of peripheral pixels are referenced with a virtual pixel located at a position separated from the target pixel stored in the buffer memory 1003 by the correction amount calculated by the coefficient generation circuit 113 as a center. Perform interpolation calculation. At this time, as shown in FIGS. 11B, 11 </ b> C, and 11 </ b> D, there is a state in which at least one of a predetermined number of peripheral pixels centering on the virtual pixel is not read into the buffer memory 1003. Exists. In such a case, the interpolation calculation is not performed correctly by the amount of pixel values that cannot be referred to, which causes an unnecessary deterioration in image quality in the corrected image.

In the present embodiment, as described above, the storage means in which a predetermined number of peripheral pixels centering on the pixel position that should have been focused on the target pixel reads and stores an image of a partial area. A method for correcting the influence of the aberration when it is not stored in will be described below.
FIG. 11A shows, for example, a horizontal direction in which a suitable interpolation calculation can be performed using a predetermined number of peripheral pixels centered at a position separated by an aberration correction amount for a certain target pixel S, for example. The correction limit is shown. That is, for the correction amount exceeding the horizontal component ZtH _max of the aberration correction amount at this time, as shown in FIGS. 11B, 11C, and 11D, pixels not stored in the storage means are used in the interpolation calculation. Therefore, the interpolation function may be controlled as follows.

As shown in FIG. 11B, when the range of the reference pixel protrudes by one pixel in the horizontal direction from the image area stored in the storage unit, the interpolation coefficients ch ′ ₀ and ch ′ ₇ become 0. Thus, the suppression coefficient ItHc is set to zero. Specifically, the microcomputer 101 may supply the following th_c and grad_c to the frequency characteristic control circuit 114, for example.

th_c = ZtH _max −3
grad_c = -1
In this way, since ItHc becomes 0 at (correction limit amount −2), processing equivalent to reducing the number of reference pixels in the interpolation calculation can be realized. That is, when the predetermined value of the number of reference pixels in the horizontal direction is 8 pixels, the interpolation coefficient is 0 for 2 pixels located at both ends, and therefore, the interpolation operation is substantially performed on 6 pixels in the horizontal direction. . As a result, an interpolation process equivalent to the case where the window function is narrowed in the interpolation function can be performed. Therefore, the reproducibility of the obtained pixel value is lowered, but the correction amount ZtH is not limited. Image quality deterioration can be corrected.

  Similarly, in FIGS. 11C and 11D, the calculation parameter of the suppression coefficient is changed so that the interpolation coefficient to be multiplied by the pixel that protrudes from the area of the image stored in the storage unit becomes 0, respectively. Thus, it is possible to perform a preferable correction process for image quality deterioration due to aberration.

  In the present embodiment, the digital camera 100 has been described as including a three-plate imaging unit. However, the present invention is not limited to this, and for example, a color filter such as an RGB Bayer array is applied to one imaging element. In addition, it may be a digital camera provided with a single-plate imaging means. In this case, for example, the same effect can be obtained by separating the image signal for each color filter and applying the correction process.

As described above, the image processing apparatus according to the present embodiment can preferably correct image quality degradation due to aberrations in consideration of a correction limit in a captured image. Specifically, the image processing apparatus reads, for each region, an image captured using the optical system, and calculates a correction amount of aberration caused by the optical system for each pixel in the region. Then, by multiplying a predetermined number of peripheral pixels centered at a position away from the pixel by the calculated correction amount of aberration and multiplying by an interpolation coefficient obtained from the interpolation function, A pixel value at the position of the pixel is derived. At this time, if there is no predetermined number of peripheral pixels centered on the position separated by the calculated aberration correction amount, the image quality due to aberration can be improved by changing the interpolation function. Achieving correction of deterioration. Specifically, the interpolation function is changed and used for the correction process so that the frequency response characteristic of the interpolation function near the Nyquist frequency is more gradual than that of the interpolation function that gives an interpolation coefficient to be multiplied by a predetermined number of surrounding pixels. .
In this way, even when the correction limit reaches such a point that a predetermined number of pixels cannot be referred to in the correction process, it is possible to suitably correct the image quality degradation caused by the aberration.

(Embodiment 2)
In the present embodiment, in addition to the first embodiment described above, an aberration correction process in consideration of MTF information representing the resolution in a captured image, which the optical system has, will be described. Note that the functional configuration of the digital camera of the present embodiment is the same as that of the first embodiment, and the MTF information of the optical system 102 provided in the digital camera 100 is stored in the optical correction database 112 and read out in the course of correction processing.

  In the first embodiment, the aberration correction amount is calculated based on the correction amount derived by the coefficient generation circuit 113 and the approximate function of the image height. However, in the present embodiment, description will be made assuming that the limit value (first upper limit value) of the correction amount is determined in advance by the capacity of the buffer memory 1003 provided in the optical correction circuit 115. For example, in FIG. 12A, when the approximate function of the correction amount and the image height derived by the coefficient generation circuit 113 is a broken line 1201, the approximate function is limited by the limit value Lm of the correction amount determined in advance by the buffer memory 1003. Will be. That is, when the limit value of the correction amount is determined in advance, the correction amount with respect to the image height is limited to the thick broken line 1202.

Here, the MTF information Co indicating a sense of resolution will be described.
The MTF information Co is a parameter that can be calculated in advance when the optical system 102 is designed, and represents the fineness of the optical image formed by the optical system 102. As shown in FIG. 13A, the MTF information usually has a characteristic of decreasing as the image height increases. In FIG. 13A, the vertical axis represents MTF information Co, which is normalized by the value of the optical center (image height is 0). When the MTF information Co is high, the reproducibility of the subject is high, and the details of the subject are imaged on the image sensor, so that “high resolution” is expressed.

  When the resolution is high, the subject image has been reproduced in detail, so that the displacement and distortion of the subject image moved due to lateral chromatic aberration and distortion become conspicuous. That is, when the resolution is high, the influence of the aberration is conspicuous, so it is necessary to correct the deviation and distortion of the subject image while keeping the aberration correction amount as high as possible. In addition, when the resolution is low, the influence of aberration is inconspicuous, so it is necessary to reduce the correction accuracy of the aberration, and conversely to maintain the high pixel reproduction accuracy after correction. Therefore, the relationship between the MTF information Co and the correction limit amount is as shown in FIG.

  That is, from FIG. 13A and FIG. 13B, the correction limit amount considering the MTF information is represented by a one-dot chain line 1203 (third upper limit value) in FIG. 12A. The high approximation function is as indicated by a thick line 1204. As described above, the approximation function 1201 of the correction amount and the image height derived by the coefficient generation circuit 113 is limited to the approximation function 1204 by the capacity of the buffer memory 1003 of the optical correction circuit 115 and the MTF information Co. In the present embodiment, such correction amount limitation is realized by providing a limiter 721 and a limiter 722 for each of the horizontal and vertical components in the frequency characteristic control circuit 114 as shown in FIG.

  As described above, the image processing apparatus according to the present embodiment can perform appropriate correction of image quality degradation caused by aberrations in consideration of MTF information representing a sense of resolution.

(Embodiment 3)
In this embodiment, in addition to the configuration of the first embodiment, when the approximate function of the correction amount and the image height is limited by the limit value of the correction amount determined from the capacity of the buffer memory 1003 as in the second embodiment, the image quality after correction is Considering the correction characteristic information Cc qualitatively showing what impression gives. When the approximation function of the correction amount and the image height is limited, the correction characteristic information Cc includes information Lw of the image height range exceeding the limit value of the correction amount and the minimum image height Lh in the image height range. Is information determined from the impression degrees Ccw and Cch received from the image quality after correction.

  The functional configuration of the digital camera 100 of the present embodiment is a configuration in which a correction characteristic information generation circuit 120 for calculating the correction characteristic information Cc is added to the configuration of the first embodiment. The correction characteristic information generation circuit 120 obtains information on the correction value limit value Lm determined in advance from the plot data Cm1 to Cm4 of the optical correction characteristic information of the optical parameters at the time of imaging and the capacity of the buffer memory 1003 of the optical correction circuit 115. It is input from. Then, the correction characteristic information generation circuit 120 calculates correction characteristic information Cc using the input information.

The circuit configuration of the correction characteristic information generation circuit 120 will be described in detail with reference to FIG. The plot data of the optical correction characteristic information of the optical parameters at the time of imaging input to the correction characteristic information generation circuit 120 is input to a function coefficient calculation circuit 1401 similar to the function coefficient calculation circuit 401 included in the coefficient generation circuit 113, and Outputs coefficients of quadratic approximation function between plots. The coefficients of the approximate functions between the plots are output to intercept calculation circuits 1402, 1403, and 1404, respectively. In each intercept calculation circuit, Lm = aRt ² + bRt + c
For the image height Rt satisfying the above, and in the characteristic coefficient calculation circuit 1405 for the image height Rt of each obtained solution, the information Lw of the image height range exceeding the limit value of the correction amount and the minimum of the image height in the range of the image height The value Lh is derived.

Further, the characteristic coefficient calculation circuit 1405 further receives from the corrected image quality from the information Lw of the image height range exceeding the limit value of the obtained correction amount and the information of the minimum image height Lh of the image height range. Correction characteristic information Cc indicating the impression level is calculated. The minimum image height Lh in the range of the image height exceeding the limit value of the correction amount and the impression degree Cch received from the image quality after correction are represented by the relationship as shown in FIG. 13C, and the image exceeds the limit value of the correction amount. The impression level Ccw received from the high range Lw and the image quality after correction is represented by the relationship as shown in FIG.
Here, the impression degree Cch and the impression degree Ccw are normalized by the normalization levels Cchn and Ccwn set from the outside. The normalization levels Cchn and Ccwn are image quality adjustment items set on the basis that the impression level received from the image quality after correction does not lead to a decrease in image quality, and this value is obtained experimentally.

  Here, the concept of the impression level received from the corrected image quality in this embodiment will be described. In the present embodiment, it is assumed that information existing at the optical center (image height is 0) is regarded as an important subject image by setting a photographing mode such as a portrait mode. That is, the observer determines the image quality of the subject image located at the center of the framing as the image quality. In other words, when the limit value of the correction amount is exceeded, it is considered that the correction accuracy of the deviation or distortion of the subject image caused by the aberration is low, and the image quality is deteriorated. For this reason, as the minimum image height Lh in the image height range is closer to the optical center, the deviation or distortion caused by the aberration of the subject image located at the center of the framing will remain uncorrected and remain for the observer. It is easy to feel that the image quality has deteriorated. In addition, when the image height range Lw that exceeds the limit value of the correction amount is large, the range of the image height that remains without being corrected for the deviation or distortion caused by the aberration in the corrected image becomes large. However, it is easy to notice the deterioration of the image quality that occurs without maintaining the correction accuracy.

  As described above, in the present embodiment, the characteristic coefficient calculation circuit 1405 selects, as the correction characteristic information Cc, the impression degree Cch and Ccw received from the qualitatively determined corrected image quality, for example, whichever is larger. This is output to the frequency characteristic control circuit 114. In other words, the higher the correction characteristic information Cc obtained by the correction characteristic information generation circuit 120, the easier the observer feels image quality degradation. Therefore, it is necessary to maintain the aberration correction amount as high as possible and correct the deviation and distortion of the subject image. There is. In addition, the lower the correction characteristic information Cc, the less the observer feels image quality deterioration. Therefore, it is necessary to lower the correction accuracy of the aberration, and conversely to maintain the high pixel reproduction accuracy after correction. Therefore, the relationship between the correction characteristic information Cc and the correction limit amount is as shown in FIG.

  When the limit value of the correction amount is determined by the capacity of the buffer memory 1003 provided in the optical correction circuit 115 and an approximate function of the correction amount and the image height is obtained from the optical parameters at the time of imaging, the image height exceeding the limit value of the correction amount is obtained. A minimum image height Lh between the range Lw and the range of the image height is calculated. Then, the correction characteristic information Cc is obtained from the obtained image height range Lw and the minimum image height Lh, and the correction limit amount Lm ′ (first) corresponding to the correction characteristic information Cc obtained from the graph of FIG. 2) is obtained. That is, the approximate function of the correction amount and the image height of the present embodiment is limited by the broken line 1205 representing the correction limit amount Lm ′ that is less than the upper limit value of the correction amount (less than the first upper limit value) in FIG. The thick line 1206 is obtained. As described above, the approximate function 1201 of the correction amount and the image height derived by the coefficient generation circuit 113 is limited to the approximate function 1206 by the capacity of the buffer memory 1003 of the optical correction circuit 115 and the correction characteristic information Cc. In the present embodiment, such correction amount limitation is realized by providing a limiter 721 and a limiter 722 for each of the horizontal and vertical components in the frequency characteristic control circuit 114 as shown in FIG.

  As described above, the image processing apparatus according to the present embodiment keeps the pixel reproduction accuracy after correction higher than the correction accuracy so that the observer can be considered less susceptible to deterioration in the image quality after correction. The limit value after correction is set to give priority to. That is, it is possible to suitably correct image quality degradation caused by aberration in consideration of correction characteristic information that is an impression degree received from the image quality after correction, which is determined by the limit value of the correction amount.

(Embodiment 4)
In the present embodiment, in addition to the configuration of the first embodiment, an aberration correction process in consideration of subject feature information representing the feature degree of a subject image obtained by detecting edge components and motion vectors in a captured image. explain. In the present embodiment, a case where the approximation function of the correction amount and the image height is limited by the limit value of the correction amount determined from the capacity of the buffer memory 1003 as in the second embodiment will be described. Note that the subject feature information Cf is a value representing whether or not there is a feature in the captured image, that is, whether or not pixel value reproduction accuracy is required at the time of correction.

  The functional configuration of the digital camera 100 of the present embodiment is a configuration in which a feature detection circuit 130 for calculating a feature evaluation value Id necessary for deriving the subject feature information Cf is added to the configuration of the first embodiment described above. Become. The feature detection circuit 130 may have a different circuit configuration according to the type of information detected as the feature of the subject image, such as gradient detection, high-frequency component detection, motion detection, and the like. Will be described below.

  When performing gradient detection (edge detection) between pixels, which are edge components, as a feature of the subject image, the configuration is generally as shown in FIG. When the RGB color digital images Sr, Sg, and Sb are input from the AFE 109, the luminance signal generation circuit 1501 calculates the luminance value y of the target pixel from the pixel value at the target pixel position of each digital image by the following equation. Is calculated.

y = 0.299Sr + 0.587Sg + 0.114Sb
Thus, the luminance value y calculated for all the pixels is output to the gradient detection circuit 1502 as a luminance signal (luminance image). The gradient detection circuit 1502 is, for example, a Sobel filter. When the luminance signal Y is input from the luminance signal generation circuit 1501, the gradient detection circuit 1502 outputs a gradient amount Yg (primary differential value of the luminance signal) at the target pixel position. The gradient amount Yg obtained by the gradient detection circuit 1502 is input to the integration circuit 1503, and the integral value of the gradient amount Yg is output as the feature evaluation value Id for each rectangular area into which the luminance signal of the captured image is segmented.

  As a feature of the subject image, when detecting a high-frequency component of a luminance signal, the configuration is generally as shown in FIG. When RGB digital images Sr, Sg, and Sb are input from the AFE 109, the luminance signal generation circuit 1501 converts the digital image into a luminance signal Y and outputs the luminance signal Y to the high frequency component detection circuit 1504. The high frequency component detection circuit 1504 is, for example, a bandpass filter, and outputs a high frequency component Yh (secondary differential value of a luminance signal) whose focus is on the ground. The high-frequency component Yh obtained by the high-frequency component detection circuit 1504 is input to the integration circuit 1503, and the integrated value of the high-frequency component Yh is output as the feature evaluation value Id for each rectangular area into which the luminance signal of the captured image is divided. .

  The feature evaluation value Id obtained by gradient detection or high-frequency component detection in this way is a region having a feature in the captured image as the feature evaluation value Id is higher, and pixel value reproduction accuracy is required at the time of correction. Therefore, the subject feature information Cf is as shown in FIG.

  Further, when detecting motion vectors existing in continuous images as the characteristics of the subject image, the configuration is as shown in FIG. Note that the configuration of FIG. 15C is a general configuration of a motion vector analysis circuit to which a matching operation using representative points is applied. When RGB digital images Sr, Sg, and Sb are input from the AFE 109, the luminance signal generation circuit 1501 converts the digital image into a luminance signal Y and outputs the luminance signal Y to the frame memory 1505. The frame memory 1505 has a capacity for holding at least two luminance representative point frames, and holds a representative point data image obtained by resampling the luminance signal Y in the horizontal and vertical directions at predetermined pixel intervals. The representative point data image created by the frame memory 1505 is also held in the buffer memory 1506.

  In the representative point matching circuit 1507, by comparing the representative point data images stored in the frame memory 1505 and the buffer memory 1506 in this way, a feature evaluation value is obtained for each rectangular area into which the luminance signal of the captured image is divided. Id is calculated and output. Specifically, the representative point data image stored in the buffer memory 1506 is divided into rectangular areas and sequentially output to the representative point matching circuit 1507. The representative point matching circuit 1507 sequentially reads out the representative point pixel group Ys having the same size as the rectangular area from the frame memory 1505 from the upper left, and the representative point pixel having the smallest difference from the representative point pixel group Yt input from the buffer memory 1506. Find the position of the group Ys. Then, the motion vector of the rectangular area is calculated from the readout position of the representative point pixel group Ys having the smallest difference and the position of the rectangular area input from the buffer memory 1506. Further, the representative point matching circuit 1507 outputs the reciprocal of the scalar quantity of the motion vector of each region as the feature evaluation value Id.

  The feature evaluation value Id obtained by motion vector detection in this manner is a region where the subject moves in the captured image as the feature evaluation value Id is lower. Therefore, the deviation or distortion of the subject image caused by aberration is less noticeable. . That is, in the case of motion vector detection, aberration correction accuracy is not required as the motion vector is large, and the correction amount can be reduced during correction to increase the pixel value reproduction accuracy. As shown in (g).

  Thus, in the present embodiment, the feature evaluation value Id is calculated by the feature detection circuit 130, and the subject feature information Cf that is the reproduction accuracy of the pixel value required at the time of correction is obtained. When the subject feature information Cf is small, there is a high demand for pixel value reproduction accuracy. Therefore, it is necessary to improve the pixel value reproduction accuracy by increasing the number of reference pixels in the correction process even if the correction amount of aberration is limited. is there. That is, the relationship between the subject feature information Cf and the correction limit amount is as shown in FIG.

  That is, when the feature evaluation value Id is calculated in the feature detection circuit 130, the microcomputer 101 obtains subject feature information Cf determined by the type of feature detection. Further, the microcomputer 101 obtains information on the correction limit amount Lm ′ (fourth upper limit value and fifth upper limit value) corresponding to the obtained subject feature information Cf from the graph of FIG. That is, the approximate function of the correction amount and the image height of the present embodiment is like a thick line 1206 that is limited to the correction limit amount Lm ′ in FIG. Thus, the approximate function 1201 of the correction amount and the image height derived in the coefficient generation circuit 113 is limited to the approximate function 1206 by the capacity of the buffer memory 1003 of the optical correction circuit 115 and the subject feature information Cf. In the present embodiment, such correction amount limitation is realized by providing a limiter 721 and a limiter 722 for each of the horizontal and vertical components in the frequency characteristic control circuit 114 as shown in FIG.

  As described above, the image processing apparatus according to the present embodiment gives priority to keeping the pixel reproducibility after correction higher than the correction accuracy so that the demand for pixel value reproducibility during correction is high. In this way, the limit value of the correction amount is set. That is, image quality degradation caused by aberration can be suitably corrected in consideration of subject feature information that is a requirement of pixel value reproduction accuracy determined by an edge component, a high-frequency component, a motion vector, and the like in a captured image.

(Embodiment 5)
In the present embodiment, the approximation function of the correction amount and the image height and the correction of the aberration are considered in consideration of all of the MTF information Co, the correction characteristic information Cc, and the subject characteristic information Cf described in the second to fourth embodiments. A method will be described. That is, the functional configuration of the digital camera 100 of the present embodiment is a configuration in which the correction characteristic information generation circuit 120 and the feature detection circuit 130 are added to the configuration of the first embodiment. Further, the optical correction database 112 manages MTF information.

Summarizing the information of the above-described embodiment: The MTF information Co, which is the resolution, becomes higher as the image height is lower. The correction accuracy is required as the MTF information Co is higher. The reproduction accuracy is higher than the correction accuracy as the MTF information Co is lower. Is required

The correction characteristic information Cc, which is the impression level of the image quality after correction,
-The higher the range of the image height that exceeds the limit value of the correction amount, the higher.-The higher the minimum image height that exceeds the limit value of the correction amount, the higher.-The higher the correction characteristic information Cc, the higher the correction accuracy is required. The lower the information Cc, the more accurate the reproduction accuracy is required.

The subject feature information Cf, which is the feature level of the subject image, is
-The higher the edge component, the higher the region-The higher the high-frequency component, the higher-The lower the motion vector, the higher the region-The higher the subject feature information Cf, the higher the correction accuracy required-The lower the subject feature information Cf, the correction accuracy Since more accurate reproducibility is required, as is clear from FIGS. 13B, 13E and 13H, as each information approaches 1, the correction accuracy of aberration is kept as high as possible. It becomes a tendency. That is, when the MTF information Co, the correction characteristic information Cc, and the subject characteristic information Cf are integrated, for example, when the lowest correction limit amount derived by the image height is selected, the approximation of the limited correction amount and the image height is performed. The function may be as indicated by a thick line 1207 in FIG.

  As described above, the image processing apparatus according to the present embodiment can appropriately correct image quality degradation caused by aberrations in consideration of the characteristics of the captured image and the limit amount of the correction amount.

  In the above-described embodiment, a digital camera including an optical system has been described as an example of an image processing apparatus. However, the embodiment of the present invention is not limited thereto. For example, the image processing apparatus may be a device that accepts input of an image such as a printer. In this case, information on aberrations of the optical system is input together with the image as one file or included in the image data information. It only has to be done. The image processing apparatus may read out information on aberrations of the optical system in the course of the correction process and perform the correction process as described above.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

Storage means for reading out a partial area of the first image and storing it as a second image;
Computing means for computing the signal value of the pixel of interest in the second image using a signal value of a peripheral pixel centered on a position different from the pixel of interest and an interpolation coefficient that multiplies the signal value of the peripheral pixel; Have
The calculation means includes a case where at least one of a predetermined number of peripheral pixels centered at a position different from the pixel of interest is not stored as the second image, and the predetermined centered at a position different from the pixel of interest. An image processing apparatus, wherein the interpolation coefficient is set to a different value when all of the peripheral pixels are stored as the second image . The calculation means centers on a position different from the target pixel when at least one of the predetermined number of peripheral pixels centered on a position different from the target pixel is not stored as the second image. The image processing apparatus according to claim 1, wherein the signal value of the target pixel is calculated using signal values of peripheral pixels less than the predetermined number . The interpolation coefficient used by the computing means is obtained from an interpolation function,
The calculation means uses a signal value of the predetermined number of peripheral pixels centered at a position different from the pixel of interest and a signal value of peripheral pixels less than the predetermined number centered at a position different from the pixel of interest. The image processing apparatus according to claim 2 , wherein the interpolation coefficient obtained from an interpolation function having different frequency characteristics is used depending on whether or not is used. The interpolation function used when calculating the signal value of the pixel of interest using the signal values of the predetermined number of surrounding pixels centered on a position different from the pixel of interest is the center of the position different from the pixel of interest. 4. The frequency characteristic in the vicinity of the Nyquist frequency is steeper than an interpolation function used when the signal value of the target pixel is calculated using signal values of peripheral pixels smaller than a predetermined number. Image processing apparatus.   2. The calculation means according to claim 1, further comprising: a first interpolation means for performing calculation using a horizontal component interpolation coefficient; and a second interpolation means for performing calculation using a vertical component interpolation coefficient. 5. The image processing device according to any one of 4 above.   The arithmetic means includes a plurality of first interpolation means, inputs signal values of the peripheral pixels to the plurality of first interpolation means, and signal values output from the plurality of first interpolation means. The image processing apparatus according to claim 5, wherein the signal value of the pixel of interest is calculated by inputting to the second interpolation unit.   The image processing apparatus according to claim 1, wherein the calculation unit determines a position different from the target pixel based on an image height of the target pixel. A step of reading out a partial region of the first image and storing it as a second image;
The calculation means calculates the signal value of the pixel of interest in the second image using a signal value of a peripheral pixel centered on a position different from the pixel of interest and an interpolation coefficient that multiplies the signal value of the peripheral pixel. And a step of
The interpolation coefficient is determined when at least one of a predetermined number of peripheral pixels centered at a position different from the pixel of interest is not stored as the second image, and when the predetermined coefficient is centered at a position different from the pixel of interest. An image processing method, wherein different values are set for a case where all of the peripheral pixels are stored as the second image .