JP6255892B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  In a photographed image by a digital camera or the like, resolution degradation often occurs. Such deterioration tends to occur in the peripheral portion with respect to the central portion of the optical axis due to the aberration of the lens optical system and the angle-of-view dependency of the aperture size, and is remarkable in an inexpensive lens optical system. Therefore, attempts have been made to recover image degradation of the peripheral portion caused by such an optical system in the captured image by image processing.

  For example, there is an example aiming at recovering the degradation of image data by performing processing such as photographing a calibration chart having a first mark and a second mark from at least two directions and specifying the position of the mark. is there. There is also known an image processing apparatus that extracts an image area corresponding to a recovery item of image data from the image data, adjusts a recovery filter according to an adjustment operation of the recovery item, and performs recovery processing of the image area by the recovery filter. There is also a method for performing correction based on a difference between a first position of an index detected from a pattern image including a plurality of indices and a second position of the index calculated using a parameter relating to the imaging apparatus. It is known (for example, see Patent Documents 1 to 3).

  In addition, as described above, the method of performing correction based on the position of the index detected in the image uses a finite number of indices, and thus requires interpolation processing. For this reason, a method of performing interpolation processing is also known (for example, see Patent Document 4).

JP 2003-307466 A JP 2011-193277 A JP 2007-34985 A JP 2004-48112 A

  However, as described above, even if interpolation processing is used in a method for performing correction based on the position of an index detected in an image, for example, due to reasons such as anisotropy in resolution degradation, correction is appropriately performed. It may not be possible.

  According to one aspect, an object of the present invention is to enable generation of a filter that accurately corrects resolution degradation in an image.

An image processing apparatus according to one aspect includes a resolution calculation unit, an interpolation unit, and a filter calculation unit. The resolution calculation unit calculates resolution information indicating the resolution status corresponding to at least two points on the image based on the image. The interpolation unit calculates resolution information corresponding to one point on the image other than at least two points based on the resolution information corresponding to each of at least two points and at least two positions on the image. The filter calculation unit calculates a filter for correcting a region including the one point of the image according to the resolution information corresponding to the one point. In this image processing apparatus, the resolution information has a resolution value corresponding to the resolution corresponding to each point of the image, an anisotropy indicating the anisotropy of the resolution, and an angle corresponding to the anisotropy of the resolution. The resolution value, the degree of anisotropy, and the angle have a pattern in which lines having different intervals and widths are sequentially arranged at equal distances in each of at least two directions with respect to each of at least two points. This corresponds to the size of the ellipse, the ellipticity, and the angle of the ellipse formed by the distribution of points having the same amplitude of the luminance value corresponding to the pattern in the image of the captured subject.

  According to the image processing apparatus, the image processing method, and the program of the embodiment, it is possible to generate a filter that accurately corrects resolution degradation in an image.

It is a block diagram which shows an example of schematic structure of the imaging device containing the image processing device by 1st Embodiment. It is a block diagram which shows an example of a structure of the coefficient analysis part by 1st Embodiment. It is a block diagram which shows an example of a structure of the interpolation part by 1st Embodiment. It is a figure which shows an example of the optical system by 1st Embodiment. It is a figure which shows the example of the opening according to the image position by 1st Embodiment. It is a figure which shows an example of the direction of the blur according to the image position by 1st Embodiment. It is a figure for demonstrating the resolution in the case of using the chart by 1st Embodiment. It is a figure which shows the result of having measured the resolution with the chart by 1st Embodiment. It is a figure which shows an example of the difference in the resolution degradation by the picked-up image position by 1st Embodiment. It is a figure which shows the anisotropy of the resolution in the chart by 1st Embodiment. It is a figure which shows the resolution analysis result of the chart by 1st Embodiment. It is a figure which shows the resolution analysis result of the chart of the captured image center part by 1st Embodiment. It is a figure explaining the calculation method of PSF by a 1st embodiment. It is a figure which shows the example of a characteristic when Fourier-transforming PSF by 1st Embodiment. It is a figure which shows an example of the elliptical PSF by 1st Embodiment. It is the figure which carried out the Fourier-transform of the elliptical PSF by 1st Embodiment. It is a figure which shows the reciprocal number of K ((omega), (theta)) by 1st Embodiment. It is a figure which shows an example of the filter at the time of adding the constant value by 1st Embodiment to the denominator. It is a figure which shows an example of the filter in the case of reducing a gain, so that it becomes high frequency by 1st Embodiment. It is a figure which shows the example which calculated the filter based on the chart by 1st Embodiment. It is a figure which shows the example which calculated nine filters from nine charts by 1st Embodiment. It is a figure which shows the example of arrangement | positioning of the calculation position of the filter by 1st Embodiment. It is a figure which shows the comparative example of the interpolation process of a filter. It is a figure which shows the comparative example which calculates a filter by interpolation. It is a figure which shows the comparative example which calculates a filter by interpolation. It is the figure which put together the comparative example of the filter calculation by the interpolation between elements. It is a figure which shows the comparative example which performs the interpolation of a filter by rotation. It is a figure explaining the interpolation in 1st Embodiment. It is a figure explaining the interpolation of the filter by a 1st embodiment. It is a figure which shows an example of the filter table by 1st Embodiment. It is a figure which shows an example of the resolution table by 1st Embodiment. It is a figure which shows an example of the comparison of a filter calculation example. It is a flowchart which shows the whole process of the imaging device by 1st Embodiment. It is a flowchart which shows the interpolation process by 1st Embodiment. It is a block diagram which shows an example of a structure of the interpolation part by a modification. It is a figure explaining the angle calculation method by a modification. It is a figure explaining the angle calculation method by a modification. It is a block diagram which shows an example of schematic structure of the imaging device by 2nd Embodiment. It is a block diagram which shows an example of a structure of the coefficient analysis part by 3rd Embodiment. It is a figure explaining an example of the determination process of the weighting coefficient by 3rd Embodiment. It is a figure explaining an example of the determination process of the weighting coefficient by 3rd Embodiment. It is a figure which shows the analysis result of the resolution after the image correction by 3rd Embodiment. It is a figure which shows the analysis result of the resolution after the image correction by 3rd Embodiment. It is a flowchart which shows the filter calculation process by 3rd Embodiment. It is a flowchart which shows the filter calculation process by 3rd Embodiment. It is a figure which shows the case where the blurring by 4th Embodiment is asymmetrical with respect to the center of an image. It is a figure which shows an example of the angle calculation method when the blurring by 4th Embodiment is asymmetrical. It is a figure which shows the effect of the calculation method of the resolution information by 4th Embodiment. It is a figure which shows an example of the magnitude | size relationship between PSF showing the blur of the image by 5th Embodiment, and the filter whose number of elements is 5x5. It is a figure which shows the structural example of the filter of the multistage by 5th Embodiment. It is a figure which shows the structural example of the filter of the multistage by 5th Embodiment. It is a block diagram which shows an example of a structure of the filter control part by 5th Embodiment, and a filter process part. It is a block diagram which shows an example of a structure of the filter control part in 5th Embodiment, and a filter process part. It is a figure which shows an example of the division | segmentation of the filter by 5th Embodiment. It is a figure which shows an example of the division | segmentation of the filter by 5th Embodiment. It is a figure which shows an example of the arrangement | positioning of the chart in the to-be-corrected subject. It is a figure which shows an example of the arrangement | positioning of the chart in the to-be-corrected subject. It is a figure which shows an example of the arrangement | positioning of the chart in the to-be-corrected subject. It is a figure which shows an example of the arrangement | positioning of the chart in the to-be-corrected subject. It is a figure which shows the hardware constitutions of a standard computer.

(First embodiment)
The image processing apparatus according to the first embodiment will be described below with reference to the drawings. FIG. 1 is a block diagram illustrating an example of a schematic configuration of an imaging apparatus 100 including an image processing apparatus according to the first embodiment. An imaging apparatus 100 shown in FIG. 1 is an apparatus for photographing a correction subject 20 as a subject and calculating a filter for correcting the image. As shown in FIG. 1, the imaging device 100 includes an optical system 1, an imaging device 2, an analog front end (AFE) 3, an image processing unit 4, a post-processing unit 5, a drive control device 6, a control device 7, and an image memory 8. , A display unit 9, a coefficient analysis unit 10, and an interpolation unit 13.

  In the example of FIG. 1, a chart 25 is arranged at the center and four corners of the correction subject 20. The chart 25 is a pattern in which wedges are arranged radially, and is called a Siemens star. For example, the optical system 1 includes lenses 11a, 11b, 11c, and a diaphragm 12. The lenses 11 a, 11 b, 11 c and the diaphragm 12 focus the light from a subject such as the correction subject 20 on the imaging surface of the imaging device 2 to form an image of the subject. The drive control device 6 is a device that controls the positions of the lenses 11a, 11b, and 11c, the degree of aperture of the aperture 12, and the like. The optical system 1 is an example, and the configuration is not limited to a specific one.

  The image sensor 2 converts light from the subject collected by the optical system 1 into an electronic signal (analog signal). The imaging device 2 includes, for example, a two-dimensional imaging device such as Charge-Coupled Device (CCD) / Complementary Metal-Oxide Semiconductor (CMOS). The image sensor 2 converts the subject image into an electronic signal (image signal) and outputs the electronic signal to the AFE 3.

  The AFE 3 converts an analog signal of the captured image into a digital signal. The AFE 3 includes, for example, an analog / digital (A / D) converter 31 and a timing generator 32. The timing generator 32 generates a timing pulse used for driving the image sensor 2 based on a control signal from the control device 7 and outputs the timing pulse to the image sensor 2 and the A / D converter 31. The A / D converter 31 converts an analog signal into a digital signal according to the timing pulse.

  The image processing unit 4 stores an image of a digital signal from the AFE 3 and performs image processing. The image processing unit 4 includes a RAW memory 41. The RAW memory 41 records, for example, an image (RAW image) converted into a digital signal by the A / D converter 31. The image processing unit 4 may perform predetermined processing such as compression processing on the RAW image. The image subjected to the predetermined processing is recorded in the image memory 8.

  The post-processing unit 5 further performs necessary processing such as changing the color tone on the image that has undergone predetermined processing to generate a display image. For example, the post-processing unit 5 reads an image that has been subjected to predetermined processing from the image memory 8, performs processing for generating an image to be displayed, and outputs the image to the display unit 9.

  The image memory 8 stores an image after predetermined processing. The display unit 9 can include, for example, a Video Random Access Memory (VRAM) that records an image and a display that outputs an image of the VRAM. Note that the imaging apparatus 100 does not necessarily have a display function, and instead of the display unit 9, a recording unit (for example, a VRAM) that records an image for display may be provided. The drive control device 6 controls the optical system 1. The control device 7 controls the AFE 3 and the post-processing unit 5.

  The coefficient analysis unit 10 analyzes the resolution at each image position from the image in which the chart is photographed, and determines appropriate filter data for improving the resolution anisotropy. Details of the coefficient analysis unit 10 will be described later. Based on the result analyzed by the coefficient analysis unit 10, the interpolation unit 13 calculates a filter by interpolation corresponding to a point (for example, a pixel position) on the image that has not been calculated from the image of the correction subject 20. A point at which a corresponding filter is calculated by interpolation is also referred to as an interpolation position.

  The filter data can be a set of parameters necessary for filtering for image correction, such as a deconvolution kernel. Specifically, the deconvolution kernel can be represented by a region in which an image of a circular or elliptical object corresponding to Point Spread Function (PSF) is distributed and data representing the weight of each pixel in the region. Such data is called a deconvolution distribution.

  FIG. 2 is a block diagram illustrating an example of the configuration of the coefficient analysis unit 10. As illustrated in FIG. 2, the coefficient analysis unit 10 includes a resolution calculation unit 101, a filter calculation unit 102, and a filter memory 103. The resolution calculation unit 101 analyzes resolution deterioration of the chart 25 in the correction subject 20 at least in the direction of the image sensor 2 according to the position of the image.

  For example, the resolution calculation unit 101 may perform resolution analysis based on the relationship between the number of lines per pixel on the horizontal axis and the amplitude intensity on the vertical axis. The resolution calculation unit 101 calculates an angle, an anisotropy degree, and a resolution value of resolution information described later. That is, the resolution calculation unit 101 obtains an ellipse by creating contour lines with a certain threshold value (about half of the maximum amplitude) for the luminance amplitude value in the image corresponding to each chart 25 of the correction subject 20. The obtained ellipse corresponds to the resolution information. Hereinafter, the ellipse corresponding to the resolution information is also referred to as resolution information ellipse. In the resolution information, the angle is an angle indicating a direction in the obtained ellipse image. The degree of anisotropy is the ellipticity of the obtained ellipse. The resolution value is the size of the ellipse (short axis or long axis length).

  The obtained resolution information is referred to as resolution information corresponding to the center point of the ellipse of the resolution information, for example. For example, the position of a pixel corresponding to the center point of the ellipse in the resolution information is also referred to as a resolution information calculation position.

  The resolution calculation unit 101 can calculate the ellipticity as the ratio between the major axis and the minor axis of the obtained ellipse. Based on the position of the chart 25 in the image, the resolution calculation unit 101 geometrically calculates the angle θ1 of the resolution information, for example. For example, the angle θ1 is calculated as the direction of the resolution information calculation position from the point on the image corresponding to the optical axis center of the image. The resolution calculation unit 101 can also calculate the angle θ1 from the major axis and minor axis of the ellipse of the resolution information. If the angle θ1 is calculated from the long axis and the short axis, the angle can be calculated in accordance with the actual degree of blur.

  For example, the resolution calculation unit 101 may calculate an angle between the vertical direction of the image and the radial direction of the ellipse as the angle of the resolution information. Here, the angle reference is not limited to the vertical direction of the image. The center of the optical axis is basically the center of the image, but the center may deviate due to the displacement of the lens. The angle, the degree of anisotropy, and the size of the ellipse in the resolution information may be relative values. The resolution calculation unit 101 stores the calculated resolution information in association with the calculated position, for example, in the filter memory 103 as a resolution information table described later.

  The filter calculation unit 102 determines the PSF from the calculated resolution information. The PSF ellipse at this time has a shape rotated 90 degrees from the ellipse of the resolution information. The filter calculation unit 102 calculates a filter according to the PSF of the image. The filter calculated by the filter calculation unit 102 is an inverse filter for correcting an image, but is hereinafter simply referred to as a filter. The filter calculation unit 102 associates the calculated filter with the calculated position on the image, and stores the filter in the filter memory 103 as a filter table to be described later.

  FIG. 3 is a block diagram illustrating an example of the configuration of the interpolation unit 13. As illustrated in FIG. 3, the interpolation unit 13 includes an interpolation position specifying unit 35, an interpolation source specifying unit 36, and a resolution information interpolation unit 37. The interpolation position specifying unit 35 specifies a position on the image other than the position where the resolution information is calculated by the resolution calculating unit 101 as a calculation position (also referred to as an interpolation position) of a filter calculation target by interpolation. The interpolation source specifying unit 36 specifies a plurality of interpolation source positions used for performing filter calculation at the interpolation position determined by the interpolation position specifying unit 35. At this time, the interpolation source position is set to one of the calculated positions at which resolution information is obtained by the resolution calculation unit 101. The interpolation position specifying unit 35 and the interpolation source specifying unit 36 preferably refer to the resolution information stored in the filter memory 103 in order to specify each position.

  The resolution information interpolation unit 37 calculates the angle, resolution value, and anisotropy of the resolution information by interpolation. As an interpolation method, for example, a nearest neighbor method, a bilinear method, a bicubic method, or the like as described in Patent Document 4 can be used. At this time, the angle can be an angle with respect to the vertical direction or the horizontal direction of the image. The interpolation unit 13 outputs the interpolated resolution information to the coefficient analysis unit 10. The coefficient analysis unit 10 calculates a filter by the filter calculation unit 102 from the interpolated resolution information by the method described above, and stores it in the filter memory 103.

  As described above, the imaging device 100 calculates each filter at a plurality of points on the image. The calculated filter may be stored in a filter table of the filter memory 103 provided in the coefficient analysis unit 10 in association with the calculation position or the interpolation position, for example.

  Next, one factor of resolution degradation in the imaging apparatus 100 as described above will be described with reference to FIGS. FIG. 4 is a diagram illustrating an example of an optical system. FIG. 5 is a diagram illustrating an example of an opening corresponding to an image position. When the optical system shown in FIG. 4 is used, the opening at the center of the optical axis is circular as shown in FIG. However, when the angle of view is large, the opening is scraped and becomes an ellipse depending on the image position. Since the resolution is degraded when the opening is narrowed, the shape of the opening corresponds to the shape of the ellipse of the resolution information in the present embodiment.

  FIG. 6 is a diagram illustrating an example of a blur direction according to an image position. When the optical system shown in FIG. 4 is used, as shown in FIG. 6, since the resolution deteriorates when the aperture is narrowed, the blur tends to deteriorate in the radial direction. Here, the radial direction is a direction from the center of the optical axis toward the end of the image. In FIG. 6, the size of the distance from the center of the ellipse indicates the size of the blur, and the direction from the center of the ellipse to a point on the ellipse indicates the direction of the blur. As shown in FIG. 6, when a concentric circle centered on the center of the optical axis is drawn, the radial blur is large. A direction orthogonal to the radial direction is referred to as a circumferential direction. Large blur means that the resolution is relatively low.

  Next, analysis of resolution related to resolution information in the present embodiment will be described. As described above, the tendency of resolution degradation can be analyzed in detail by photographing a pattern such as the chart 25 in which edges are distributed radially.

  FIG. 7 is a diagram for explaining the resolution when a chart is used. In the example shown in FIG. 7, in order to measure the resolution in the direction of the arrow 26, a plurality of data is acquired in a direction perpendicular to this direction. When the chart shown in FIG. 7 is used, the line width decreases as the distance from the end of the chart 25 toward the center increases, and the number of lines per unit pixel increases. The central part of the chart 25 represents a high frequency component. Further, the amplitude (Intensity) of the luminance value decreases from the end toward the center.

  By using, for example, a wedge-shaped subject that spreads radially like a chart 25 shown in FIG. 7, it becomes possible to analyze the resolution by direction (Modulation Transfer Function: MTF). Note that the chart is not limited to a Siemens star as long as the change in resolution with respect to the spatial frequency can be measured.

  FIG. 8 is a diagram showing the result of measuring the resolution with the chart 25. FIG. 8 shows a graph in which the resolution is measured in the direction indicated by the arrow 26 in FIG. The vertical axis shown in FIG. 8 indicates the amplitude of the luminance value, and the horizontal axis indicates the number of lines per pixel (Line Pair: LP). In this analysis, it is possible to analyze a state (MTF) in which the amplitude decreases toward the center, and deteriorates as the high frequency component (rightward in the horizontal axis) is reached.

  FIG. 9 is a diagram illustrating an example of a difference in resolution degradation depending on a captured image position. In the example shown in FIG. 9, when a plurality of charts are photographed side by side, it is possible to analyze the resolution degradation at the end with respect to the center. In the example shown in FIG. 9, the Dx direction indicates the circumferential direction, and the Dy direction indicates the radial direction. The definition of the Dx direction and the Dy direction is the same in the following drawings.

  FIG. 10 is a diagram illustrating the anisotropy of the resolution in the chart 25. As shown in FIG. 10, it can be seen that, in the peripheral portion including the image edge portion, the resolution is not only deteriorated but also the resolution is anisotropic. That is, in the chart 25, the radial direction is more blurred than the circumferential direction, and the resolution is inferior.

  FIG. 11 is a diagram showing a resolution analysis result of the chart shown in FIG. As shown in FIG. 11, the resolution in the radial direction is worse than the resolution in the circumferential direction. Thereby, it can be seen that there is anisotropy in the resolution at the edge of the image, and the resolution can be measured quantitatively.

  FIG. 12 is a diagram showing the resolution analysis result of the chart in the central portion of the captured image. The chart here is, for example, the chart 27 in FIG. As shown in FIG. 12, there is no significant difference between the resolution in the radial direction and the resolution in the circumferential direction. Therefore, no anisotropy of resolution is observed at the center of the image. As described above, when the resolutions of the charts 25 and 27 are compared, the anisotropy of the resolution is small in the central part, but the anisotropy is larger in the end part.

  In order to correct the blur including resolution degradation as described above, there is a method of correcting using a point spread function (PSF). The PSF can be said to be a function representing the degree of blur, for example. In the present embodiment, the PSF is represented as an ellipse obtained by rotating an ellipse of resolution information by 90 degrees.

  FIG. 13 is a diagram for explaining a PSF calculation method. As shown in FIG. 13, in the chart 25, an ellipse 51 of resolution information is obtained by connecting points having the same amplitude of the luminance values shown in FIG. PSF 53 is obtained by rotating this ellipse 51 by 90 degrees.

Here, if the original image is x and the PSF is k, the blurred image y is an image obtained by convolving x and k, and is expressed by the following Expression 1.

In practice, noise n is included, but is omitted here for the sake of simplicity.
Equation 1 becomes Equation 2 when Fourier transformed.
Y (ω) = K (ω) X (ω) (Formula 2)
Here, ω is a spatial frequency.

Next, the filter Kinv is obtained by the reciprocal of K if it is simply obtained.
Kinv (ω) = 1 / K (ω) (Equation 3)
From this, the Fourier transform X (ω) of the original image is obtained by Equation 4, and the original image is calculated by performing inverse Fourier transform on this.
X (ω) = Kinv (ω) Y (ω) (Formula 4)

  In this way, when the PSF is Fourier-transformed and the filter function (inverse filter) by the reciprocal number is calculated, division by the spatial frequency is performed, so that there is a case where zero division occurs in the high frequency region. Zero division is division by “0” or a value close to “0”. When K (ω) is close to “0” at a high frequency, the reciprocal becomes too large, and high frequency noise is emphasized.

FIG. 14 is a diagram illustrating a characteristic example when the PSF is subjected to Fourier transform. In FIG. 14, the horizontal axis represents the frequency ω, and the vertical axis represents K (ω). As shown in FIG. 14, K (ω) is close to “0” at high frequencies. As described above, when K (ω) is close to “0”, noise at high frequency is expanded in Kinv (ω). Therefore, in order to reduce the noise at the high frequency, a correction term is inserted into the denominator as shown in Equation 5 to prevent emphasis at the high frequency.
Kinv (ω) = 1 / (K (ω) + λ) (Formula 5)
Since the filter is a complex number, it is expressed using a conjugate complex number.

  Here, consider the case where the PSF is an ellipse. FIG. 15 is a diagram illustrating an example of an elliptical PSF. In the example shown in FIG. 15, the resolution is worse in the Dy direction than in the Dx direction. That is, the resolution in the Dy direction is deteriorated compared to the Dx direction.

  The elliptical PSF is represented by k (r, θ). Here, r represents a radius, and θ represents a direction. Thus, the elliptical PSF is expressed as a function of the radius r and the direction θ. When an elliptical PSF is Fourier transformed, K (ω, θ) = fk (r, θ). f represents a Fourier transform. For example, K (ω, θ) after Fourier transform is a function of the spatial frequency ω and the direction θ.

  FIG. 16 is a diagram obtained by Fourier transforming an elliptical PSF. In FIG. 16, the horizontal axis represents the frequency ω, and the vertical axis represents K (ω, θ). As shown in FIG. 16, it can be seen that the resolution characteristics differ between the Dx direction and the Dy direction.

FIG. 17 is a diagram illustrating the reciprocal of K (ω, θ). In FIG. 17, the horizontal axis represents the frequency ω, and the vertical axis represents Kinv (ω, θ). Kinv (ω, θ) is expressed by the following Equation 7. In Kinv (ω, θ), for example, since the denominator is close to “0” at high frequencies, noise at high frequencies is expanded.
Kinv (ω, θ) = 1 / K (ω, θ) (Expression 7)

Therefore, in order to reduce this high frequency noise, a correction term is inserted into the denominator to prevent high frequency emphasis. Equation 8 represents a filter that reduces high frequency noise.

  FIG. 18 is a diagram illustrating an example of a filter when a constant value is added to the denominator as represented by Expression 8. FIG. 19 is a diagram illustrating an example of a filter when the gain is decreased as the frequency increases. In the present embodiment, as shown in FIGS. 18 and 19, it is preferable to reduce the noise by applying a weight to each frequency component.

  As described above, for example, the imaging apparatus 100 obtains a filter at the center position of the five charts 25 and 27 by photographing the correction subject 20. FIG. 20 is a diagram illustrating an example in which a filter is calculated based on a chart. As shown in FIG. 20, the filters 52 and 54 are calculated corresponding to the charts 25 and 28, respectively.

  In the filters 52 and 54, the intensity of the filter is expressed in black and white, and the white is stronger and the black is less intense. It is shown by the shading in the figure that the filters 52 and 54 have anisotropy in the radial direction and the circumferential direction, respectively. In other words, the filters 52 and 54 are filters that have a large radial strength difference and improve the resolution in the radial direction with a large resolution degradation. The filters 52 and 54 have a shape corresponding to an elliptical shape. That is, the resolution information obtained by the chart measurement is reflected in the filter.

  FIG. 21 is a diagram illustrating an example in which nine filters are calculated from nine charts. As shown in FIG. 21, the correction image 55 has nine charts. As described above, the resolution information is obtained by obtaining the MTF for each chart. The resolution information example 57 is an example in which an ellipse of resolution information is shown according to the position of the chart. The resolution information example 57 shows ellipses 61, 64, 65, etc. of resolution information corresponding to the respective charts.

  The filter 70 is a filter calculated based on the PSF obtained by rotating the ellipses 61 and 64 of the resolution information example 57 by 90 degrees. The filter 70 has nine filters 71 to 79 corresponding to the nine charts.

  In the filter 70 of FIG. 21, the intensity is expressed in black and white, the filter intensity is stronger near white and the intensity is smaller near black. In addition, in one filter, the larger the intensity difference is, the larger the resolution degradation is. Further, the intensity distribution of the filter and the ellipse shape of the resolution information correspond to each other.

  The filters 71-79 are designed to improve the direction in which the resolution is poor. Generally, since the resolution degradation in the radial direction is large, the filter characteristics improve the resolution in the radial direction. Further, the filter tends to have a stronger amplitude intensity as it goes around the correction image 55. The greater the ellipse of the resolution information, the stronger the filter strength.

  However, in the above method, since one filter is calculated from one chart, the information amount is small when the number of charts is small. Therefore, for example, a method of performing a process of calculating a filter by interpolation based on a filter calculated from a chart and correcting an image with more filters is conceivable.

  FIG. 22 is a diagram illustrating an arrangement example of filter calculation positions. As shown in FIG. 22, in the filter calculation position arrangement example 59, nine filter calculation positions 67, 68 and the like are shown. In the filter calculation position arrangement example 80, 108 filter calculation positions 81, 83, etc. are shown. In this way, the calculation position of the filter is increased.

  FIG. 23 is a diagram illustrating a comparative example of filter interpolation processing. The filter 70 shows filters 71 to 79 calculated for the calculation positions of nine filters. The filter calculation example 85 shows filters calculated for the calculation positions of 108 filters by interpolation processing. Each of the filters 71-79 is composed of a finite number of elements. The filter calculation example 85 shows an example of 108 filters calculated by interpolating a finite number of elements of the filter 71 and the filter 74 for comparison.

  FIG. 24 shows a comparative example in which a filter at a position between the filter 71 and the filter 74 shown in FIG. 23 is calculated by interpolation. As shown in FIG. 24, a filter 86 is generated from the filter 71 and the filter 74. It is assumed that the filter 86 is calculated by interpolating a finite number of elements of the filter 71 and the filter 74. At this time, the radial direction based on the calculated position of the calculated filter 86 is the direction of the arrow 87. That is, at the calculation position corresponding to the filter 86, the direction of the arrow 87 is the direction indicating the anisotropy of the resolution. However, the direction indicating the anisotropy of the filter 86 calculated by interpolating elements is the direction of the arrow 88, which is shifted from the direction indicating the actual anisotropy. Thus, since the filter 71 is stronger than the filter 74, the resolution information of the filter 74 is lost, and the resolution information of the filter 71 remains preferentially.

  FIG. 25 shows a comparative example in which a filter at a position between the filter 71 and the filter 75 is calculated by interpolation. FIG. 25 is an example in which the filter 71 and the filter 75 having the same anisotropy direction but a large intensity difference are interpolated. In FIG. 25, as the shape of the filter, an ellipse of resolution information corresponding to each filter is represented. As shown in FIG. 25, when interpolation is performed using filters having the same anisotropy direction, the ellipse shape of the resolution information is given priority to the filter 71 having high strength, and the information of the filter 75 is missing. Become. If data is lost during interpolation as in the example of interpolating filter elements in the above comparative example, the effect of improving the direction of poor resolution is reduced.

  FIG. 26 is a table summarizing comparative examples of filter calculation by interpolation between elements. As shown in FIG. 26, when the filters 71 and 74 are calculated from the ellipses 61 and 64 of the resolution information and the elements are interpolated, the filter 86 that is strongly influenced by the filter 71 is calculated.

  FIG. 27 is a diagram illustrating a comparative example in which filter interpolation is performed by rotating the calculated filter 86. As shown in FIG. 27, since the filter 86 is a finite element filter, information is lost and the anisotropy of the ellipse of the resolution information cannot be calculated.

  FIG. 28 is a diagram for explaining interpolation in the present embodiment. As shown in FIG. 28, in the present embodiment, the interpolation unit 13 interpolates the anisotropy, the resolution value, and the angle based on the shapes of the ellipse 61 and the ellipse 64 of the resolution information. Thereby, the ellipse 68 of the resolution information at the interpolation position is calculated. The coefficient analysis unit 10 calculates the PSF based on the calculated resolution information, and obtains the filter 91, for example.

  At this time, when the ellipse 68 is calculated from the ellipse 61 and the ellipse 64 of the resolution information by interpolation, for example, the resolution information interpolating unit 37, for example, the long axis 91 of the ellipse 61 and the long axis 93 of the ellipse 64 The angle with respect to the horizontal direction is calculated. Then, the resolution information interpolation unit 37 may interpolate the angle of the ellipse 68 based on the calculated angle, each calculated position, and the calculated position of the ellipse 68.

  FIG. 29 is a diagram illustrating filter interpolation according to the present embodiment. As shown in FIG. 29, the resolution information example 57 shows ellipses 61, 62, 64, 65, etc. of the resolution information at each position calculated based on the correction image 55.

  The resolution information interpolation example 99 shows the interpolated resolution information represented by an ellipse 68, for example, interpolated based on the resolution information example 57. In the resolution information interpolation example 99, only a part of the ellipse is displayed, but the ellipse is calculated by performing the same interpolation over the entire surface. The filter calculation example 109 shows an example of a filter calculated based on the resolution information in the resolution information interpolation example 99. Here, the magnitude of the filter value is represented by luminance. In the example of FIG. 29, 108 filters are calculated from nine pieces of resolution information.

  FIG. 30 is a diagram illustrating an example of the filter table. As shown in FIG. 30, the filter table 125 is a table in which filter calculation positions and filters are associated with each other. The filter table 125 stores a filter calculated based on the correction image 55. Furthermore, a filter calculated from resolution information interpolated based on resolution information calculated based on the correction image 55 may be stored. Note that the calculated filter is used, for example, to correct an image of an area including the calculated position. The area including the calculation position can be, for example, a rectangular area including the calculation position in the center.

  FIG. 31 is a diagram illustrating an example of a resolution table. The resolution table 127 is a table in which the resolution information calculation position is associated with the resolution value η, anisotropy μ, and angle θ of the calculated resolution information.

  FIG. 32 is a diagram illustrating an example of comparison of filter calculation examples. 32, a filter calculation example 85 is a comparative example different from the calculation method of the present embodiment, which is calculated by interpolating filter elements as shown in FIGS. 23 to 27 described above. In contrast, the filter calculation example 109 is a calculation example according to this embodiment in which resolution information is interpolated and a filter is calculated based on the interpolated resolution information.

  As shown in FIG. 32, in the filter calculation example 85 calculated by interpolation of filter elements from only the original nine filters, the angle information at the original nine sampling positions remains, and the accuracy is low. bad. However, as in the filter calculation example 109, by calculating the angle of the resolution information at the calculation position of each filter and calculating the filter coefficient, anisotropic resolution degradation can be accurately corrected.

  Thus, since the filter calculation example 85 is not based on the resolution information at each filter calculation position, the resolution degradation cannot be corrected with high accuracy. On the other hand, in the filter calculation example 109, since the resolution information is calculated at each calculation position, the anisotropy of the resolution can be improved with high accuracy.

  Hereinafter, the image processing method according to the present embodiment will be further described with reference to a flowchart. FIG. 33 is a flowchart showing the overall processing of the imaging apparatus 100. The following processing will be described assuming that each function shown in FIGS. 1 to 38 is performed.

  As shown in FIG. 33, the resolution calculation unit 101 of the coefficient analysis unit 10 reads, for example, an image obtained by photographing the correction subject 20 from the post-processing unit 5 (S131). The resolution calculation unit 101 extracts, for example, a chart portion such as the chart 25 from the correction image 55 or the like (S132). The resolution calculation unit 101 calculates resolution information based on the extracted chart portion image and stores the resolution information in, for example, the resolution table 127 of the filter memory 103 (S133).

  The interpolation unit 13 reads the resolution information calculated by the resolution calculation unit 101 from the resolution information table 127 and performs an interpolation process (S134). Details of the interpolation processing will be described later. Further, the interpolation unit 13 outputs the resolution information subjected to the interpolation process to the coefficient analysis unit 10. The coefficient analysis unit 10 may store the interpolated resolution information in the resolution information table 127. In the coefficient analysis unit 10, the filter calculation unit 102 calculates a PSF based on the resolution information stored in the resolution information table 127, and obtains filter data of the filter (S135).

  FIG. 34 is a flowchart showing the interpolation processing in the interpolation unit 13. As shown in FIG. 34, in the interpolation unit 13, the interpolation position specifying unit 35 specifies a filter calculation position that is an interpolation position for calculating filter data by interpolation (S141). The interpolation source specifying unit 36 specifies the interpolation source position corresponding to the resolution information used for the interpolation from the calculated position corresponding to the already calculated resolution information based on the specified interpolation position (S142). At this time, the interpolation source specifying unit 36 reads resolution information corresponding to a value to be specified from the resolution information table 127.

  The resolution information interpolation unit 37 performs an interpolation process for calculating the resolution of the position specified based on the resolution information read by a predetermined method (S143). The resolution information interpolation unit 37 outputs the calculated resolution information to the coefficient analysis unit 10.

  As described above, according to the imaging apparatus 100 according to the first embodiment, more than a certain number of image correction filters are interpolated based on an image obtained by photographing the correction subject 20 having a certain number of charts. Calculated. In the case of interpolation, resolution information indicating the degree of blur at the calculated position corresponding to the chart position of the image actually captured by the imaging apparatus 100 is used. The imaging apparatus 100 calculates a filter at a calculation position by interpolation based on the interpolated resolution information.

  As described above, the imaging apparatus 100 according to the first embodiment calculates resolution information indicating the degree of blur at a calculation position other than the position of the chart in the correction subject 20 by interpolation. As a result, filter data at more calculation positions than the number of original charts is calculated. At this time, the calculated filter data reflects information indicating resolution anisotropy. In this embodiment, since the resolution information is calculated at each calculation position, it is possible to generate a filter that improves the resolution anisotropy with high accuracy. Therefore, according to the imaging device 100, it is possible to calculate a filter that can accurately correct anisotropic resolution degradation in a captured image.

  At this time, since the resolution information is calculated from the image obtained by photographing the correction subject 20, for example, deterioration of the peripheral portion of the image, data on the lens surface shape, thickness, glass material, etc. of the optical system are in a predetermined format. It is not necessary to calculate based on the lens data described in. Furthermore, since the correction filter is calculated based on the actual image, even when the degradation of the peripheral portion actually obtained differs from the result obtained from the lens data due to the manufacturing error of the lens and the photographing conditions, It can be set as a filter which corrects accurately.

  In this embodiment, the imaging device 100 captures a chart that is large enough to measure the resolution, and calculates filters at more calculation positions than the original number of charts based on the captured image. Therefore, it is not necessary to arrange many charts in order to obtain filters at many positions, and when the resolution is calculated from the image, a situation where the chart is too small and difficult to calculate is avoided. In addition, a driving mechanism other than the imaging device is not required, such as shooting while moving a large chart.

  When performing the filter calculation by interpolation, the resolution information is interpolated, so that there is no step at the block boundary as in the example of using the same filter for each block. In addition, the problem of calculating a filter that is affected by the stronger filter, such as occurs when the filter elements are interpolated, and in which the angle information of resolution degradation is not appropriately interpolated is also avoided.

  As described above, the blur state is calculated based on the image obtained by photographing the chart, and the image correction filter is calculated based on the result of interpolating the resolution information indicating the blur state. The anisotropy of can be improved with high accuracy.

(Modification of the first embodiment)
Hereinafter, modifications of the imaging device 100 according to the first embodiment will be described. In this modification, the same configurations and operations as those of the first embodiment are denoted by the same reference numerals, and redundant description is omitted.

  This modification is a modification in the resolution information interpolation process. The imaging apparatus according to the present modification includes an interpolation unit 14 instead of the interpolation unit 13 in the imaging apparatus 100. FIG. 35 is a block diagram illustrating an example of the configuration of the interpolation unit 14 according to the present modification. The interpolation unit 14 includes an elliptical shape interpolation unit 38 and an angle calculation unit 39 instead of the resolution information interpolation unit 37 of the interpolation unit 13. The elliptical shape interpolation unit 38 interpolates the resolution value and the degree of anisotropy in the resolution information. This modification is different from the first embodiment in that the angle calculation unit 39 obtains the ellipse angle of the resolution information geometrically instead of interpolation.

36 and 37 are diagrams for explaining an angle calculation method according to this modification. As shown in FIG. 36, the image 96 is an image having nine charts. In the image 96, it is assumed that the center of the optical axis corresponds to the point 98. At this time, the case where the angle of the resolution information at the calculation position 97 is calculated will be described. The calculation position 97 is a position corresponding to the upper left chart of the image 96. The distance r represents the distance between the point 98 and the calculated position 97. In the image 96, the coordinates (the distances in the horizontal direction and the vertical direction) of the calculation position 97 with the point 98 as the origin are represented as (dx, dy). At this time, the angle θ in the radial direction corresponding to the calculated position 97 is expressed by Equation 9 below.
θ = tan ⁻¹ (dx / dy) (Equation 9)

  As shown in FIG. 37, for example, when the calculation positions are 108 in total, for example, the angle θ1 of the resolution information at the calculation position 111 and the angle θ2 at the calculation position 113 are obtained when a concentric circle is drawn about the center 117. The angle can correspond to the radial direction. Therefore, by specifying the coordinates of each calculation point, the angle of the resolution information can be geometrically calculated using Equation 9.

  The angle of the resolution information is calculated by the above processing. The coefficient analysis unit 10 calculates the PSF based on the resolution information including the angle calculated as described above and the resolution value and the anisotropy calculated by the interpolation described in the first embodiment. Based on this, filter data is calculated.

  As described above, according to the modification of the first embodiment, when the resolution information is obtained by interpolation, the angle is interpolated from the angle of the ellipse in the calculated resolution information and the positional relationship between the calculated positions. Instead, it can be obtained from the coordinates of the interpolation position in the image. Note that the resolution value may be obtained by interpolation or may be obtained according to the distance from the center. In addition, the vicinity of the center may have a filter strength of “0” (no correction is performed).

  In the first embodiment and the modification, the example in which interpolation is performed based on two pieces of resolution information has been described. However, for example, other pieces of resolution information such as four pieces may be used. Further, as will be described later, the calculation of the correction filter can be executed by, for example, a standard computer different from the imaging apparatus 100.

(Second Embodiment)
Next, an imaging device according to a second embodiment will be described. In the present embodiment, the same reference numerals are given to the same configurations and operations as those in the first embodiment or the modified example thereof, and the duplicate description is omitted. The second embodiment is an embodiment in the case where image correction is performed using a filter calculated by the first embodiment or a modification thereof, for example.

  FIG. 38 is a block diagram illustrating an example of a schematic configuration of an imaging apparatus 200 that corrects an image using, for example, a filter generated using the imaging apparatus 100 according to the first embodiment. In FIG. 38, the same components as those of the imaging device 100 of FIG. The imaging device 200 includes an image processing unit 15 instead of the image processing unit 4 of the imaging device 100.

  The image processing unit 15 includes a RAW memory 41, a filter control unit 43, and a filter processing unit 45. For example, the filter control unit 43 holds the filter table 125 created by the imaging device 100. The filter control unit 43 refers to the filter table 125, identifies a filter corresponding to the position of the processing target image, and outputs the filter to the filter processing unit 45. That is, the filter control unit 43 outputs each filter corresponding to each position of the processing target image to the filter processing unit 45. The filter processing unit 45 performs filtering at the corresponding image position using the filter acquired from the filter control unit 43. Note that the calculated filter is used, for example, to correct an image of an area including the calculated position. Therefore, the corresponding image position becomes each pixel position in the area including the calculated position. In this way, the imaging apparatus 200 restores the entire image by correcting each region including the calculated position, for example, with a filter calculated according to each calculated position calculated by the imaging apparatus 100.

  With the configuration described above, the imaging apparatus 200 can correct and output an image using the filter generated according to the first embodiment or a modification thereof. Thereby, it is possible to improve the image quality while improving the anisotropy of different resolutions at each position of the image. Therefore, according to the imaging apparatus 200, it is possible to accurately correct anisotropic resolution degradation in a captured image.

  Note that the imaging apparatus 100 can have a function for calibrating the imaging apparatus 200, for example. In the imaging apparatus 200, for example, the filter control unit 43 may have the functions of the coefficient analysis unit 10 and the interpolation unit 13 according to the first embodiment to generate the filter table 125.

(Third embodiment)
Next, an imaging apparatus according to a third embodiment will be described. In the present embodiment, the same configurations and operations as those of the first embodiment are denoted by the same reference numerals, and redundant description is omitted. The third embodiment is another embodiment relating to the coefficient analysis of the first embodiment and its modifications.

  FIG. 39 is a block diagram illustrating an example of the configuration of the coefficient analysis unit 11 according to the third embodiment. The imaging device according to the third embodiment includes this coefficient analysis unit 11 instead of the coefficient analysis unit 10 of the imaging device 100. The coefficient analysis unit 11 includes a resolution calculation unit 101, a filter memory 103, and a filter calculation unit 104. The filter calculation unit 104 includes an adjustment unit 121, an image correction unit 122, and a coefficient determination unit 123.

  For example, the resolution calculation unit 101 analyzes at least two directions of resolution degradation of an image in which a subject having a radial pattern (such as the correction subject 20) is captured. At this time, the resolution calculation unit 101 calculates the resolution information as described above. Thereby, at least the resolution anisotropy in the imaging element 2 direction is analyzed. The filter calculation unit 104 calculates filter data based on the resolution information analyzed by the resolution calculation unit 101. The adjustment unit 121 adjusts the weighting coefficient ε that does not depend on the direction and the weighting coefficient γ that depends on the direction, and outputs the result to the image correction unit 122. The image correction unit 122 corrects the image with the filter data calculated by the adjustment unit 121. The image correction unit 122 outputs the corrected image to the resolution calculation unit 101, and the resolution calculation unit 101 performs the above-described processing on the corrected image.

  The coefficient determining unit 123 determines the weighting coefficient based on the resolution analysis result for the corrected image so that the difference in resolution degradation between the two directions becomes small. The coefficient determination unit 123 holds the analysis result of the image corrected by various weighting coefficients, and determines the weighting coefficients ε and γ so that the difference in the spatial frequency value is minimized at a predetermined amplitude intensity, for example. To do.

  Hereinafter, the filter calculated by the coefficient analysis unit 11 will be described. The filter calculation unit 104 calculates a filter that improves resolution anisotropy. Hereinafter, a calculation procedure of a filter that improves the anisotropy of the resolution, for example, performs adjustment in a direction with a poor resolution will be described.

Consider the original image x, PSF: k, and blurred image y as shown in Equation 1. When obtaining the original image x, if Equation 10 is minimized as an inverse problem, an image close to the original image is obtained.
Usually, some regularization term is applied when solving the inverse problem. Therefore, the regularization term is added and the inverse problem is solved from Equation 11.

Since the problem this time requires directionality, differential terms in the horizontal (horizontal) direction (x direction) and vertical (vertical) direction (y direction) of the image are added as regularization terms.
ε: weighting coefficient dm, dn: differential filter in matrix direction

In order to minimize Expression 12, the result of partial differentiation of Expression 12 by x may be set to “0”. When Fourier transform is performed and X (ω) is solved, the following Expression 13 is obtained.
X (ω) = K (ω) Y (ω) / (K (ω) ² + ε {Dm (ω) ² + Dn (ω) ² })
... (Formula 13)
X (ω), Y (ω), K (ω), Dm (ω), and Dn (ω) represent Fourier transforms of x, y, k, dm, and dn, respectively.

The filter Kinv (ω) in the frequency domain satisfies Expression 14.
X (ω) = Kinv (ω) Y (ω) (Equation 14)

Therefore, the filter Kinv (ω) satisfies the following Expression 15.
Kinv (ω) = K (ω) / (K (ω) ² + ε {Dm (ω) ² + Dn (ω) ² })
... (Formula 15)
Expression 15 becomes Expression 16 when a conjugate complex number is used.

In the present embodiment, in order to adjust the direction with poor resolution, the axis of the differential coefficient is rotated in the direction of the angle θ using the rotation matrix.
Dx (ω, θ) = Dm (ω) cos θ−Dn (ω) sin θ (Expression 17)
Dy (ω, θ) = Dm (ω) sinθ−Dn (ω) cosθ (Expression 18)
As described above, using a rotation matrix can provide directionality. According to Equations 17 and 18, the Dn direction is rotated by θ to become the Dy direction, and the Dm direction is rotated by θ and becomes the Dx direction.

Here, it is assumed that the ellipse PSF is k (r, θ) and the ellipse PSF after Fourier transform is K (ω, θ) = fk (r, θ). When Expression 17, Expression 18, and K (ω, θ) are substituted into Expression 16, and the weight γ according to the direction is set, Expression 19 is established.
γ: weighting factor according to filter direction ε: overall weighting factor

  This equation 19 makes it possible to adjust the weight of the directionality of the filter used in each embodiment. For example, the coefficient analysis unit 11 adjusts the weighting coefficient γ in the direction with the poor resolution (Dy direction). For example, by reducing the weighting factor γ, it is possible to improve the radial resolution with poor resolution. In this way, the resolution anisotropy can be improved by adjusting the weighting factor.

  Hereinafter, the process by the coefficient determination unit 123 will be further described. 40 and 41 are diagrams for explaining an example of a weighting factor determination process. 40 and 41, the horizontal axis represents resolution and the vertical axis represents amplitude intensity. As shown in FIG. 40, the coefficient determination unit 123 determines a weighting coefficient so that the difference in spatial frequency becomes small at a predetermined amplitude intensity (threshold 1) (determination process 1). As shown in FIG. 41, the coefficient determining unit 123 may determine the weighting coefficients ε and γ so that the difference in amplitude intensity is minimized at a predetermined spatial frequency (decision process 2).

  Note that a plurality of threshold values 1 and 2 may be set, and the coefficient determination unit 123 may determine the weighting coefficient so that the sum of squares of the differences at each threshold value is minimized. The coefficient determining unit 123 may determine the weighting coefficient so that the predetermined difference is equal to or less than a preset threshold value. This threshold may be set by a prior experiment or the like.

  Also, the coefficient determination unit 123 determines that the difference between the difference square sum of the two-direction resolution at the center of the image and the difference square sum of the two-direction resolution at the peripheral portion of the image different from the image central portion You may determine a weighting coefficient so that it may become. The coefficient determination unit 123 may determine the weighting coefficient so that the sum of squares of the difference between the image center and the image periphery is minimized. As a result, when the resolution anisotropy is reduced, the resolution of the central portion of the image and the resolution of the peripheral portion of the image are made the same, so that the resolution of the entire image becomes the same, and the image quality can be improved. The determination of minimization by the coefficient determination unit 123 may be calculated using a minimization function. Examples of the minimizing function include a simplex search method, a steepest descent method, and a conjugate gradient method.

  As described above, the filter calculation unit 104 changes and adjusts the weighting factor, obtains an inverse filter with the adjusted weighting factor, corrects the image with the obtained inverse filter, and obtains a resolution analysis result of the corrected image. Based on this, an optimum weighting factor is determined. The adjustment of the weighting factor, calculation of the inverse filter, correction by filtering, and resolution analysis are repeated until the optimum weighting factor is determined.

  42 and 43 are diagrams showing the analysis results of the resolution after image correction according to the third embodiment. 42 and 43, the horizontal axis represents resolution, and the vertical axis represents amplitude intensity. In the example shown in FIG. 42, γ = 1/300. As shown in FIG. 42, the resolution is improved in both the Dx and Dy directions, and the difference between the directions is small. Therefore, the resolution anisotropy can be improved.

  FIG. 43 shows the analysis result of resolution after image correction when γ = 1. The analysis result shown in FIG. 43 is a result in the case of using a filter in which the resolution anisotropy is not corrected by the weighting coefficient. As shown in FIG. 43, the resolution is improved in each direction, but the difference is large, and the anisotropy of the resolution is not improved as much as the example of FIG.

  Hereinafter, an operation of the imaging apparatus according to the third embodiment will be described with reference to a flowchart. The overall operation of the imaging apparatus according to the third embodiment is the same as the operation described with reference to FIG.

  44 and 45 are flowcharts showing filter calculation processing according to the third embodiment. The following processing is described as being performed by each function shown in FIG. The filter calculation process corresponds to the process of S135 in FIG.

  As shown in FIG. 44, the filter calculation unit 104 first sets initial values of the weighting coefficients ε and γ (S151). The adjustment unit 121 and the image correction unit 122 optimize the filter data (S152). The coefficient determination unit 123 determines filter data (S153). The determined filter data is preferably stored in the filter memory 103. This process is preferably repeated until the optimum weighting factor is determined as described above.

  As shown in FIG. 45, the image correction unit 122 first acquires the calculated filter from the adjustment unit 121 (S161). The image correction unit 122 corrects the image with the acquired filter (S162). The resolution calculation unit 101 calculates resolution information in the corrected image (S163). The adjustment unit 121 compares the calculated resolution with a threshold value (S164). The adjustment unit 121 determines whether the resolution has converged (S165).

  If it is determined that the resolution has converged (S165: YES), the process returns to S152 in FIG. If it is not determined that the image has converged (S165: NO), the process returns to S161 and is repeated. Note that the above processing describes filter calculation for one calculation position, and a plurality of filters for the entire image are calculated by repeating the above processing for all calculation positions.

  In the third embodiment, the filter is calculated as described above, but the resolution information is calculated by the resolution calculation unit 101 by interpolating the resolution information as in the first embodiment and the modification. By the method. As the angle calculation method, any method of the first embodiment or the modification can be applied.

  In the third embodiment, when calculating the filter, information other than the angle at each position, for example, the resolution value, the degree of anisotropy, the weighting coefficient, and the like are calculated by interpolating the resolution information. Each value obtained by interpolation is used for filter calculation at each position together with, for example, a geometrically calculated angle. The image is corrected using filters at all the calculated positions.

  As described above, according to the third embodiment, in addition to the effect of the image processing according to the first embodiment, by introducing a weighting factor for filter calculation, the resolution anisotropy is further improved. It becomes possible to improve.

(Fourth embodiment)
Next, an imaging apparatus according to a fourth embodiment will be described. In the present embodiment, the same reference numerals are given to the same configurations and operations as those in the first to third embodiments, and the duplicate description will be omitted. The fourth embodiment is another form relating to resolution information calculation of the first to third embodiments and the modification.

  In the resolution information calculation of the first to third embodiments and the modified examples, the example in which the center of the image coincides with the point having the smallest resolution anisotropy has been described. The present embodiment is an example in the case where the blur state is asymmetric with respect to the center of the image.

  FIG. 46 is a diagram illustrating a case where the blur is asymmetric with respect to the center of the image. As shown in FIG. 46, in the image 170, the graphics 173, 174, 177, and 179 indicating the resolution information are not symmetric with respect to the graphic 175 at the center of the image. Here, the point with the smallest blur and the center coordinate of the blur is a point 172. Also, the left graphics 174 and 177 in FIG. 46 are larger than the right graphics 173 and 179. This indicates that the image 170 is more blurred on the left side of the figure. As described above, when the degree of blur is asymmetrical, the center can be set by the inverse ratio of the resolution value in the resolution information.

  FIG. 47 is a diagram illustrating an example of an angle calculation method when the blur is asymmetric. As shown in FIG. 47, when the center coordinate of the blur is a point 172, the direction from the point 172 toward the calculation position of the resolution information is a radial direction with a poor resolution. In the example of FIG. 47, the direction connecting the point 172 and the point 181 is the direction where the resolution is bad at the point 181. Similarly, the direction connecting the point 172 and the point 183 is the direction where the resolution is poor at the point 183. Based on these bad resolution directions, the angles θ1 and θ2 of the resolution information are calculated.

  FIG. 48 is a diagram showing the effect of the resolution information calculation method according to the present embodiment. In FIG. 48, a filter calculation example 191 is a filter calculated based on an image obtained by photographing the correction subject 20. The filter calculation example 193 is a filter calculated based on the filter calculation example 191 using the resolution information calculation method according to the present embodiment.

  By calculating the angle of the resolution information by shifting the center coordinates as in the filter calculation example 193, a filter having a strong intensity on the left side of the figure as a whole is calculated. In this case, the anisotropy degree and the resolution value of the resolution information may be obtained by interpolation or may be obtained according to the distance from the center.

  As described above, even in the case of an imaging device with an asymmetric degree of blur, it is possible to correct an image with high accuracy by performing filter calculation using the blur center as the center coordinate when determining the angle of resolution information. It becomes. At this time, the center coordinates may be calculated and set by the inverse ratio of the size of the ellipse of the resolution information. In this way, by calculating the angle by shifting the center coordinates, it is possible to calculate a filter having a stronger intensity on the side where the degree of blurring is larger.

  Furthermore, depending on the manufacturing error of the lens and the shooting conditions, the peripheral edge degradation symmetry that is actually obtained is different from the result obtained from the lens data, and the center of the peripheral edge deterioration based on the lens data is different from the center of the captured image. However, it is possible to correct with high accuracy.

(Fifth embodiment)
Next, an imaging apparatus according to a fifth embodiment will be described. In the present embodiment, the same configurations and operations as those in the first to fourth embodiments are denoted by the same reference numerals, and redundant description is omitted. The fifth embodiment is another form of the configuration of the image processing unit 15 of the first to fourth embodiments and the modified example.

  When a filter calculated in the image processing apparatus according to the first to fourth embodiments is mounted, there is a demand to use a filter having a predetermined number of elements or less from the viewpoint of memory and calculation amount. At present, it is considered to use a filter of 5 × 5 or less, for example. However, if the number of elements of the filter is reduced, the size of the PSF representing blur is larger than the number of elements, the amount of information is lost, and the image quality may deteriorate.

  FIG. 49 is a diagram illustrating an example of the relationship between the size of the PSF representing the blur of the image and the filter 215 having the number of elements of 5 × 5. As shown in FIG. 49, it is assumed that the PSF is larger than the size of the 5 × 5 filter 215. One element of the filter corresponds to one pixel of the image. In the example shown in FIG. 49, the number of elements of the filter 215 is smaller than the size of the PSF. For this reason, when image correction is performed using such a filter 215, PSF information is lost, and thus there is a problem of degradation in image correction quality such as generation of moire. Hereinafter, a method for preventing the quality of image correction from being deteriorated due to missing information even when the number of elements is small will be described.

Equation 20 below shows a case where the corrected image x is generated by convolving the calculated filter F with the original image y and performing a difference process with the original image.

As another example, the following formulas 21 to 24 indicate that the corrected image x is generated by convolving the calculated filter F ′ with the original image y.

  First, an outline for preventing the occurrence of moire will be described. As described above, since the loss of high-frequency information differs depending on the direction by making the filter having anisotropy in resolution, moiré occurs. That is, moire occurs because the degree of correction of frequency degradation differs depending on the direction.

  Therefore, by passing through a finite high-pass filter, the portion where the luminance changes abruptly is reduced, and the difference in the correction degree of the high-frequency information that differs depending on the direction is reduced. As a result, it is possible to prevent the occurrence of moire caused by deterioration of frequency information that varies depending on the direction.

  The finite high-pass filter may be a finite filter in which the sum of elements is 0 and at least two elements are non-zero. Hereinafter, a finite high-pass filter will be used as the finite filter. In addition, by dividing the calculated filter and generating a plurality of stages of filters having a predetermined number of elements or less, it is possible to clear mounting problems.

  51 and 52 are diagrams illustrating a configuration example of a plurality of stages of filters. As shown in FIG. 51, the image processing unit 210 corresponds to Expression 20. The image processing unit 210 corrects the original image 213 with the filter set 211 and outputs a corrected image 219. The filter set 211 is configured by dividing the filter F of Expression 20 into a plurality of stages of filters (5 × 5) 215 and 217.

  As illustrated in FIG. 52, the image processing unit 220 corresponds to Expressions 21 to 24. The image processing unit 220 corrects the original image 213 with the filter set 211 and outputs a corrected image 227. The filter set 221 is configured by dividing the filter F ′ of Expression 24 into a plurality of stages of filters (5 × 5) 223 and 225. In the fifth embodiment, anisotropy is improved by configuring a plurality of anisotropic filters for a PSF larger than the number of filter elements and combining the plurality of filters.

  Next, an imaging apparatus including the image processing apparatus according to this embodiment will be described. Here, the filter control units 250 and 280 and the filter processing units 270 and 290 will be described with reference to FIGS. 52 and 53.

  FIG. 52 is a block diagram illustrating an example of the configuration of the filter control unit 250 and the filter processing unit 270. The filter control unit 250 and the filter processing unit 270 have configurations corresponding to the image processing unit 220 in FIG. The filter control unit 250 includes a filter storage unit 251, a filter acquisition unit 252, a filter calculation unit 253, and a filter generation unit 254.

  The filter storage unit 251 stores at least the first filter 261, the second filter 262, and the multistage filter 263. Each filter may be stored in a different storage area.

  The first filter 261 is, for example, a filter having anisotropy calculated based on anisotropic resolution information. The first filter 261 is, for example, each filter of the filter table 125 illustrated in FIG. As described above, the filter of the filter table 125 preferably includes a filter generated by interpolation from a filter generated based on the correction subject 20.

  The second filter 262 is a filter calculated by the filter calculation unit 253. For example, the second filter is a filter in which a high-pass filter is convoluted with the first filter 261. The high pass filter will be described later. The multistage filter 263 is a filter group generated by the filter generation unit 254.

  The filter acquisition unit 252 acquires a finite filter having anisotropy calculated based on anisotropic resolution information. For example, the filter acquisition unit 252 acquires the first filter 211 from the filter storage unit 251. The filter acquisition unit 252 outputs the acquired first filter 261 to the filter calculation unit 253.

  The filter calculation unit 253 calculates the second filter by convolving a first filter 261 acquired from the filter acquisition unit 252 with a finite filter in which the sum of elements is 0 and at least two elements are non-zero. .

  At this time, the filter calculation unit 253 calculates the second filter having a larger number of elements than the size of the blur (the size of the PSF ellipse). The filter calculation unit 253 includes a determination unit 265, and the determination unit 265 determines the size of the blur from the size of the PSF (or ellipse of resolution information) stored in the filter storage unit 251, for example. Note that when the PSF can be acquired by simulation from the lens design value, the size of the blur may be determined from the acquired PSF. For example, the filter calculation unit 253 calculates the second filter by convolving a finite high-pass filter with the first filter having a larger number of elements than the PSF of the image.

The filter calculation unit 253 preferably holds a finite high-pass filter in advance. If the finite high-pass filter is, for example, a 3 × 3 filter, the following Expression 25 or Expression 26 may be used.

  Since the anisotropy handled in the present embodiment is a filter according to an angle in an arbitrary direction, it is preferable that all elements have non-zero coefficients as shown in Expression (26).

  If the high-pass filter is 3 × 3, the filter calculation unit 253 calculates a 9 × 9 filter by convolving the two filters by setting the filter Kinv to 7 × 7. As described above, the filter calculation unit 253 convolves the high-pass filter and the filter, and calculates the desired number of elements.

Here, if the 7 × 7 filter is expressed as F7 and the high-pass filter is expressed as Lap, the 9 × 9 filter F9 calculated by the filter calculation unit 253 is expressed by Expression 27 below.

  The filter calculation unit 253 stores the second filter F9 calculated by Expression 27 described above in the filter storage unit 251. The filter calculation unit 253 may be provided in another device, and the filter control unit 250 may store the second filter 262 obtained by another device.

  The filter generation unit 254 acquires the second filter 262 from the filter storage unit 251, and generates a plurality of stages of filters having a predetermined number of elements or less from the second filter 262. The generation of the multi-stage filter will be described later. The filter generation unit 254 stores the generated multistage filter 263 in the filter storage unit 251.

  Next, the filter processing unit 270 will be described. The filter processing unit 270 includes a convolution operation unit 271 and a subtraction unit 272. For example, the convolution operation unit 271 acquires an image from the RAW memory 41 illustrated in FIG. 38 and performs a filter process for convolving the image with the multistage filter 263.

  In the convolution operation unit 271, the multistage filter 263 may perform the filter processing by performing a plurality of times of processing on one convolution circuit, or may perform the filter processing by preparing a plurality of convolution circuits. . The convolution operation unit 271 outputs the filtered image to the subtraction unit 272.

  The subtraction unit 272 subtracts the filtered image from the image acquired from the RAW memory 41 to generate a corrected image. The corrected image is output to the post-processing unit 5 and the image memory 8.

  Next, the filter control unit 280 and the filter processing unit 290 corresponding to Equations 21 to 24 will be described. FIG. 53 is a block diagram illustrating an example of a schematic configuration of the filter control unit 280 and the filter processing unit 290 according to the fifth embodiment. First, the filter control unit 280 will be described. The filter control unit 280 includes a filter storage unit 284, a filter acquisition unit 202, a filter calculation unit 282, and a filter generation unit 283.

  In the configuration of the filter control unit 280, the same components as those shown in FIG. The filter storage unit 284 stores the third filter 285 calculated by the filter calculation unit 282 and the multistage filter 286 calculated by the filter generation unit 283.

  The filter calculation unit 282 calculates the third filter 285 that eliminates the above-described subtraction process between images. The filter calculation unit 282 can eliminate the subtraction process by performing equation transformation using Equations 21 to 24. In this example, it is assumed that a 9 × 9 third filter (F9 ′) is generated. As a result, the same result as the above-described processing can be obtained, and one finite filter (F9 ′) that does not require subtraction processing between images can be generated. The filter calculation unit 282 stores the calculated filter F9 ′ in the filter storage unit 284.

  The filter generation unit 283 acquires the third filter 285 from the filter storage unit 284, and generates a plurality of stages of filters having a predetermined number of elements or less from the third filter 285. The generation of the multi-stage filter will be described later. The filter generation unit 283 stores the generated multistage filter 286 in the filter storage unit 284. The filter processing unit 290 has a convolution operation unit 291. The convolution operation unit 291 performs a process of convolving the multistage filter 286 to generate a corrected image x.

  Next, generation of a plurality of stages of filters will be described. In the following, an example in which the filter generation unit 283 illustrated in FIG. 53 generates a plurality of stages of filters will be described. However, the filter generation unit 254 illustrated in FIG. This processing can also be performed by the coefficient analysis unit 10 in FIG.

  First, when the filter generation unit 283 divides a filter having the number of elements larger than the PSF into a filter group having a number of elements equal to or less than a predetermined number of elements, the filter generation unit 283 assumes one filter and selects other filters based on the assumed filter. Find by optimization.

For example, the third filter F9 ′ is divided into two 5 × 5 filters 223 (F5A ′) and a filter 225 (F5B ′). At this time, the filter generation unit 283 can define the following Expression 28.
e is a differential error. Hereinafter, an example of a dividing method into a plurality of stages of filters will be described.

As an example of the filter dividing method, the filter 223 is a filter obtained by extracting the number of 5 × 5 elements in the central portion of the third filter F9 ′. In this division method, the filter generation unit 283 can determine the filter 223, and can obtain the filter 225 by minimizing the evaluation function represented by the following Expression 29.

Using the filter obtained here, a corrected image x is further generated based on Expression 30 below.

At this time, the difference error e does not become “0”. Therefore, accuracy can be improved by setting a gain for each of the plurality of stages of filters. At this time, the correction image x is expressed by the following Expression 31 using the gains G1 and G2.

  The gains G1 and G2 can be determined as follows. For example, first, the image processing unit 15 sets initial values of the gains G1 and G2. Using this gain, image correction is performed by Equation 31 to obtain resolution information. Based on this, the filter is calculated as described above, and the value of Equation 29 is calculated. The above is repeated and the gains G1 and G2 that minimize the value of Expression 29 are determined.

  54 and 55 are diagrams illustrating an example of filter division. 54 and 55, the size of the filter is represented by a luminance value. FIG. 54 shows that the filter example 231 is divided into a filter example 233 and a filter example 235. Here, the filter example 231 includes nine 9 × 9 filters. The filter example 233 and the filter example 235 each have nine 5 × 5 filters.

  FIG. 55 shows that the filter example 241 is divided into a filter example 243 and a filter example 245. Here, the filter example 241 includes 108 9 × 9 filters. Filter example 243 and filter example 245 each have 108 5 × 5 filters.

  As described above, according to the image processing method of the fifth embodiment, the filter can be divided into a plurality of filters having a small number of elements. Therefore, even when a filter with a small number of elements is used due to the limitation of the calculation amount, it is possible to prevent a reduction in correction quality that occurs when the number of elements of the filter is smaller than the PSF. Further, a gain corresponding to each of the divided filters is set. As a result, an image correction filter with higher accuracy can be calculated.

  As described above, even when the filter is divided into a plurality of stages, the resolution information indicating the degree of blur is calculated at the calculation position corresponding to each chart in the image obtained by photographing the correction subject 20, and the plurality of stages are calculated after the filter is calculated. Divide into filters.

  The method for dividing the filter is not limited to the above, and other methods may be used. The above example is also an example regarding the number of filter elements after division, the coefficient, and the method of obtaining the gain, and is not limited to the description example.

(Modification of the number of charts)
Hereinafter, the number of charts in the correction subject 20 will be described. 56 to 59 are diagrams showing an example of the chart arrangement in the correction subject 20. In the first to fifth embodiments and the modifications described above, chart photographing requires a chart near the center serving as a reference with little resolution degradation and a chart around the periphery with great degradation.

  As shown in FIG. 56, in the chart arrangement example 351, charts are arranged at the center and at the four corners. In order to perform the image processing, it is preferable that there are at least five charts arranged as in the chart arrangement example 351. For example, when the image is point-symmetric, if there is a center and one of the four corners, the arrangement is substantially the same as the chart arrangement example 351.

  As shown in FIG. 57, in the chart arrangement example 353, twelve charts are arranged in 3 × 4 length and width. In the case of the chart arrangement example 353, there is no chart arranged at the reference center, but one chart near the center or a plurality near the center is used as a reference.

  As shown in FIG. 58, in the chart arrangement example 355, 5 × 5 = 25 charts are arranged. As an example of the chart arrangement, an example as shown in FIG. 58 may be used.

  FIG. 59 shows a chart arrangement example 357 having 12 × 9 = 108 charts. For example, in the first embodiment, an example in which 108 filters are calculated as a whole based on nine charts has been described with reference to FIG. In the chart arrangement example 357, 108 charts are arranged, so that 108 filters can be obtained without using interpolation. However, with such a large number, the resolution cannot be measured depending on the performance of the imaging apparatus, and a limited number may be required.

  As described above, various numbers of charts to be arranged on the correction subject 20 can be applied, but it is preferable to be determined based on the properties and performance of the imaging apparatus.

  The filter calculation in the first to fifth embodiments and modifications may be performed by an individual information processing apparatus such as a standard computer. The captured image of the correction subject 20 is transferred to the server by the terminal or the imaging device via the communication network, the filter calculation is performed in the server, and the filter calculation result is transferred to the terminal or the imaging device to perform image correction processing. You may do it. The terminal or the imaging apparatus may transfer filter information of the number of shots calculated from the chart shot image to the server, and calculate a filter larger than the number of shots in the server.

  Here, an example of a computer that is commonly applied to cause the computer to perform the operations of the image processing methods according to the first to fifth embodiments and the modifications will be described. FIG. 60 is a block diagram illustrating an example of a hardware configuration of a standard computer. As shown in FIG. 60, a computer 300 includes a central processing unit (CPU) 302, a memory 304, an input device 306, an output device 308, an external storage device 312, a medium driving device 314, a network connection device, and the like via a bus 310. It is connected.

  The CPU 302 is an arithmetic processing unit that controls the operation of the entire computer 300. The memory 304 is a storage unit for storing in advance a program for controlling the operation of the computer 300 or using it as a work area when necessary when executing the program. The memory 304 is, for example, a random access memory (RAM), a read only memory (ROM), or the like. The input device 306 is a device that, when operated by a computer user, acquires various information input from the user associated with the operation content and sends the acquired input information to the CPU 302. Keyboard device, mouse device, etc. The output device 308 is a device that outputs a processing result by the computer 300, and includes a display device and the like. For example, the display device displays text and images according to display data sent by the CPU 302.

  The external storage device 312 is a storage device such as a hard disk, and stores various control programs executed by the CPU 302, acquired data, and the like. The medium driving device 314 is a device for writing to and reading from the portable recording medium 316. The CPU 302 can perform various control processes by reading and executing a predetermined control program recorded on the portable recording medium 316 via the medium driving device 314. The portable recording medium 316 is, for example, a Compact Disc (CD) -ROM, a Digital Versatile Disc (DVD), a Universal Serial Bus (USB) memory, or the like. The network connection device 318 is an interface device that manages transmission / reception of various data performed between the outside by wired or wireless. A bus 310 is a communication path for connecting the above devices and the like to exchange data.

  A program that causes a computer to execute the image processing methods according to the first to fifth embodiments is stored in, for example, the external storage device 312. The CPU 302 reads a program from the external storage device 312 and causes the computer 300 to perform an image processing operation. At this time, first, a control program for causing the CPU 302 to perform image processing is created and stored in the external storage device 312. Then, a predetermined instruction is given from the input device 306 to the CPU 302 so that the control program is read from the external storage device 312 and executed. The program may be stored in the portable recording medium 316.

  The present invention is not limited to the embodiments described above, and various configurations or embodiments can be adopted without departing from the gist of the present invention.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
Resolution information indicating resolution information corresponding to at least two points on the image, respectively, based on the image;
Resolution information corresponding to one point on the image other than the at least two points is calculated based on the resolution information corresponding to each of the at least two points and the position of the at least two points and the one point on the image. An interpolation unit;
A filter calculating unit that calculates a filter for correcting a region including the one point of the image according to the resolution information corresponding to the one point;
An image processing apparatus comprising:
(Appendix 2)
The resolution information has a resolution value corresponding to the resolution corresponding to each point of the image, an anisotropy indicating the anisotropy of the resolution, and an angle corresponding to the anisotropy of the resolution,
The resolution value, the degree of anisotropy, and the angle include a pattern in which lines with different intervals and widths are sequentially provided at equidistant positions in each of at least two directions with respect to each of the at least two points. Supplementary note 1, which corresponds to the size, ellipticity, and angle of the ellipse formed by the distribution of points having the same amplitude of luminance values corresponding to the pattern in the image of the subject. An image processing apparatus according to 1.
(Appendix 3)
The image processing apparatus according to appendix 2, wherein the interpolation unit geometrically calculates the angle corresponding to the one point based on a position on the image of the one point.
(Appendix 4)
The angle is a direction connecting a point at which the anisotropy degree on the image is most isotropic and a calculation position of the filter, according to any one of appendix 2 and appendix 3. Image processing device.
(Appendix 5)
The image processing apparatus according to any one of Supplementary Note 2 to Supplementary Note 4, wherein the filter calculation unit determines a weighting factor according to the angle.
(Appendix 6)
The image processing apparatus according to appendix 3, wherein the weighting factor is determined such that a difference in resolution between the direction of the angle and a direction orthogonal to the direction of the angle is minimized.
(Appendix 7)
If the filter has a first finite number of elements, and the range corresponding to the number of elements is greater than the ellipse, the filter may be reduced to a second finite number less than the first finite number. The image processing apparatus according to any one of appendix 2 to appendix 6, wherein correction is performed by dividing the filter into a plurality of filters having elements and convolving the image.
(Appendix 8)
Resolution information indicating the resolution status corresponding to at least two points on the image is calculated based on the image,
Resolution information corresponding to one point on the image other than the at least two points is calculated based on the resolution information corresponding to each of the at least two points and the position of the at least two points and the one point on the image. ,
Calculating a filter for correcting a region including the one point of the image according to the resolution information;
An image processing method.
(Appendix 9)
The resolution information has a resolution value corresponding to the resolution at each point of the image, an anisotropy indicating the anisotropy of the resolution, and an angle corresponding to the anisotropy of the resolution,
The resolution value, the degree of anisotropy, and the angle calculated on the basis of the image are sequentially equidistant from each other at least in two directions with respect to each of the at least two points. Corresponding to the size of the ellipse formed by the distribution of points having the same amplitude of the luminance value corresponding to the pattern, the ellipticity, and the angle of the ellipse in the image in the image obtained by photographing the subject including the pattern provided at the position The image processing method according to appendix 8, wherein
(Appendix 10)
The image processing method according to claim 9, wherein the angle corresponding to the one point is geometrically calculated based on a position of the one point on the image.
(Appendix 11)
The angle is a direction connecting a point where the degree of anisotropy on the image is most isotropic and a calculation position of the filter, The appendix 9 or appendix 10, characterized in that Image processing method.
(Appendix 12)
12. The image processing method according to any one of appendix 9 to appendix 11, wherein the filter is calculated by determining a weighting factor according to the angle.
(Appendix 13)
13. The image processing method according to claim 12, wherein the weighting factor is determined so that a difference in resolution between the direction of the angle and a direction orthogonal to the direction of the angle is minimized.
(Appendix 14)
If the filter has a first finite number of elements, and the range corresponding to the number of elements is greater than the ellipse, the filter may be reduced to a second finite number less than the first finite number. The image processing method according to any one of appendix 9 to appendix 14, wherein correction is performed by dividing the filter into a plurality of filters having elements and convolving the image.
(Appendix 15)
Resolution information indicating the resolution status corresponding to at least two points on the image is calculated based on the image,
Resolution information corresponding to one point on the image other than the at least two points is calculated based on the resolution information corresponding to each of the at least two points and the position of the at least two points and the one point on the image. ,
Calculating a filter for correcting a region including the one point of the image according to the resolution information;
A program that causes a computer to execute processing.

1 Optical System 2 Image Sensor 3 AFE
4 Image processing unit 5 Post processing unit 6 Drive control unit 7 Control unit 8 Image memory 9 Display unit 10 Coefficient analysis unit 11 Lens 11a Lens 11b Lens 11c Lens 12 Aperture 13 Interpolation unit 15 Image processing unit 20 Correction subject 25 Chart 31 A / D converter 32 Timing generator 35 Interpolation position specifying unit 36 Interpolation source specifying unit 37 Resolution information interpolation unit 41 RAW memory 43 Filter control unit 45 Filter processing unit 51 Ellipse 53 PSF
55 Image for Correction 57 Resolution Information Example 59 Filter Calculation Position Arrangement Example 100 Imaging Device 101 Resolution Calculation Unit 102 Filter Calculation Unit 103 Filter Memory 109 Filter Calculation Example 121 Adjustment Unit 122 Image Correction Unit 123 Coefficient Determination Unit 125 Filter Table 127 Resolution Information Table

Claims (8)

Resolution information indicating resolution information corresponding to at least two points on the image, respectively, based on the image;
Resolution information corresponding to one point on the image other than the at least two points is calculated based on the resolution information corresponding to each of the at least two points and the position of the at least two points and the one point on the image. An interpolation unit;
A filter calculating unit that calculates a filter for correcting a region including the one point of the image according to the resolution information corresponding to the one point;
I have a,
The resolution information has a resolution value corresponding to the resolution corresponding to each point of the image, an anisotropy indicating the anisotropy of the resolution, and an angle corresponding to the anisotropy of the resolution,
The resolution value, the degree of anisotropy, and the angle include a pattern in which lines having different intervals and widths are sequentially arranged at equal distances in each of at least two directions with respect to each of the at least two points. Corresponding to the size of the ellipse formed by the distribution of points having the same amplitude of the luminance value corresponding to the pattern, the ellipticity, and the angle of the ellipse in the image in the image obtained by photographing the subject,
An image processing apparatus. The image processing apparatus according to claim 1 , wherein the interpolation unit geometrically calculates the angle corresponding to the one point based on a position on the image of the one point. Said angle, the point at which the degree of anisotropy on the image is most isotropically, to claim 1 or claim 2, characterized in that a direction connecting the calculated position of the filter The image processing apparatus described. The filter calculation unit, the image processing apparatus according to any one of claims 1 to 3, characterized in that determining the weighting coefficients corresponding to said angle. The image processing apparatus according to claim 4 , wherein the weighting factor is determined so that a difference in resolution between the direction of the angle and a direction orthogonal to the direction of the angle is minimized. If the filter has a first finite number of elements, and the range corresponding to the number of elements is greater than the ellipse, the filter may be reduced to a second finite number less than the first finite number. the image processing apparatus according to claim 5 is divided into a plurality of filters to be corrected by convolving the image from claim 1, wherein having elements. Resolution information indicating the resolution status corresponding to at least two points on the image is calculated based on the image,
Resolution information corresponding to one point on the image other than the at least two points is calculated based on the resolution information corresponding to each of the at least two points and the position of the at least two points and the one point on the image. ,
Depending on the resolution information, and calculates the filter for correcting a region including the one point of the image,
The resolution information has a resolution value corresponding to the resolution corresponding to each point of the image, an anisotropy indicating the anisotropy of the resolution, and an angle corresponding to the anisotropy of the resolution,
The resolution value, the degree of anisotropy, and the angle include a pattern in which lines having different intervals and widths are sequentially arranged at equal distances in each of at least two directions with respect to each of the at least two points. Corresponding to the size of the ellipse formed by the distribution of points having the same amplitude of the luminance value corresponding to the pattern, the ellipticity, and the angle of the ellipse in the image in the image obtained by photographing the subject,
An image processing method. Resolution information indicating the resolution status corresponding to at least two points on the image is calculated based on the image,
Resolution information corresponding to one point on the image other than the at least two points is calculated based on the resolution information corresponding to each of the at least two points and the position of the at least two points and the one point on the image. ,
Calculating a filter for correcting a region including the one point of the image according to the resolution information;
Let the computer execute the process ,
The resolution information has a resolution value corresponding to the resolution corresponding to each point of the image, an anisotropy indicating the anisotropy of the resolution, and an angle corresponding to the anisotropy of the resolution,
The resolution value, the degree of anisotropy, and the angle include a pattern in which lines having different intervals and widths are sequentially arranged at equal distances in each of at least two directions with respect to each of the at least two points. Corresponding to the size of the ellipse formed by the distribution of points having the same amplitude of the luminance value corresponding to the pattern, the ellipticity, and the angle of the ellipse in the image in the image obtained by photographing the subject,
A program characterized by that .