JP2015109562A - Image processing device, imaging device, control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suitably correct a blur of an image while reducing amplification of noise.SOLUTION: An image processing device applies a recovery process for recovering an influence of aberrations of an imaging optical system to respective pixels of an image including the influence of aberrations of the imaging optical system. Further, the image processing device estimates a noise increment resulting from the application of the recovery process for each of the pixels of the image to which the recovery process is applied based upon frequency characteristics of the image before the recovery process is applied and frequency characteristics of the recovery process. Then noise of the image to which the recovery process is applied is reduced based upon estimated noise increments.

Description

  The present invention relates to an image processing apparatus, an imaging apparatus, a control method, and a program, and more particularly to an image processing technique for improving image quality.

  One of the deteriorations in image quality of an image obtained by imaging with an imaging device such as a digital camera is so-called blur, which is degradation related to sharpness. The blur in the image is caused by spherical aberration, coma aberration, field curvature, astigmatism, and the like of the imaging optical system. Due to such aberrations, for example, asymmetric blur such as coma aberration, color blur due to different blur for each color component, and the like are included in the captured image.

  The aberration is expressed as a point spread function (PSF). Further, by performing Fourier transform on the function, the aberration can be expressed as an optical transfer function (OTF) in the frequency space. OTF can be represented by a complex number, an amplitude component that is an absolute value indicates MTF (Modulation Transfer Function), and a phase component that is a declination indicates PTF (Phase Transfer Function). That is, the blur in the image is caused by the influence exerted by the amplitude component and the phase component of the image by the optical transfer function of the imaging optical system. In addition, since the PSF is different for each color component, a different blur occurs for each color component that causes color bleeding.

  The blur caused by the aberration generated in the image is recovered (corrected or restored) by applying a recovery filter based on the OTF of the imaging optical system to the captured image. However, in the process of such recovery processing, the sharpness of the image is improved by correcting the influence of the aberration, while noise included in the image is amplified. For this reason, a technique for reducing noise amplification in the recovery process is disclosed. Patent Document 1 discloses a method for executing a recovery process in which the gain of a recovery filter applied in the recovery process is weakened and the noise falls within the allowable range when the amount of noise is outside the allowable range. . Further, Patent Document 2 discloses a method of preparing a plurality of recovery filters in advance and adaptively applying a filter with the least noise amplification.

JP 2007-299068 A JP 2011-120309 A

  However, in the methods such as Patent Documents 1 and 2, noise amplification can be suppressed, but on the other hand, the blur recovery effect is also reduced, and thus there is a possibility that the blur is not suitably corrected in the corrected image. there were.

  In particular, when the applied gain is uniformly reduced so that the local or overall noise amount of the image falls within the allowable range as in Patent Document 1, the recovery effect can be obtained even in the region where the blur is suitably corrected by the conventional method. There was a possibility of reduction. Because of the aberration characteristics, the blur intensity differs for each area of the image. Therefore, the gain for canceling (completely recovering) the influence of the OTF of the imaging optical system also differs for each area. In the conventional method, the applied gain is determined according to the setting of the strength of the recovery process, but if the gain that determines the gain required for complete recovery exceeds the gain, the gain required for complete recovery is determined for the image area. Had been applied. That is, since any one of the applied gains determined as the complete recovery gain is applied for each region, the corrected image includes a region that is not a completely recovered region. However, when the applied gain is reduced uniformly as in Patent Document 1, there is a possibility that the recovery effect of the region that should be completely recovered may be reduced. As a result, there is a possibility that suitable correction may not be performed even in a region where the influence of noise can be ignored when it is completely recovered.

  The present invention has been made in view of the above-described problems, and provides an image processing apparatus, an imaging apparatus, a control method, and a program that appropriately correct blur of an image while reducing noise amplification. Objective.

  In order to achieve the above object, an image processing apparatus of the present invention is characterized by having the following configuration. Specifically, the image processing apparatus includes an acquisition unit that acquires an image including the influence of the aberration of the imaging optical system, and a recovery process that recovers the influence of the aberration of the imaging optical system on each pixel of the image acquired by the acquisition unit. For each pixel of the image to which the recovery process is applied, and to estimate the amount of noise increase due to the application of the recovery process based on the frequency characteristics of the image acquired by the acquisition means and the frequency characteristics of the recovery process And a reduction means for reducing noise of the image to which the recovery processing is applied based on the noise increase amount estimated by the estimation means.

  With such a configuration, according to the present invention, it is possible to suitably correct image blur while reducing noise amplification.

1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. The block diagram which showed the internal structure of the image process part 107 which concerns on embodiment of this invention. The flowchart which illustrated the recovery process performed in the digital camera 100 which concerns on embodiment of this invention. The flowchart which illustrated the filter production | generation process performed in the filter production | generation part 201 which concerns on embodiment of this invention The figure for demonstrating the production | generation of the recovery filter which concerns on embodiment of this invention The figure for demonstrating the relationship between the input gain which concerns on embodiment of this invention, and a perfect recovery gain The figure for demonstrating the processing function in the relaxation process which concerns on embodiment of this invention The flowchart which illustrated the coefficient determination process performed in the relaxation coefficient determination part 204 which concerns on embodiment of this invention

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following embodiment, an example in which the present invention is applied to a digital camera capable of correcting the influence of blur generated in an image by an imaging optical system as an example of an image processing apparatus will be described. However, the present invention can be applied to any device capable of correcting the influence of blur generated on an image by the imaging optical system.

<< Configuration of Digital Camera 100 >>
FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.

  The system control unit 101 is, for example, a CPU. The system control unit 101 controls the operation of each block included in the digital camera 100. Specifically, the system control unit 101 controls the operation of each block by reading an operation program for each block stored in a ROM (not shown) and developing the program in, for example, a memory 106 described later.

  The imaging unit 103 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 103 photoelectrically converts the optical image formed on the imaging surface by the imaging optical system 102 and outputs the obtained analog image signal to the A / D conversion unit 104. The imaging optical system 102 is disposed in a lens unit (hereinafter referred to as a lens) that can be attached to and detached from the digital camera 100, and this lens can be attached to another digital camera 100.

  The A / D converter 104 applies A / D conversion processing to the input analog image signal to convert it into digital image data (hereinafter simply referred to as image data), and outputs it. The image data output by the A / D conversion unit 104 is written into the memory 106 or the image display memory 110 via the image processing unit 107 and the memory control unit 105 or directly via the memory control unit 105.

  The memory control unit 105 controls data writing to the memory 106 that is a volatile memory and data reading from the memory 106. The memory 106 may be used as a processing memory for image processing in the image processing unit 107, for example.

  The image processing unit 107 applies various image processing such as demosaic processing (development processing) and color conversion processing to the input image data. In the present embodiment, the image processing unit 107 executes a later-described recovery process that cancels the influence generated on the captured image by the optical transfer function (OTF) of the imaging optical system 102. In addition, the image processing unit 107 of the present embodiment executes a later-described mitigation process for reducing noise on the image to which the recovery process is applied.

<Internal Configuration of Image Processing Unit 107>
Here, an internal configuration related to the recovery processing of the image processing unit 107 of the present embodiment will be described with reference to FIG.

  First, the image data that is the target of the recovery process is input to the recovery processing unit 202 and the feature amount calculation unit 203. The image processing unit 107 also reads OTF data, which is information indicating the OTF of the imaging optical system 102 used for the recovery process, from the memory 106 based on the identification information indicating the type of lens including the imaging optical system 102, for example. It is issued and input.

  The filter generation unit 201 generates a recovery filter that corrects the influence caused by the OTF based on the OTF data. The filter generation unit 201 generates a recovery filter based on information such as a lens type, a zoom position, a diaphragm, a subject distance, and a shooting mode when image data to be filtered is captured.

  The recovery processing unit 202 applies the recovery filter generated by the filter generation unit 201 to the input image data, and executes a recovery process for correcting the influence caused by the OTF of the imaging optical system 102.

  In addition, the image processing unit 107 of the present embodiment uses the feature amount calculation unit 203, the relaxation coefficient determination unit 204, and the relaxation processing unit 205 in order to reduce (relieve) noise amplification that occurs during the recovery process. Execute mitigation processing.

  The feature amount calculation unit 203 calculates the frequency characteristic (frequency distribution) as a feature amount for each local region of the input image data. The feature amount is referred to when estimating the amount of noise increase when the recovery process using the recovery filter is applied.

  The relaxation coefficient determination unit 204 estimates the amount of noise increase after recovery based on the feature amount of each area of the image data calculated by the feature amount calculation unit 203. Then, the relaxation coefficient determination unit 204 determines the relaxation coefficient of each region used in the relaxation processing based on the estimated noise increase amount of each region.

  The relaxation processing unit 205 performs a relaxation process using the relaxation coefficient of each region determined by the relaxation coefficient determination unit 204 on the image data that has been subjected to the recovery process by the recovery processing unit 202. The relaxation processing unit 205 generates image data in which the noise amplified by the recovery process is corrected based on the relaxation coefficient determined for each region.

  The output unit 206 applies an encoding process such as a compression process to the image data that has been subjected to the relaxation process in the relaxation processing unit 205 and outputs the image data. The output unit 206 applies image processing such as color processing and gamma processing to the image data as necessary.

  The recording medium 108 is a recording device such as a built-in memory of the digital camera 100, a memory card, or an HDD. The image data photographed in response to the photographing instruction is recorded on the recording medium 108 after image processing for recording is applied in the image processing unit 107.

  The display unit 109 is a display device that the digital camera 100 has, such as an LCD. The display unit 109 functions as an electronic viewfinder by displaying captured image data captured by the imaging unit 103 and stored in the image display memory 110 by the memory control unit 105. Specifically, the captured image data read from the image display memory 110 is converted into an analog image signal by the D / A conversion unit 111 and input to the display unit 109 to be displayed as a through image (live view). ).

  The operation input unit 112 is a user interface of the digital camera 100 such as a shutter switch or a menu button. When the operation input unit 112 detects that an operation input to the user interface has been performed, the operation input unit 112 outputs a control signal corresponding to the performed operation to the system control unit 101. For example, when the operation input unit 112 detects that the shutter switch is half-pressed, the operation input unit 112 outputs the SW1 signal to the system control unit 101. At this time, the system control unit 101 starts operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) processing in response to the reception of the SW1 signal. Control each block to

《Recovery filter principle》
First, the principle of the recovery filter generated by the filter generation unit 201 of the image processing unit 107 will be described.

As described above, due to the aberration of the imaging optical system 102, the subject image is moved from the position where it should originally be formed in the optical image to be imaged. Since the aberration can be expressed as a point spread function (PSF), the optical image to be originally formed is f (x, y), the actually formed optical image is g (x, y), and the PSF is h ( When the function is expressed as x, y), the relationship between the functions is
Can be expressed as Here, * indicates convolution (convolution integration, product sum), and (x, y) indicates, for example, two-dimensional coordinates on the image sensor.

Since the point spread function (PSF) is expressed as an optical transfer function (OTF) by converting to the frequency domain, the convolution is excluded by converting the equation (1) to the frequency domain by Fourier transform.
Can be expressed as Here, H (u, v) is an optical transfer function obtained by Fourier transform of the point spread function h (x, y). G (u, v) and F (u, v) are obtained by Fourier transform of g (x, y) and f (x, y), respectively. Note that (u, v) indicates frequency components and phase components (coordinates) in the two-dimensional frequency domain (complex plane).

That is, in order to remove the influence of the aberration from the image data obtained by imaging, by dividing both sides of Equation (2) by OTF and performing inverse Fourier transform,
As described above, a recovery filter of R (x, y) may be applied to the image data. That is, by applying a recovery filter to the captured image data, it is possible to obtain image data corresponding to an optical image that should be originally formed. Here, R (x, y) represents the inverse Fourier transform of 1 / H (u, v). That is, since the recovery filter is defined based on OTF, it is possible to correct the influence of the amplitude component and the phase component generated in the image data.

  Note that the recovery filter is generated based on the OTF, and thus exhibits different characteristics for each area of the image data. Thereby, suitable correction can be executed with an appropriate gain for each area of the image data.

  Also, the amount of information in the recovery filter and OTF can be enormous. For example, when it is necessary to refer to the surrounding 300 pixels in the correction of one pixel, a data amount that is several hundred times that of the number of pixels of the image data to be processed is required by simple calculation. The OTF is not limited to the type of lens employed, and changes depending on the shooting conditions such as the aperture and zoom position. Therefore, the OTF data is stored in the storage area for each combination of the type of the imaging optical system 102 and the shooting conditions. It is not realistic to keep it.

  For this reason, in the present embodiment, the OTF data approximates the OTF (optical transfer function) of the imaging optical system 102 by a polynomial, and the coefficients of the polynomial are used for image height and declination (information in the circumferential direction and diagonal direction). The description will be made assuming that the data is included in each. Specifically, the OTF data is obtained by polynomial approximation of an OTF calculated from a design value related to the lens of the imaging optical system 102 for each imaging condition and discretized in a specific frequency range.

  Even if the lens type is the same, if the pixel pitch of the image sensor of the digital camera to be mounted is different, the required OTF frequency band is different. For this reason, in order to be able to share data that is the basis for generating OTF data among a plurality of digital cameras, it is necessary to prepare data that can be used for various pixel pitches. Therefore, the maximum value of the frequency to be extracted is set sufficiently large so that it can be flexibly adapted to various digital cameras. That is, it is assumed that the OTF data can be extracted in a later-described process according to, for example, a Nyquist frequency that is a characteristic of the digital camera 100 at the time of shooting. However, the recording format of the OTF data is not limited to this, and is a format in which the information of the recovery filter and the OTF is converted or generated for the purpose of reducing the amount of information, such as a table in which the OTF is discretely sampled. Good.

《Recovery processing》
A specific process of the recovery process executed by the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. This recovery process will be described as being started when, for example, a shooting process is performed in response to a shooting instruction from the user and image data to which the recovery process is applied is generated.

  In step S301, the system control unit 101 refers to information indicating the application amount of the recovery process, and determines whether or not to apply the recovery process to the image data (target image data) that is the application target. Specifically, the system control unit 101 makes the determination in this step according to whether or not the application amount of the recovery process is zero. If the system control unit 101 determines that the recovery process is to be applied to the target image data, it moves the process to S302. If it determines that the recovery process is not to be applied, it moves the process to S306.

  Note that the application amount (application degree) of the recovery process may be set by the user using a GUI such as a slider. Also, the application amount of the recovery process may be set uniformly for the entire image, or may be set for each region. In the present embodiment, the recovery processing application amount is assumed to indicate a recovery gain (input gain) that is uniformly set for the entire region by the user and used in the recovery processing in the recovery processing unit 202. Here, the “gain” is a value (amplification factor) by which the MTF (amplitude component) of the application target image is multiplied in order to correct the influence of the aberration. That is, the frequency characteristic of the gain corresponds to the frequency filter converted from the recovery filter applied in the recovery process. Note that the frequency characteristic of the gain that completely recovers exactly matches the inverse characteristic of the OTF of the imaging optical system 102.

  In S <b> 302, the memory control unit 105 reads the target image data and the OTF data of the imaging optical system 102 corresponding to the shooting conditions at the time of shooting the target image data from, for example, the memory 106 and inputs them to the image processing unit 107. Further, the memory control unit 105 acquires information indicating the input gain of the recovery process and transmits the information to the image processing unit 107.

  In step S303, the filter generation unit 201 executes filter generation processing for generating a recovery filter used for target image data recovery processing based on the OTF data.

<Filter generation processing>
Here, the filter generation process executed in the filter generation unit 201 will be described in detail using the flowchart of FIG. In this filter generation process, a recovery filter that reduces the influence of aberration is generated for each pixel of the target image data. However, generating a recovery filter having correction information for each of all pixels is not practical from the viewpoint of the amount of data and the amount of calculation. Therefore, in this embodiment, the recovery filter is generated with a configuration having correction information for each sampling point set discretely with respect to the target image data. In this case, for the pixels other than the sampling points whose correction information is not included in the generated recovery filter, the recovery processing is performed based on the correction information generated by interpolating the correction information about the neighboring sampling points. And

  In step S401, the filter generation unit 201 generates an OTF for each of a predetermined number of sampling points set vertically downward from the optical center of the target image data based on the OTF data corresponding to the shooting conditions. In the present embodiment, as shown in FIG. 5A, the number of sampling points is 10, and the sampling points are set so as to equally divide the target image data from the optical center to the maximum image height.

  In step S402, the filter generation unit 201 expands each sampling point by rotating it 90 degrees counterclockwise around the optical center, and ¼ region of the target image data as shown in FIG. 5B. An OTF for a given sampling point is generated. As described above, since the OTF is composed of the MTF and the PTF, it is possible to easily generate an OTF with different PTFs by rotating the OTF for each sampling point generated in S401. It is.

  In the present embodiment, the OTF at the reference sampling point set for the target image data is rotated to be developed into an OTF at a predetermined sampling point. However, the present invention is not limited to this. That is, in this embodiment, for convenience, the reference sampling point is set vertically downward from the optical center of the target image data, but it will be understood that sampling points may be set for other positions. Further, in the present embodiment, the OTF is generated for the reference sampling point in order to reduce the calculation amount, and then the sampling point is rotated and developed. However, when the computing capability is sufficient, the configuration may be such that OTF is generated based on the OTF data for all the two-dimensionally arranged sampling points as shown in FIG.

  In step S <b> 403, the filter generation unit 201 extracts information on a necessary frequency band from the OTF for each sampling point arranged two-dimensionally according to the Nyquist frequency calculated from the pixel pitch of the imaging element included in the imaging unit 103. .

  In step S404, the filter generation unit 201 applies an optical low-pass filter or an aperture deterioration characteristic correction coefficient to the OTF at each sampling point from which a necessary frequency band is extracted, and further considers the characteristics of the digital camera 100. Convert to

  In step S405, the filter generation unit 201 determines an applied gain to be applied to each sampling point from the OTF and the input gain at each sampling point. Specifically, the filter generation unit 201 first determines a complete recovery gain for bringing the image at the point into a complete recovery state for the OTF at each sampling point. Since the MTF of the OTF changes according to the spatial frequency, in this embodiment, the MTF for a frequency that is half the Nyquist frequency calculated from the pixel pitch is completely recovered (MTF is set to 1). Determine as the full recovery gain. Then, the filter generation unit 201 determines an applied gain that is the maximum gain multiplied by the entire OTF at each sampling point from the complete recovery gain and the input gain determined as described above. At this time, the filter generation unit 201 compares the input gain with the complete recovery gain for each sampling point, and determines that the complete recovery gain is the applied gain if the complete recovery gain does not exceed the input gain. That is, since the input gain is equal to or greater than the complete recovery gain, the applied gain is set to the complete recovery gain in order to avoid overcorrection due to the application of the input gain. In addition, when the input gain is less than the complete recovery gain, the filter generation unit 201 determines the input gain as the applied gain. Thus, the applied gain determined for each sampling point in this step differs for each sampling point.

  When 48 sampling points are set for a quarter region of the target image data, the influence of the change in the imaging position due to the aberration increases as the sampling point has a higher image height. When the input gain is set to a relatively low value (for example, 5), for example, as shown in FIG. 6A, only four points set near the optical center are used as the applied gain. Then, the input gain is the applied gain for the sampling points that exist at other positions where the image height is high. When the input gain is set to a relatively high value (for example, 10), for example, as shown in FIG. 6B, the sampling points at which the complete recovery gain becomes the applied gain increases. As can be seen from FIGS. 6A and 6B, the higher the input gain that is set, the wider the distribution of the applied gain determined for each sampling point.

  In step S <b> 406, the filter generation unit 201 generates a recovery filter related to a ¼ region of the target image data, which corresponds to the inverse characteristic of the OTF of the imaging optical system 102. Specifically, the filter generation unit 201 generates a recovery filter at a sampling point by performing inverse Fourier transform on the inverse characteristics of the OTF after application of the applied gain and the OTF before application.

  In step S <b> 407, the filter generation unit 201 duplicates the recovery filter for the sampling point set for the ¼ region of the target image data to the sampling point at the corresponding position in the symmetrical relationship, and recovers corresponding to the entire target image data. Generate a filter. Specifically, as shown in FIGS. 5C and 5D, the filter generation unit 201 reverses the positional relationship in at least one of the horizontal direction and the vertical direction for the recovery filter of the ¼ area. Then, a recovery filter relating to the entire target image data is generated.

  As described above, this filter generation process can generate a two-dimensional recovery filter of real number data from a two-dimensional distribution (sampling points) of OTF which is complex number data including phase information.

  After the recovery filter is generated, the recovery processing unit 202 applies the recovery filter to each pixel of the target image data in S304 of the recovery process, and generates recovered image data subjected to the recovery process. The recovery filter is a recovery two-dimensional data array having a width and a height for one sampling point. When the recovery filter is applied to the target pixel corresponding to the sampling point, the pixel of the area including the target pixel and corresponding to the two-dimensional data array is multiplied by the filter coefficient and added. The pixel value of the target pixel is obtained. For pixels that do not correspond to sampling points, recovery processing is performed by applying recovery two-dimensional data generated by interpolation from nearby sampling points.

  In S <b> 305, the relaxation coefficient determination unit 204 determines a relaxation coefficient to be used for the relaxation process from the recovery filter generated by the filter generation unit 201 and the frequency characteristics (features) of each region of the target image data. Execute.

The below-described relaxation processing performed in S306 by the relaxation processing unit 205 of the present embodiment removes random noise having an amplitude equal to or smaller than a threshold in each local region of the recovered image data. In the relaxation process, the relationship between the restored image data X (n) (n is a region or a pixel position) to be relaxed and the corrected image Y (n) after the relaxation process is
Represented as: Here, for example, K is a constant of K = 1, and M and N are numerical values for specifying a region or a pixel. Further, F is a non-linear function indicating the characteristic that the absolute value is suppressed to ε or less as shown in FIG. In the coefficient determination process in this step, ε, which is the maximum absolute value of the function F, is determined as a relaxation coefficient.

  Note that the frequency characteristic information of each region of the target image data used for the coefficient determination process is generated by the feature amount calculation unit 203 after the target image data is input to the image processing unit 107. Specifically, the feature value calculation unit 203 applies, for example, Fourier transform to the image of each region corresponding to the sampling point related to the recovery filter and converts it to the frequency space, thereby obtaining the frequency characteristic information of the region. Generate. In the present embodiment, the region for determining the relaxation coefficient is described as being provided corresponding to the sampling point of the recovery filter, but the embodiment of the present invention is not limited to this. That is, the sampling points of the recovery filter and the area for determining the relaxation coefficient are not the same number, and the positions may not be associated with each other.

<Coefficient determination process>
Here, details of the coefficient determination process executed by the relaxation coefficient determination unit 204 of the present embodiment will be described in detail with reference to the flowchart of FIG.

  In step S <b> 801, the relaxation coefficient determination unit 204 selects a region (target region) for which the relaxation coefficient has not been determined from the regions set by the feature amount calculation unit 203 for the target image data.

  In S802, the relaxation coefficient determination unit 204 performs Fourier transform on the recovery filter at the sampling point corresponding to the target region, and calculates the frequency characteristic of the recovery filter. The frequency characteristic of the recovery filter indicates the signal amplification factor for each frequency.

  In step S803, the relaxation coefficient determination unit 204 obtains a noise amplification factor for each frequency from the frequency characteristics of the recovery filter in the target region according to the graph shown in FIG.

  In S804, the relaxation coefficient determination unit 204 performs Fourier transform on the image data of the target region before applying the recovery filter, and calculates the frequency characteristics of the image data in the target region. The relaxation coefficient determination unit 204 calculates a noise increase amount by integrating the result obtained by multiplying the frequency characteristic of the image data by the noise amplification factor. At this time, the noise increase amount is calculated by integrating the result obtained by multiplying the frequency characteristics of the image data by the noise amplification factor in a specific frequency band in which the influence of noise is conspicuous, not in all frequency bands. You may do it. Alternatively, the amount of increase in noise may be calculated by focusing on only the frequency at which the influence of noise is most noticeable. The relaxation coefficient determination unit 204 determines a relaxation coefficient related to the target region used for the relaxation processing based on the amount of noise increase in the target region. The noise increase amount statistically shows a value between 1.0 and 3.0, and in this embodiment, the noise increase amount multiplied by 10 is used as a relaxation coefficient.

  In step S <b> 805, the relaxation coefficient determination unit 204 determines whether there is a region in the target image data where the relaxation coefficient has not been determined. The relaxation coefficient determination unit 204 returns the process to S801 if it is determined that there is a region where the relaxation coefficient has not been determined, and completes this coefficient determination process if it determines that there is no region.

  After the relaxation coefficient related to each area of the target image data is determined by the coefficient determination process, the relaxation processing unit 205 executes the relaxation process for each area of the recovered image data based on the relaxation coefficient in S306. If it is determined in S301 that the restoration process is not applied to the target image data, the mitigation processing unit 205 executes the mitigation process of the target image data based on the mitigation coefficient of the specified value in this step.

  Note that the mitigation process performed in S306 is not limited to the above-described method. For example, the relaxation processing unit 205 sets a peripheral area of a predetermined size (11 × 11 pixels) around the target pixel of the recovered image data, and the signal level with the target pixel is selected from the pixels included in the peripheral area. Pixels whose difference is within the threshold are extracted. Then, the relaxation processing unit 205 obtains an average value of the signal levels of the extracted pixel and the target pixel, and uses this average value as a signal of the target pixel after the relaxation process. By executing this for each pixel in each region, a corrected image after relaxation processing can be obtained. Then, the threshold for comparing the level difference may be set to ε, and this ε may be changed as the relaxation coefficient by the above method.

  In step S307, the output unit 206 performs color processing, gamma processing, and encoding processing on the recovered image data to which the relaxation processing is applied, and outputs the image data converted into the recording format. Then, the system control unit 101 transmits the output image data to the recording medium 108 for recording.

  As described above, in the recovery process of the present embodiment, the amount of increase in noise generated in the image due to the application of the recovery process is estimated based on the frequency characteristic of the gain applied to each pixel in the process. Then, noise reduction processing corresponding to the estimated increase in noise can be performed for each pixel of the image after the recovery processing, so that it is possible to obtain an image with reduced noise while maintaining the effect of the recovery processing. it can.

  Although the present embodiment has been described on the assumption that noise is reduced by the function relaxation processing as shown in FIG. 7, the processing related to noise reduction is not limited to this. The process related to noise mitigation may be performed for each pixel or region so as to reduce the noise based on a value obtained by estimating the amount of increase in noise of the corresponding part in the image after the restoration process.

  As described above, the image processing apparatus according to the present embodiment can appropriately correct blurring of an image while reducing noise amplification. Specifically, the image processing apparatus applies a recovery process for recovering the influence of the aberration of the imaging optical system to each pixel of the image including the influence of the aberration of the imaging optical system. Further, the image processing apparatus estimates, for each pixel of the image to which the recovery process is applied, an increase in noise due to the application of the recovery process based on the frequency characteristics of the image before the recovery process is applied and the frequency characteristics of the recovery process. Then, the noise of the image to which the recovery process is applied is reduced based on the estimated noise increase amount.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (8)

An acquisition means for acquiring an image including the influence of aberration of the imaging optical system;
Recovery means for applying a recovery process for recovering the influence of the aberration of the imaging optical system to each pixel of the image acquired by the acquisition means;
For each pixel of the image to which the recovery process has been applied, an estimation unit that estimates the amount of noise increase due to the application of the recovery process based on the frequency characteristic of the image acquired by the acquisition unit and the frequency characteristic of the recovery process;
An image processing apparatus comprising: a reduction unit configured to reduce noise of an image to which the recovery process is applied based on the noise increase amount estimated by the estimation unit. The estimation means includes a frequency distribution of the amplitude component of the area set for the image acquired by the acquisition means, and a frequency distribution of the amplification factor of the amplitude component in the recovery process for the pixel corresponding to the set area. The image processing apparatus according to claim 1, wherein the noise increase amount is estimated on the basis of the image quality.   The image processing apparatus according to claim 1, wherein an amplification factor of the amplitude component in the recovery process is determined based on an optical transfer function of the imaging optical system.   The image processing apparatus according to claim 3, further comprising a setting unit that sets a maximum value of an amplification factor of the amplitude component in the recovery process. An imaging optical system;
An image pickup unit that photoelectrically converts an optical image formed by the image pickup optical system and picks up an image including an influence of aberration of the image pickup optical system;
Recovery means for applying a recovery process for recovering the influence of the aberration of the imaging optical system to each pixel of the image imaged by the imaging means;
For each pixel of the image to which the recovery process has been applied, an estimation unit that estimates the amount of noise increase due to the application of the recovery process based on the frequency characteristic of the image captured by the imaging unit and the frequency characteristic of the recovery process;
An image pickup apparatus comprising: a reduction unit configured to reduce noise of an image to which the recovery process is applied based on the noise increase amount estimated by the estimation unit. An acquisition step of the image processing apparatus for acquiring an image including the influence of the aberration of the imaging optical system;
A recovery step in which the recovery means of the image processing apparatus applies a recovery process to recover the influence of the aberration of the imaging optical system to each pixel of the image acquired in the acquisition step;
The estimation means of the image processing apparatus, for each pixel of the image to which the recovery process is applied, noise due to the application of the recovery process based on the frequency characteristic of the image acquired in the acquisition step and the frequency characteristic of the recovery process An estimation process for estimating the increase amount;
The image processing apparatus includes: a reduction step in which the reduction unit of the image processing apparatus includes a reduction step of reducing noise of the image to which the recovery processing is applied based on the noise increase amount estimated in the estimation step. Control method of the device. A method for controlling an imaging apparatus having an imaging optical system,
An imaging step in which an imaging unit of the imaging apparatus photoelectrically converts an optical image formed by the imaging optical system and captures an image including an influence of aberration of the imaging optical system;
A recovery step in which the recovery means of the imaging apparatus applies a recovery process that recovers the influence of the aberration of the imaging optical system to each pixel of the image captured in the imaging step;
The estimation unit of the imaging apparatus increases noise due to the application of the recovery process based on the frequency characteristics of the image captured in the imaging process and the frequency characteristics of the recovery process for each pixel of the image to which the recovery process is applied. An estimation step for estimating the quantity;
An image pickup apparatus comprising: a reduction step for reducing noise of an image to which the restoration process is applied, based on the noise increase amount estimated in the estimation step. Control method.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1 thru | or 4.