JP5479187B2 - Image processing apparatus and imaging apparatus using the same - Google Patents

Description

  The present invention relates to image processing, and more particularly to an image restoration (restoration) process.

  Along with the digitization of information, various image processing apparatuses and methods for photographed images have been proposed by handling images as signal values. An image obtained by imaging a subject with a digital camera has deteriorated in image quality due to the aberration of the imaging optical system as compared with an image from which the influence of the aberration is removed. Examples of the deterioration include blurring of an image and coloring caused by a difference in blurring method for each wavelength. Image blur is caused by spherical aberration, coma, field curvature, astigmatism, etc. of the imaging optical system. Colored is axial chromatic aberration, color spherical aberration, color coma, magnification of the imaging optical system. This is due to chromatic aberration. Optically, this blur is also called a point spread function (PSF).

  As a method for correcting such blurring of an image, there is known a method of correcting using information on an optical transfer function (OTF) of an imaging optical system. This method is called “image restoration” and “image restoration”. Hereinafter, processing for correcting image degradation using information on the optical transfer function (OTF) of the imaging optical system will be referred to as image restoration processing.

  As a filter used for image restoration processing, a Wiener filter that controls the degree of restoration according to the intensity ratio (SNR) of an image signal and a noise signal is known.

  In Patent Document 1, the parameter α is used for the image restoration filter in order to adjust the degree of restoration. Since the filter is changed from an inactive filter (α = 0) to an inverse filter (α = 1) according to the adjustment parameter α, the image is displayed in a range from the original photographed image to the maximum recovered image. The degree of recovery can be adjusted.

  Although not image restoration processing, Patent Document 2 discloses that as a method of correcting only chromatic aberration of magnification, blurring and coloring of an image are separated without using an image restoration filter, and geometric conversion processing and pixel interpolation are performed on the coordinates of pixel signal values. A method of correcting using processing is disclosed.

JP 2007-183842 A Japanese Patent Laid-Open No. 06-113309

  However, in the conventional technology, since the mutual action of the image restoration processing and the magnification chromatic aberration correction is not taken into consideration, coloring may occur even though the image processing is performed.

Accordingly, an object of the present invention is to provide an image processing apparatus capable of obtaining a higher quality image with reduced coloring.
The false color is a color generated by image processing.

In order to solve the above problems, the present invention provides:
Image restoration processing means for restoring an input image having a plurality of color components;
Difference information acquisition means for acquiring difference information between the input image and the recovered image subjected to the recovery process;
Adjustment coefficient setting means for setting an adjustment coefficient for adjusting the degree of recovery in the recovery process;
Recovery adjustment image generation means for generating a recovery adjustment image by combining the difference information according to the adjustment coefficient with respect to the input image;
Magnification chromatic aberration correction means for adjusting a correction amount of chromatic aberration of magnification with respect to the input image or the restoration adjustment image according to the adjustment coefficient.

  The effect of the present invention is that a high-quality output image can be obtained by adjusting the magnification chromatic aberration correction amount according to the degree of recovery.

Schematic configuration diagram of an image processing device Illustration of the flow of image processing Illustration of image recovery filter Illustration of image processing using adjustment factors Schematic diagram of change in recovery degree Explanatory diagram of linearity of recovery degree adjustment Overview of processing flow for adjustment coefficient setting Schematic configuration diagram of imaging device Explanatory drawing of image restoration filter selection and correction Explanatory drawing of the processing flow of Example 1 Explanatory diagram of the image restoration process Explanatory diagram regarding the amount of chromatic aberration correction Explanatory drawing of image processing related to lateral chromatic aberration correction Illustration of image processing related to white balance correction Schematic configuration diagram of an information processing apparatus for executing image processing Illustration of the image processing system Explanation of correction information

  Embodiments of the present invention will be described below with reference to the drawings.

  First, prior to description of specific embodiments, image processing techniques used in the embodiments will be described.

  An example of a functional block diagram of the image processing apparatus of the present invention is shown in FIG. The image processing apparatus of the present invention includes an image restoration processing unit 11, a difference information acquisition unit 12, an adjustment coefficient setting unit 13, a magnification chromatic aberration correction processing unit 14, and a restoration adjustment image generation processing unit 15.

Next, each means will be described with reference to FIG. FIG. 2 shows a flow of image processing, and symbols g, f, fd, S, and Sd represent images in each process. The supplementary symbol m of each symbol represents the color component of the image. For example, if the image has R, G, B (red, blue, yellow) color components, _Am is (A _R , A _G , A _B ), and (A R component, A G component, B component of A).

First, the image restoration processing means 11 performs an image restoration process on the input image g _m having RGB color components by using an image restoration filter for each color component to obtain a primary restored image fd _m .

Then difference data acquisition means 12, an original input image g _m from the primary restored image fd _m by subtracting a signal value for each corresponding pixel as shown in Equation 1, for each color component restoration component information S _m (difference information) is generated.

Next, synthesized in accordance with the color combination ratio adjustment factor ω that can represent the recovery component information S _m as a matrix having coefficients elements 3 × 3, and generates a color combining restoration component information Sd _m. The color composition adjustment coefficient ω is set by the adjustment coefficient setting means 13. 2 represents the color composition ratio adjustment coefficient of the color component B for generating the color composition recovery component information Sd _A for the color component A when ω _AB . As an example, generation of the R component color synthesis recovery component information Sd _R will be described. The color synthesis recovery component information Sd _R for R includes the color synthesis ratio adjustment coefficients ω _RR , ω _RG , ω and the signal values of the respective R, G, B recovery component information S _R , S _G , S _B. It is generated by multiplying each _RB and combining between the color components. Combining is to generate a single image by adding signal values for corresponding pixels of a plurality of images. Then, the same process is performed for the other color components, thereby generating color synthesis recovery component information. Therefore, generation of the color combining restoration component information Sd _m can be expressed as Equation 2. Σ relating to n in Equation 2 represents taking the sum of R, G, and B.

  That is, the difference information is acquired by combining the difference amount for each color component between the input image and the restored image according to the color composition ratio adjustment coefficient indicating the mixing ratio of the color components.

Then, the recovery adjust the image generation processing unit 15, it synthesizes the input image g _m color combining restoration component information Sd _m and the original of each color component for each color component to obtain a recovery adjust image f _m. This process can be expressed as Equation 3.

  This is the basic flow of the image processing of the present invention. However, when the image restoration filter is an image restoration filter that corrects lateral chromatic aberration, the image restoration processing is performed without considering the value of the color composition ratio adjustment coefficient ω. Otherwise, false colors may occur. Therefore, the magnification chromatic aberration correction processing means 14 corrects the component (pixel value) of the image restoration filter in accordance with the color composition ratio adjustment coefficient ω set by the adjustment coefficient setting means 13, and the image is obtained using the corrected image restoration filter. Perform recovery processing.

  When the color composition ratio adjustment coefficient ω is set in advance, the magnification chromatic aberration correction processing unit 14 corrects the image restoration filter according to the set value. When the color composition ratio adjustment coefficient ω is set or changed after the image restoration process is performed, a feedback unit may be provided to correct the image restoration filter. Thus, by correcting the magnification chromatic aberration correction component of the image restoration filter according to the color composition ratio adjustment coefficient ω (adjustment coefficient), the magnification chromatic aberration correction according to the degree of restoration is performed, so that the color shift is reduced. A good quality image can be obtained.

  Next, each process and data in FIG. 2 will be described in more detail.

(Input image g _m )
The input image g _m is a digital image obtained by photoelectric conversion by the imaging device via the imaging optical system, and is deteriorated by an optical transfer function (OTF) due to the aberration of the imaging optical system including the lens and various optical filters. doing. The imaging optical system can also use a mirror (reflection surface) having a curvature in addition to the lens.

  The input image has a plurality of color components, such as RGB, and may be a mosaic image having a signal value of one color component for each pixel, or the mosaic image is subjected to color interpolation processing (demosaicing processing). A demosaic image having signal values of a plurality of color components in each pixel may be used. This mosaic image can also be called a RAW image as an image before various image processing such as signal value conversion such as color interpolation processing (demosaicing processing) and gamma conversion, and image compression such as JPEG.

  In order to obtain an input image having a plurality of color components with a single-plate image sensor, color filters having different spectral transmittances are arranged in each pixel, and each pixel has a signal value of one color component as described above. Get a mosaic image. In this case, an image having a plurality of color component signal values in each pixel can be generated by performing the above-described color interpolation processing.

  In order to obtain an input image using a multi-plate, for example, three-plate image sensor for a single plate, a color filter having a different spectral transmittance is arranged for each image sensor, and a different color component for each image sensor What is necessary is just to acquire an image signal value. In this case, since each image sensor has a signal value of each color component for the corresponding pixel, each pixel has a signal value of a plurality of color components without performing color interpolation processing in particular. An image can be generated.

The focal length of the lens in the input image g _m, the diaphragm can be attached to various types of correction information for correcting the imaging conditions and the image such as shooting distance. When a series of processing from imaging to output is performed by a single closed imaging device, the image can be acquired in the device without adding imaging condition information and correction information to the image. However, when a RAW image is acquired from an imaging device and correction processing or development processing is performed by another image processing device, it is preferable to add shooting condition information and correction information to the image as described above. However, if the correction information is stored in advance in the image processing apparatus and the system in which the adjustment coefficient can be selected from the imaging condition information is configured, it is not always necessary to add the correction information to the image. The correction information here is, in particular, the color composition ratio adjustment coefficient ω and the magnification chromatic aberration correction amount associated therewith. This correction information will be described later.

(Image recovery processing)
Next, the recovery process (image recovery process) enclosed by the square in FIG. 2 will be described. In order to describe the image restoration filter used for the image restoration processing, a schematic diagram of the image restoration filter is shown in FIG. The image restoration filter can determine the number of taps according to the aberration characteristics of the imaging system and the required restoration accuracy. In FIG. 3A, the image restoration filter is an 11 × 11 tap two-dimensional filter as an example. Each tap of the filter corresponds to each pixel of the image, and this image restoration filter is subjected to convolution processing (convolution integration, product sum) on the pixels of the input image in the image restoration processing step. In order to improve the signal value of a pixel, the convolution process is to match the pixel with the center of the image restoration filter, and multiply the image signal value by the filter coefficient value for each corresponding pixel of the image and the image restoration filter. Is to take. Then, it is generally known as a process for replacing the sum as a signal value of the central pixel.

  In FIG. 3A, values in each tap are omitted, but one cross section of this image restoration filter is shown in FIG. The distribution of the values (coefficient values) of each tap of the image restoration filter plays a role of ideally returning the signal value spatially spread by the aberration to the original one point. The image restoration filter can be obtained by calculating or measuring an optical transfer function (OTF) of the imaging system and performing inverse Fourier transform on a function based on the inverse function.

  Since it is necessary to consider the influence of noise in the image restoration process, an image restoration filter suitable for the restoration process can be used by selecting a Wiener filter or various related image restoration filter creation methods.

  Note that the image restoration filter can be generated in consideration of not only the OTF of the imaging optical system but also the factors that degrade the OTF during the imaging process. For example, there is an optical low-pass filter having birefringence that degrades the OTF during the imaging process, and suppresses high-frequency components with respect to the frequency characteristics of the OTF. Further, the shape and aperture ratio of the pixel aperture of the image sensor also affect the frequency characteristics, and the spectral characteristics of the light source and the spectral characteristics of various wavelength filters can be cited as factors that degrade the OTF. It is desirable to create an image restoration filter based on an optical transfer function (OTF) in a broad sense that considers these factors.

  In the case of an apparatus that does not have an imaging optical system such as a lens, such as a scanner (reading apparatus) or an X-ray imaging apparatus that performs imaging by bringing an imaging element into close contact with a subject surface, the OTF corresponds to a transfer function of the imaging system. Even in an apparatus that does not have an imaging optical system, the output image is deteriorated due to image sampling and the like, and this deterioration characteristic corresponds to the imaging system transfer function. Therefore, a recovery image can be generated by generating an image recovery filter based on the transfer function of the system without having an imaging optical system. Hereinafter, an imaging optical system or a system that does not have an imaging optical system but can acquire an OTF is referred to as an imaging system.

  Further, when performing image restoration processing on an image having a plurality of color components, it is possible to obtain a higher quality image by preparing an image restoration filter for each of the plurality of color components. For example, when the input image is an RGB color image, three image restoration filters corresponding to the R, G, and B color components may be created. By doing so, chromatic aberration due to the imaging system can be corrected satisfactorily.

  As shown in FIG. 3A, the number of vertical and horizontal taps of the image restoration filter need not be the same in the vertical and horizontal directions, and can be arbitrarily changed. Specifically, by using a two-dimensional filter that divides the image restoration filter into 100 or more, for aberrations that spread greatly from the imaging position such as spherical aberration, coma aberration, axial chromatic aberration, and off-axis color flare due to the imaging system. However, it can recover more accurately. Also, asymmetrical aberrations such as coma, off-axis color flare, sagittal flare, field curvature, and astigmatism can be recovered with high accuracy.

  In addition, if the degradation process of the image to be subjected to image restoration processing is linear, the reverse process for restoring the original image before degradation can be processed with high accuracy, so that the input image is subjected to various adaptive nonlinear processes. Preferably not. Therefore, it is preferable to perform the image restoration process on the mosaic image (RAW image). However, even in the demosaic image, if the deterioration process by the color interpolation process is linear, it is possible to perform a higher-accuracy recovery process by taking this deterioration function into consideration when generating the image recovery filter. Further, when the required accuracy of recovery is low, or when only images that have been subjected to various image processing can be obtained, recovery processing may be performed on the demosaic image.

(Primary recovery image fd _m )
Next, the primary recovery image fd _m (recovered image) generated by the image recovery process will be described. In many of the conventional techniques, an image obtained by performing various image processes on the primary recovery image is used as an output image. If the primary recovery image satisfies the desired quality can be used as an output image f _m. However, if there is a difference between the optical transfer function (OTF) in the actual photographing state and the optical transfer function (OTF) assumed when the image restoration filter is created, the restored image has different desired characteristics. It will be a thing. Such differences in the optical transfer function (OTF) depend on manufacturing errors of the imaging device, focus shift due to mechanism accuracy, focus shift due to three-dimensional object shooting, spectral fluctuation of the illumination light source, sensor luminance saturation, processing accuracy of the imaging device, and the like. Occur. Furthermore, since the degree of recovery differs for each color component, a false color is generated in the image. Even if the aberration is greatly recovered and the sharpness of the image is improved, if there is such a false color, the image quality will be low. It can also be said that the false color becomes noticeable because the sharpness is improved. In order to suppress the occurrence of this false color, the following processing is performed.

(Recovery component information S _m )
The original input image g _m from the primary restored image fd _m as shown in Equation 1, to generate a recovery component information S _m for each color component _(difference information) by performing the subtraction process for each pixel for each color component Can do. This recovery component information S _m includes negative effects generated by the aberration component and the image restoration processing of an input image g _m. For example, a noise increase component, a ringing component, and a false color component are also included. The component here can also be said to be information that a pixel has.

(Color composition ratio adjustment coefficient ω, color composition recovery component information Sd _m , recovery adjustment image f _m )
The color combination ratio adjustment factor omega, in order to generate a color combining restoration component information Sd _m of one color component A, a factor that determines the combination ratio of the at least two color components including the color components A. Thus, the process of generating a color combining restoration component information Sd _m from the recovery component information S _m can be expressed as Equation 4 that describes expand Equation 2 and the same color component m, the n.

  The method for determining, setting, or selecting the nine color composition ratio adjustment coefficients ω in Equation 4 will be described with two specific examples.

First, a description will be given color combination ratio adjustment factor to obtain the same image as the primary restored image fd _m as a recovery adjustment image omega. Put the remaining ingredients and 0 diagonal component of the color combination ratio adjustment coefficient of formula 4 omega as 1 (unit matrix), the color combining restoration component information Sd _m equal to recover component information S _m of their own color component Become. In this case, the recovery adjustment image is intended to correct the aberration component to the maximum, but at the same time, there is a tendency that a possibility of generating a false color increases. However, by adjusting the color composition ratio adjustment coefficient, the recovery adjustment image can be used as the input image, so that the stability of the system can be increased.

The second is a color composition ratio adjustment coefficient ω for preventing false colors from being generated. Place 1/3 all the elements of the color composition ratio adjustment coefficient of formula 4 omega, color combining restoration component information Sd _m becomes recovery component information S _m for all the color components to those averaged, color synthesis recovery component The information Sd _R , Sd _G and Sd _{B all have the} same value. The fact that the color synthesis recovery component information Sd _m is equal means that there is no difference in additional information regarding the color component when the color synthesis recovery component information Sd _m is synthesized with the input image g _m in the subsequent process. False color does not occur. However, since the aberration information of each color component are averaged, so that the recovery degree in comparison with the case where the output image first primary restored image fd _m above, i.e. the sharpness decreases. However, even if the recovery component information _Sm is averaged, the recovery component information S _R , S _G , and S _B of each color component has a certain positive correlation (similarity), so that the input image g _m Thus, the sharpness of the recovery adjustment image is improved. Therefore, the condition that all the elements of the color composition ratio adjustment coefficient ω are 1/3 is the recovery condition that eliminates the risk of false color generation.

  The setting of the color composition ratio adjustment coefficient ω when the false color generation risk is maximized and minimized is described above. By making it possible to set the color composition ratio adjustment coefficient ω continuously, it is possible to adjust the balance between the false color occurrence risk and the degree of recovery. This makes it possible to obtain an image having a desired image quality while balancing the risk of false color occurrence and the degree of recovery (the degree of sharpness recovery).

  As shown in Expression 4, the color composition ratio adjustment coefficient ω has nine setting degrees of freedom, and thus setting of each element value may be complicated. For example, this is a case where the general user sets nine color composition ratio adjustment coefficients ω using the operation means.

  Therefore, in order to reduce this complexity, a method for reducing the parameter for controlling the adjustment coefficient by providing a dependency relationship between the elements of the color composition ratio adjustment coefficient ω will be described.

  As an example of a method for determining the color composition ratio adjustment coefficient ω, two constraint conditions are first added. The first condition is that the sum for each row of the matrix ω of Equation 4 is set to 1 as shown in Equation 5.

This means that, for example, the mixture ratio of the recovery component information S _R , S _G , and S _B for generating the R component color synthesis recovery component information Sd _R is normalized. Thus the mixture ratio to normalize, or are weighted in any respectively different color combining restoration component information Sd _m ratio is to be easily compared.

  The second condition is that the sum for each column of the matrix ω is 1 as shown in Equation 6.

This means that when generating each color synthesis recovery component information Sd _R , Sd _G , Sd _B , the recovery component information S _R , S _G , S _B is distributed to each color component and used up.

  When the above two constraints are applied, the color composition ratio adjustment coefficient ω can be expressed as Equation 7.

Furthermore, in order to suppress the risk of a false color while ensuring recovery degree, the color synthesizing restoration component information Sd _m has a high similarity between the color components, i.e. towards differences is small is preferred. Therefore, it means that the recovery component information S _m of a certain color component can be as much as possible evenly distributed in the color combining restoration component information Sd _m for each color component, each row it is false color risk small variance of the formula 7 Can be reduced. Based on this, when the variance of each column in Equation 7 is minimized, the color composition ratio adjustment coefficient ω can be expressed as Equation 8.

  Since the set parameter can be set by the single set value ω in Expression 8, the adjustment of the balance between the recovery degree and the risk of occurrence of false color can be easily controlled. If ω = 1 in Equation 8, the matrix ω is a unit matrix, and both the degree of recovery and the risk of false color occurrence are maximized. If ω = 1/3, the matrix ω is 1/3 for all elements, and there is no risk of false color generation. Therefore, if the color composition ratio adjustment coefficient ω is reduced in the range of 1/3 ≦ ω ≦ 1, the risk of false color generation can be reduced.

  As another effect, the balance can be adjusted with one parameter, so that usability can be further improved.

  As another effect, it is possible to control adjustment parameters suitable for the side providing the image pickup apparatus and the image processing apparatus with a small degree of freedom, thereby improving work efficiency in the apparatus development process and the production process.

Although an example of the method for determining the color composition ratio adjustment coefficient ω has been described so far, the determination method is not limited to this. For example, if all elements of the matrix ω to 0 (zero), the color combining restoration component information Sd _m is 0 (zero) in all color components, the recovery adjust image f _m is the input image g _m itself. In this way, the output image is not limited to 1/3 ≦ ω ≦ 1, but by adjusting the color composition ratio adjustment coefficient ω in the range of 0 ≦ ω ≦ 1, the output image can be displayed in the range from the input image to the image with the maximum recovery degree. A recovery adjustment image can be obtained by adjustment. Further, the correction can be further emphasized by setting the right side of Expression 5 to be larger than 1.

  Further, the setting freedom of each element of the matrix ω is not limited to one, and can be adjusted with another setting freedom based on another constraint condition. For example, according to Equation 7, the degree of freedom for setting the adjustment coefficient is 6.

(Recovery strength adjustment coefficient μ)
The color composition ratio adjustment coefficient described above is a parameter that can adjust the mixing ratio of each color component. On the other hand, the recovery strength adjustment coefficient described below is a parameter that can adjust the recovery degree.

Using the method of setting the color composition ratio adjustment coefficient equation 8 omega, further as the recovery adjust image f _m, it is described how the input image g _m and emphasis correction image obtained.

How to obtain a recovery adjust image f _m with recovery strength adjustment factor μ in ω determined matrix by Equation 8 is expressed by Equation 9.

If μ = 0, the second term on the right side of Equation 9 is 0 (zero), so that the input image g _m itself is obtained as the recovery adjustment image f _m . Also, if mu = 1, the formula 9 is equal to Equation 3 to obtain a recovery adjust image f _m recovery degree and false color risk using the color combination ratio adjustment factor ω determined by Equation 8 is adjusted Can do. The basic range of the recovery strength adjustment coefficient μ is 0 ≦ μ ≦ 1, but by making μ> 1, it is possible to obtain an image with enhanced correction.

Furthermore, if the recovery intensity adjustment coefficient μ is changed for each color component, the degree of recovery can be adjusted for each color component. This is because the degree of recovery for each color component when the optical transfer function (OTF) changes for each color component due to factors such as spectral fluctuations of the light source that illuminates the subject and manufacturing errors in the imaging system, and the balance of chromatic aberration changes. It is effective to adjust the strength of For example, when the spectral characteristic (intensity ratio for each wavelength) of the illumination light source changes, the amount of aberration changes for each color component. In these cases, the recovery intensity adjustment factor μ by setting for each color component, it is possible to obtain a recovery adjust image f _m which is suitable for each color component in accordance with the spectral characteristic at the time of shooting. Expression 10 is an expression for changing the recovery intensity adjustment coefficient μ for each color component.

  In addition, when there is a manufacturing error in another imaging system using the recovery strength adjustment coefficient μ, for example, there is a thing with a different degree of deterioration at a symmetrical position of the image, and the input image is blurred or its relative color difference. May appear. As for blurring, the recovery strength adjustment coefficient μ is set in accordance with the fluctuation of the blur amount depending on the position of the image, so that the deterioration of the image due to the manufacturing error can be reduced. With respect to coloring, a manufacturing error can be absorbed by setting the recovery intensity adjustment coefficient μ for each color component according to the variation of the coloring amount depending on the position of the image. That is, it is possible to obtain a more preferable image quality by adjusting the recovery intensity adjustment coefficient μ according to the feature amount of the pixel.

  In this way, by using the recovery strength adjustment coefficient μ, it is possible to adjust the degree of recovery while facilitating the determination of the color composition ratio adjustment coefficient ω as in Expression 8. FIG. 4 shows the flow of processing when the recovery strength adjustment coefficient μ is introduced. By separating the color composition ratio adjustment coefficient ω and the recovery intensity adjustment coefficient μ, the color composition ratio adjustment coefficient ω is used to adjust the balance between the degree of recovery and the risk of occurrence of false color, and the degree of recovery in the color composition ratio adjustment coefficient ω is adjusted. Adjustment can be made with the recovery strength adjustment factor μ.

  A schematic diagram of the change in the degree of recovery by this adjustment is shown in FIG. The risk of false color generation can be adjusted by changing the color composition ratio adjustment coefficient ω, and the degree of recovery is adjusted between the set color composition ratio adjustment coefficient ω and the input image by changing the recovery intensity adjustment coefficient μ. be able to.

Will now be described linearity of degree of recovery Recovery adjusted image f _m with respect to the color composition ratio adjustment factor ω and changes in the recovery intensity adjustment factor μ shown in FIG. And color synthesis ratio adjustment factor ω and recovery strength adjustment factor μ together 1 and the maximum degree of recovery Recovery adjusting image f _m equal 1 reference primary restored image fd _m. Then, the similarity between the primary recovery image fd _m and the recovery adjustment image f _m is defined as the evaluation function J _m of Equation 11.

The double vertical line with the symbol 2 on the right side indicates a two-dimensional norm, and the denominators X and Y indicate the number of pixels in the horizontal and vertical directions of the image. When Expression 9 is substituted as the restoration adjustment image f _{m and} Expression 8 is substituted as the color composition ratio adjustment coefficient ω, the evaluation function J _m becomes a linear expression with respect to the color composition ratio adjustment coefficient ω and the recovery intensity adjustment coefficient μ. It can be seen that the linear recovery degree can be adjusted.

FIG. 6B shows the experimental results of evaluating the linearity of the recovery degree adjustment using the test image of FIG. From this result, it can be seen that the linearity shown in FIG. 5 is correctly reproduced.
Similar to the reduction in the number of adjustment parameters, such recovery degree adjustment linearity has an advantage that it is easy to make correspondence between a set value and a response (recovered image) when the user variably adjusts.

  Although it has been described above that the risk of false color occurrence is reduced using the color composition ratio adjustment coefficient ω, adjusting the degree of recovery using the recovery intensity adjustment coefficient μ can suppress noise and ringing. The evaluation of image quality as an output image varies depending on the purpose. For example, in the case of portraits, noise and ringing are very disturbing. On the other hand, when you want to read numbers from a car license plate with a surveillance camera, it is most important to identify the numbers even if there is noise or ringing. In addition, when an adverse effect such as noise, ringing, or false color appears for some reason, it is important to guarantee at least a captured image (input image) as an output image. In these cases, it is possible to cope with a high degree of freedom by adjusting the recovery strength adjustment coefficient μ. Also in general photography, the image quality required as an output image varies depending on the user and the subject, from a soft image with flare due to the remaining aberration to a sharp image from which aberration has been removed. This case can also be dealt with by adjusting the recovery strength adjustment coefficient μ.

  Other effects of the present invention will be described with reference to the processing flow of FIG. In both cases of adjusting the false color occurrence risk and adjusting the restoration degree, it is not necessary to recalculate the image restoration filter because only the image composition ratio is changed. Further, it is not necessary to perform convolution processing on the input image every time the adjustment parameter is changed. The captured image is used as an input image, and an image restoration filter is generated using a restoration parameter as an initial value, or image restoration processing is performed on the input image using an image restoration filter prepared in advance as an initial value. Image restoration processing is performed using an adjustment parameter prepared in advance for the restored image, an adjustment parameter set by the user, or an adjustment parameter automatically determined from the image information to obtain a restored image. The restored image is evaluated, and it is determined whether the restored adjustment image (output image) is used as it is or whether the restoration degree is changed again. When changing the degree of recovery, the adjustment parameter is changed and the image composition process is performed again. These adjustment parameters are the color composition ratio adjustment coefficient ω and the recovery intensity adjustment coefficient μ described above. As described above, since the degree of recovery can be controlled in a later step, it is not necessary to change the filter for each pixel in the initial recovery process for extracting the recovery component information. In other words, even if the color composition ratio adjustment coefficient is changed to adjust the degree of recovery and the risk of false color (detrimental) occurrence, there is no need to recalculate the image recovery filter, reducing the load on image processing and speeding up processing. It can be performed.

  Thus, when performing image processing more in terms of the necessity of recalculation of the image restoration filter at the time of adjustment and the necessity of convolution processing of the input image and the image restoration filter that are image restoration processing, It became possible to reduce processing load.

  Here, the adjustment of the adjustment parameter accompanying the determination of adoption as the output image or the change of the recovery degree may be made based on subjective evaluation by the user, or an image evaluation function is set in advance. And you can do it automatically.

  Further, the adjustment coefficient may be set according to the feature amount of the pixel of the input image. The feature amount of a pixel is a signal value of the pixel, a change amount of the pixel value, or the like. Changing the adjustment coefficient according to the feature amount of these pixels means changing the degree of recovery depending on the position of the image. In the present invention, the final degree of recovery is performed by synthesizing the image in units of pixels in the image synthesizing process, and can be easily adjusted by simply changing the mixing ratio at that time. That is, unlike the conventional method, there is no need to regenerate the recovery filter for each pixel, or it is not necessary to store the recovery filter for each pixel value in advance. be able to.

  Other setting values for the adjustment coefficient include setting information relating to changes in ISO sensitivity and SN ratio, zoom (focal length), shooting distance (focusing distance), aperture value, and settings corresponding to the above manufacturing errors. There are values.

  The basic processing of the image processing according to the present invention has been described. However, in the actual processing, the steps described here can be collectively processed when several steps can be processed at the same time. It is also possible to add necessary processing steps as appropriate before and after each step. Furthermore, the formulas and equal signs used in the description are not intended to limit the specific algorithm of the image processing method of the present invention, and can be modified as necessary within a range where the object can be achieved.

  Hereinafter, specific examples of the present invention will be described.

  FIG. 8 shows an example of a basic configuration when the image processing method of the present invention is applied to an imaging apparatus. A subject image (not shown) is formed on the image sensor 102 by the imaging optical system 101. The imaging light is converted into an electrical signal by the image sensor 102, converted into a digital signal by the A / D converter 103, and input to the image processing unit 104. The image processing unit 104 performs image restoration processing together with predetermined processing. First, information on the imaging state of the imaging apparatus is obtained from the state detection unit 107. The state detection unit 107 may obtain state information directly from the system controller 110. For example, imaging state information regarding the imaging system may be obtained from the imaging optical system control unit. Next, an image restoration filter corresponding to the imaging state is selected from the storage unit 108, and an image input to the image processing unit 104 is adjusted for image restoration processing and a degree of restoration corresponding to the adjustment parameter. The adjustment parameters here are a color composition ratio adjustment coefficient ω, a recovery intensity adjustment coefficient μ, and the like. An image restoration filter selected from the storage unit 108 according to the imaging state may be used as it is, or an image restoration filter prepared in advance may be corrected and processed into a filter more suitable for the imaging state. it can.

  Then, the output image processed by the image processing unit 104 is stored in the image recording medium 109 in a predetermined format. This output image is adjusted to a restoration degree that takes into account the pixel value, the amount of change in the pixel value, and the like by an image restoration process using the image processing method of the present invention, and is further sharpened with a balance between the degree of false color occurrence and the degree of restoration. This is a converted image. The image processing unit 104 includes at least a calculation unit and a temporary storage unit (buffer). The image is written (stored) and read out from the primary storage unit as necessary for each step of the image processing. For example, in the restored image generation step, it is necessary to temporarily store the input image acquired first in order to synthesize the input image and the color synthesis restoration component information. Further, the storage unit for temporarily storing is not limited to the temporary storage unit (buffer), but may be the storage unit 108, which is suitable for the data capacity and communication speed of the storage unit having a storage function. It can be appropriately selected and used.

  Further, the display unit 105 may display an image obtained by performing a predetermined display process on the image after the recovery process, or no image recovery process is performed for high-speed display, or a simple recovery process You may display the image which performed.

  A series of control is performed by the system controller 110, and mechanical driving of the imaging optical system is performed by the imaging optical system control unit 106 according to an instruction from the system controller 110. The aperture of the aperture 101a is controlled as an F number shooting state setting. The focus lens 101b is controlled in its position by an auto focus (AF) mechanism (not shown) or a manual manual focus mechanism (not shown) in order to adjust the focus according to the subject distance. This imaging optical system may include an optical element such as a low-pass filter or an infrared cut filter. However, when an element that affects the characteristics of an optical transfer function (OTF) such as a low-pass filter is used, an image restoration filter is used. It may be necessary to consider at the time of creation. Even in the infrared cut filter, since it affects each PSF of the RGB channel that is an integral value of the point spread function (PSF) of the spectral wavelength, especially the PSF of the R channel, it is necessary to consider at the time of creating the image restoration filter. There is a case.

  Further, although the imaging optical system 101 is configured as a part of the imaging apparatus, it may be an interchangeable type as in a single-lens reflex camera. Functions such as aperture diameter control and manual focus may not be used depending on the purpose of the imaging apparatus.

  FIG. 9 is an explanatory diagram regarding selection and correction of the image restoration filter. FIG. 9 is a schematic diagram of the image restoration filter group stored in the storage unit 108. The stored filters are discretely arranged in the imaging state space with the three states of zoom position, aperture diameter, and subject distance as axes. The coordinates of each point (black circle) in the imaging state space are the state positions of the image restoration filter stored in advance. In FIG. 9, for the sake of explanation, the position of the filter is arranged on a grid point orthogonal to each state, but the position of each filter may be deviated from the grid point. Further, in order to illustrate the number of types of imaging states, a three-dimensional diagram for three states is shown. However, a four-dimensional or more imaging state space for four or more states may be used.

  A specific method for selecting the image restoration filter will be described. Assume that a state indicated by a large white circle in FIG. 9 is an actual imaging state detected. When a filter exists in the actual imaging state position or in the vicinity thereof, the filter can be selected and used for image restoration processing. One method is to calculate the distance in the imaging state space between the actual imaging state and the stored imaging state, and select the one with the shortest distance. In FIG. 9, a filter at a position indicated by a small white circle is selected. As another method, filter selection can be weighted by a direction in the imaging state space. That is, it is a method of selecting the product of the distance in the imaging state space and the direction weight as the evaluation function.

  Next, a method for correcting and generating the image restoration filter will be described. In correcting the filter, the distance in the imaging state space between the actual imaging state and the stored imaging state is calculated, and the one with the shortest distance is selected. At this time, since the state difference amount becomes the smallest, the subsequent correction amount can also be reduced, and a filter close to the original filter in the imaging state can be generated. In FIG. 9, a filter at a position indicated by a small white circle is selected. State difference amounts ΔA, ΔB, ΔC between the state of the selected image restoration filter and the actual imaging state are calculated. By calculating a state correction coefficient based on this state difference amount and correcting the selected image restoration filter, an image restoration filter corresponding to the actual imaging state can be generated. As another method, an image restoration filter suitable for the imaging state can be generated by selecting a plurality of image restoration filters in the vicinity of the actual imaging state and performing interpolation processing according to the state difference amount. In this interpolation process, the coefficient values of the corresponding taps between the two-dimensional filters may be interpolated using linear interpolation, polynomial interpolation, spline interpolation, or the like.

  The optical transfer function (OTF) used to generate the image restoration filter is an optical transfer function (OTF) in which light incident on the imaging optical system transmits an optical element to form an image, and is an optical design tool or optical analysis. It can be obtained by calculation using a tool. Furthermore, the optical transfer function (OTF) can be actually measured and obtained in the state of the imaging optical system alone or the imaging apparatus.

  Further, since the optical transfer function (OTF) changes according to the image height (image position) of the imaging optical system even in one photographing state, the recovery processing of the present invention is performed by dividing the image according to the image height. It is desirable to change every time. The image restoration filter may be scanned on the image while performing convolution processing, and the filter may be sequentially changed for each predetermined region.

  The storage unit 108 may store image correction information including a color synthesis ratio adjustment coefficient and a restoration strength adjustment coefficient in addition to the image restoration filter.

  The image pickup apparatus according to the present embodiment has a basic configuration in which the image processing method described in the introduction of the description of the embodiment is performed by the image processing unit 104. FIG. 10 shows a specific flow related to the image restoration processing performed by the image processing unit 104. The mark ● in the figure indicates pixel data such as an image, and the others indicate processing. The image processing unit 104 acquires an input image in the acquisition process. The input image here is a mosaic image having one color component information for each pixel. Next, the imaging state information is obtained from the state detection unit 107, the image restoration filter corresponding to the imaging state is selected from the storage unit 108, and the input image is restored using the image restoration filter in the image restoration process. A primary recovery image is generated.

  In the recovery component information generation step, recovery component information is generated from the difference between the signal values of the pixels of the input image and the primary recovery image. Here, after this, color interpolation processing (demosaicing processing) of the input image and the recovery component information is performed in order to perform image synthesis between the color components. This color interpolation processing can be performed as normal color interpolation processing, and the subsequent processing can be performed on the color-interpolated image. Here, for example, color interpolation is performed by a simple method different from the normal color interpolation processing. You can also. Then, setting values of the adjustment parameters of the color composition ratio adjustment coefficient ω and the recovery intensity adjustment coefficient μ are acquired, and the adjustment parameters are applied to the recovery component information for each color component in the color synthesis recovery component information generation step. Color synthesis recovery component information is generated. Only one of the color synthesis ratio adjustment coefficient ω and the recovery intensity adjustment coefficient μ as the adjustment parameters can be used as necessary. In this embodiment, the user does not set adjustment parameters automatically, but sets them automatically. As this setting method, it is possible to automatically select from preset values prepared in advance according to the imaging conditions and image height. Further, it is possible to determine the feature amount of the pixel from the image and automatically change and set the adjustment parameter.

  Here, the correction amount of the lateral chromatic aberration may be set according to the set adjustment coefficient, or the correction amount of the lateral chromatic aberration may be set according to the adjustment coefficient stored in advance.

  Next, in the recovered image generation step, the color composite recovery component information is combined with the input image to generate a secondary recovered image. In this restored image generation step, it can be performed on the color-interpolated image, or the color synthesis recovery component information is masked for each color component and returned to the color component information array at the time of the mosaic image, It can also be combined with the mosaic input image.

  As other necessary image processing, color interpolation processing (demosaicing processing), shading correction, distortion aberration correction, and the like are performed to obtain an output image. Further, the lateral chromatic aberration correction using geometric transformation may be performed, and the correction amount is changed according to the adjustment coefficient. Various image processes including the other processes described here can be inserted before, after, or in the middle of the above flow as necessary.

  FIG. 11 is a schematic diagram showing how an image changes due to image processing when the color composition ratio adjustment coefficient ω = 1/2 and the recovery intensity adjustment coefficient μ = 1. In each square frame in the figure, the horizontal direction is the position of the image, and the vertical direction is the pixel value, representing a cross section of the image.

So far, the color components have been described as three components of R, G, and B, but here, the description will be made simply with two components. The condition of ω = 1/2 is a condition that the false color occurrence risk becomes 0 (zero) in the case of two color components. g _m schematically shows a cross section of a deteriorated input image. A thick solid line indicates the first color component, and a dotted line indicates the second color component.

Processing of FIG. 11 (a) is a case of using the image restoration filter including the magnification chromatic aberration correction components, it performs the magnification chromatic aberration correction with respect to further recover adjusted image f _m later. The process (b) is a case where an image restoration filter that does not include a magnification chromatic aberration correction component is used, and magnification chromatic aberration correction (correction by magnification chromatic aberration correction means) is performed on the input image g _{m in the} previous stage. . In other words, the process of FIG. 11B does not include the function of correcting the chromatic aberration of magnification in the image restoration filter, but the magnification chromatic aberration correction g _m | shift process by the geometric transformation process of coordinates is performed before the image restoration process process. Is going on. Process (c) is a case of using the image restoration filter does not include the chromatic aberration of magnification correction component further performs the magnification chromatic aberration correction with respect to the recovery adjust image f _m later.

g _m | shift is an image obtained by performing magnification color correction by geometric transformation of coordinates for (b). fd _m is an image after the image restoration processing is performed on each of g _m and g _m | shift.

Primary restored image (restored image) fd _m in FIG. 11, the process (a), (b), is corrected blur component of the image with (c), it has been corrected magnification chromatic aberration simultaneously in (a). S _m is, (a) and for the (c) subtracting the _{g m} from fd _m (difference information acquisition), _g m from fd _m for (b) | in recovery component information obtained by subtracting the shift is there. Sd _m is color synthesis recovery component information obtained by averaging the recovery component information S _m of the first color component and the second color component. f _m is a recovery adjustment image (secondary recovery image) obtained by combining the color synthesis recovery component information Sd _m and the input image g _m . f _m | shift performs lateral chromatic aberration correction by geometric transformation of coordinates on the recovery adjustment image f _m . O _m is an output image that has undergone these image processes.

  The correction amount of the magnification chromatic aberration correction component of the image restoration filter in FIG. 11A is changed in accordance with the color synthesis ratio adjustment coefficient or the restoration intensity adjustment coefficient. A method for setting the amount of chromatic aberration of magnification according to the adjustment coefficient (color composition ratio adjustment coefficient, recovery intensity adjustment coefficient) will be described.

  When the adjustment coefficient (color composition ratio adjustment coefficient, recovery intensity adjustment coefficient) is set large, the degree of recovery is high and the lateral chromatic aberration is corrected. Therefore, the lateral chromatic aberration correction amount is reduced. Further, when the adjustment coefficient (color composition ratio adjustment coefficient, recovery strength adjustment coefficient) is set small, the magnification chromatic aberration correction amount is increased because the degree of recovery is low and the magnification chromatic aberration is not corrected compared to the case where the degree of recovery is high. In other words, the magnification chromatic aberration correcting means corrects magnification chromatic aberration of a color component having a smaller adjustment coefficient than that of a color component having a larger adjustment coefficient.

  More specifically, in FIG. 6, when the color composition ratio adjustment coefficient ω is 1/3 (when the composition ratio of each color component is the same), the correction of the chromatic aberration of magnification is performed regardless of the recovery intensity adjustment coefficient μ. Therefore, the magnification chromatic aberration correction amount is maximized. The maximum here is the amount of lateral chromatic aberration of the image before recovery. Here, the maximum value of the color synthesis ratio adjustment coefficient and the recovery intensity adjustment coefficient is set to 1 as specific values.

  That is, when the color composition ratio adjustment coefficient ω is 1 and the recovery intensity adjustment coefficient μ is 1, the magnification chromatic aberration correction amount may be zero because the magnification chromatic aberration is corrected. Further, when the recovery intensity adjustment coefficient μ is set to zero, the magnification chromatic aberration correction amount is maximized because the magnification chromatic aberration is not corrected regardless of the color composition ratio adjustment coefficient ω. That is, when the value of the color composition ratio adjustment coefficient is smaller than 1 (when the color composition ratio adjustment coefficient is close to 1/3), the magnification chromatic aberration correction amount is set to be large. When the color composition ratio adjustment coefficient is the maximum (1), the magnification chromatic aberration correction amount is set to zero.

  The magnification chromatic aberration correction amount based on the relationship between the color composition ratio adjustment coefficient ω and the recovery strength adjustment number μ can be calculated from ω and μ by holding it as a function, or stored and referenced as table data. It is also possible to decide by doing.

  As described above, by changing the magnification chromatic aberration correction amount according to the adjustment coefficient, it is possible to obtain a high-quality image in which coloring is further reduced. More specifically, when the second adjustment coefficient is larger than the first adjustment coefficient, the magnification chromatic aberration correction amount for the pixel for which the first adjustment coefficient is set is calculated as the magnification chromatic aberration correction for the pixel for which the second adjustment coefficient is set. It is characterized by being smaller than the amount. In the above, the preferable form at the time of giving a magnification chromatic aberration correction to the image restoration filter was demonstrated.

Next, the image restoration filter does not include the magnification chromatic aberration correction component, and the magnification chromatic aberration correction can be performed in a step before the image restoration processing. FIG. 11B is used as an explanatory diagram of the third embodiment. FIG. 11 (a), the compared with (c), FIG. 11 (b) sharpest output image O may perform magnification chromatic aberration correction in the preceding step than to produce a color combining restoration component information Sd _m as _It can be seen that _m is obtained. That is, each order in the step of generating a color combining restoration component information Sd _m, if the magnification chromatic aberration recovery component information S _m is left, color synthesis recovery component information Sd _m will have a spread, including chromatic aberration of magnification The color component correction accuracy decreases. Therefore, it is preferable to perform a combination of lateral chromatic aberration correction by geometric conversion and image restoration processing that does not have a lateral chromatic aberration correction function. More preferably, since the degree of recovery of the recovered image changes in accordance with the adjustment coefficient, it is preferable to adjust the magnification chromatic aberration correction amount by geometric conversion in accordance with the adjustment coefficient.

  That is, after correcting the chromatic aberration of magnification by geometric transformation for the input image, the image recovery process is performed to obtain a recovered image. Then, difference information between the restored image and the input image is acquired, and the difference information is calculated on the restored image according to an adjustment coefficient to generate a first restored adjusted image. The second recovery adjustment image is obtained by changing the correction amount of the chromatic aberration of magnification by the geometric transformation again according to the adjustment coefficient and performing the correction of the chromatic aberration of magnification by the geometric transformation on the generated first recovery adjustment image. (Output image) is obtained. Here, the chromatic aberration correction function for magnification is not added to the image restoration process.

  As described above, even when magnification chromatic aberration correction is performed by geometric transformation, the sharpness of the first recovery adjustment image changes according to the setting value of the adjustment coefficient. Therefore, the magnification color correction according to the adjustment coefficient when generating the second recovery adjustment image is particularly effective when fine adjustment of the magnification color correction amount is desired.

  The degree to which the correction amount of the magnification color is adjusted in accordance with the degree of change in sharpness (adjustment coefficient magnitude) depends on the magnification color correction amount by geometric transformation performed before generating the first restoration adjustment image. Depends on adjustment factor correspondence. In this embodiment, since the magnification color correction is performed before the first restoration adjustment image is generated, the magnification chromatic aberration correction is optimum when the sharpness is minimum (the adjustment coefficient is minimum). That is, as the adjustment coefficient increases (becomes sharper), lateral chromatic aberration occurs, and the amount of aberration increases. Therefore, when the first adjustment coefficient is set to a value larger than the second adjustment coefficient, when the second recovery adjustment image is generated, the magnification color correction amount corresponding to the first adjustment coefficient is set to the first adjustment coefficient. The amount of chromatic aberration of magnification corresponding to the adjustment coefficient of 2 is set larger.

  Since the lateral chromatic aberration can be corrected by coordinate transformation processing of coordinates, blur correction is performed by image restoration processing as shown in FIG. On the other hand, it is preferable that correction of lateral chromatic aberration (colored) is removed or reduced by a lateral chromatic aberration correction step by coordinate transformation based on a correction amount according to the adjustment coefficient, and blur is adjusted as an image restoration process by an image restoration filter. .

  As described above, by changing the correction amount of the chromatic aberration of magnification according to the recovery degree of the image recovery process, that is, the adjustment coefficient, it is possible to correct the image so that coloring is not conspicuous.

  (A) and (c) of FIG. 12 are schematic views in two cases where the amount of blur is different for the edge profile of the image. Each of the thick solid line, the thin solid line, and the dotted line indicates different color components of the image. In (b) and (d), the color components represented by thick solid lines are fixed with respect to (a) and (c), respectively, and the remaining two color components are translated (geometrically converted) to perform appropriate correction. The magnification chromatic aberration is corrected by the amount. At this time, the correction amounts A and B are different. Therefore, it is necessary to make the correction amount of the lateral chromatic aberration correspond to the degree of blur, that is, the degree of recovery. By using the image processing method of the first or second embodiment, it is possible to correct the chromatic aberration of magnification that changes in accordance with the degree of image restoration with an optimal correction amount, so that a high-quality output image can be obtained.

  FIG. 15 shows a schematic configuration diagram for causing the information processing apparatus to execute image processing. Reference numeral 151 denotes an image processing unit, 152 denotes a RAM, 153 ROM, 154 denotes a CPU, and 155 denotes a network interface card (NIC) connected to a network. Reference numeral 156 denotes a display unit such as a monitor, 157 denotes a storage device such as a hard disk drive or a memory card, and 158 denotes an I / F to which other devices can be connected. Reference numeral 159 denotes an operation unit such as a keyboard and a mouse, and reference numeral 150 denotes a system bus that interconnects the above-described elements. The CPU 154 loads a program (image processing program) stored in the ROM 153 or the storage device 157 into the RAM 152 which is a work memory, and executes the program. By controlling each element via the system bus 150 in accordance with the program, the function of the image processing program is realized. FIG. 15 shows a general configuration of an information processing apparatus that performs image processing, and even if a part of the configuration is missing or another device is added, it is within the scope of the present invention.

  FIG. 16A shows the configuration of an image processing system that is Embodiment 3 of the present invention. The image processing apparatus 201 is configured by computer equipment, and is loaded with image processing software (image processing program) 206 for causing the computer equipment to execute the image processing method described in the first and second embodiments.

  The imaging device 202 includes a camera, a microscope, an endoscope, a scanner, and the like. The storage medium 203 stores an image (captured image data) generated by imaging such as a semiconductor memory, a hard disk, or a server on a network.

  The image processing device 201 acquires captured image data from the imaging device (imaging device) 202 or the storage medium 203, and outputs output image (recovery adjustment image) data obtained by performing predetermined image processing, the output device 205, the imaging device 202, and the storage. Output to at least one of the media 203. The output destination may be a storage unit built in the image processing apparatus 201, and the output image data may be stored in the storage unit. Examples of the output device 205 include a printer. A display device 204 that is a monitor is connected to the image processing apparatus 201, and the user can perform an image processing operation through the display device 204 and can evaluate the recovery adjustment image. The image processing software 206 has a development function and other image processing functions as necessary in addition to an image restoration processing function (a magnification chromatic aberration correction function according to an adjustment coefficient) and a restoration degree adjustment function.

  FIG. 16B shows the configuration of another image processing system. When the image processing according to the first to third embodiments is performed by the imaging device (imaging device) 202 alone as in the first embodiment, the recovery adjustment image can be directly output from the imaging device 202 to the output device 205.

  In addition, by mounting the image processing apparatus that executes the image processing methods of the first and second embodiments on the output device 205, the output device 205 can perform image restoration processing or restoration degree adjustment. Furthermore, a higher quality image can be provided by adjusting the degree of recovery in consideration of the image deterioration characteristics at the time of output of the output device 205.

  Here, the content of correction data and the flow of correction data for performing image processing including correction processing including a magnification chromatic aberration correction function according to the adjustment coefficient will be described. FIG. 17 shows the contents of the correction data. The correction information set has the following information regarding correction.

Correction control information The correction control information includes setting information related to which of the imaging device 202, the image processing device 201, and the output device 205 performs the recovery processing and the recovery degree adjustment processing, and data transmitted to other devices along with the setting information. Selection information. For example, when only the restoration process is performed by the imaging apparatus 202 and the restoration degree is adjusted by the image processing apparatus 201, it is not necessary to transmit an image restoration filter, but at least a photographed image and a primary restoration image (or restoration component information) are included. It is necessary to transmit.

Imaging Device Information Imaging device information is identification information of the imaging device 202 corresponding to the product name. If the lens and the camera body are interchangeable, the identification information includes the combination.

Imaging state information The imaging state information is information relating to the state of the imaging device at the time of shooting. For example, focal length, aperture value, shooting distance, ISO sensitivity, white balance, and the like.

Imaging Device Individual Information Imaging device individual information is identification information of individual imaging devices with respect to the above imaging device information. Since the optical transfer function (OTF) of the imaging apparatus has individual variations due to variations in manufacturing errors, the individual imaging apparatus information is effective information for setting an optimum recovery degree adjustment parameter individually. In addition, a correction value for correcting to an optimum recovery degree adjustment parameter is included.

Image Recovery Filter Group The image recovery filter group is a set of image recovery filters used in image recovery processing. When a device that performs image restoration processing does not have an image restoration filter, it is necessary to transmit the image restoration filter from another device.

・ Recovery component information When image recovery processing has already been performed and recovery component information has been generated, if the captured image and the recovery component information are transmitted to another device, the recovery degree adjustment processing can be performed by another device. .

Adjustment Parameter Group The adjustment parameter group is a color composition ratio adjustment coefficient ω, a recovery intensity adjustment coefficient μ, or a set of magnification chromatic aberration correction amounts associated with these adjustment coefficients. As described above, the color composition ratio adjustment coefficient ω and the recovery intensity adjustment coefficient μ can be changed according to the position of the image. It can also be changed according to the shooting state. The adjustment parameter group data may be table data of the adjustment coefficient itself or a function for determining the adjustment coefficient.
A magnification chromatic aberration correction amount determined according to the adjustment coefficient can also be held as one of the adjustment parameters. Further, the magnification chromatic aberration correction amount can be automatically determined by table data or a function according to the adjustment coefficient, or can be variably set by the user. Further, the user can select and set from a plurality of preset values prepared in advance.

User setting information The user setting information is an adjustment parameter for adjusting the degree of recovery according to the user's preference or a correction function of the adjustment parameter. Although the user can variably set the adjustment parameter, using the user setting information can always obtain a desired output image as an initial value. Further, it is preferable that the user setting information is updated by the learning function with the sharpness most preferred from the history of the user determining the adjustment parameter. Furthermore, a provider (manufacturer) of the imaging apparatus can provide preset values corresponding to some sharpness patterns via a network.

  The correction information set is preferably attached to individual image data. By attaching the necessary correction information to the image data, the recovery degree adjustment process can be performed by any device equipped with the image processing apparatus of the present invention. Further, the contents of the correction information set can be selected automatically and manually as necessary. For example, when the restoration degree adjustment process is performed by another device, the image restoration filter group is basically unnecessary if the restoration component information is included in the correction information set.

  An example of processing when a correction information set is used is shown. First, the image processing apparatus 201 acquires an image from the imaging apparatus 202 or the storage medium 203 and acquires correction information for performing image processing. Here, the correction information may be stored in advance in the storage means or the like, or the adjustment coefficient set by the adjustment coefficient setting means by the user input or selection, or the adjustment coefficient attached to the image is used. May be. When data relating to magnification chromatic aberration correction is set in the correction information, the image processing apparatus reads data relating to the magnification chromatic aberration correction and adjusts the amount of magnification chromatic aberration correction according to the data.

  As mentioned above, although the Example regarding the system using the image processing of this invention was shown, a various deformation | transformation and change are possible within the range of the summary.

(Modification)
FIGS. 13A and 13B are flowcharts including the magnification chromatic aberration correction process. In FIG. 13A, the magnification chromatic aberration correction process is first performed on the input image, and the image restoration filter that does not have the magnification color correction component described above is used in the image restoration process. Since the magnification chromatic aberration correction process is performed in a process prior to the demosaicing process, a provisional color interpolation process may be performed as necessary in the magnification chromatic aberration correction process.

  On the other hand, in FIG. 13B, first, in the image restoration process, the above-described image restoration filter having no magnification color correction component is used, and the magnification chromatic aberration correction process is performed on the restoration component information. However, in this case, since it is necessary to perform the magnification chromatic aberration correction process on the input image in order to generate the secondary recovery image, the method of FIG. 13A is more preferable.

  Next, the relationship between white balance and recovery degree adjustment will be described with reference to FIGS. White balance is used to adjust the color tone of a digital camera to produce an appropriate color or a color intended by a photographer. Further, the mixing ratio of the RGB signals of the RAW image for generating white is used as the white balance coefficient.

  As described above, in order to generate the color synthesis recovery component information, the recovery component information is color-synthesized between the color components according to the color synthesis ratio adjustment coefficient ω. At this time, if the white balance is not taken into consideration, the intended mixing ratio cannot be achieved by setting ω. For example, when ω = 1/3, the recovery components of R, G, and B are evenly averaged, so that the obtained color synthesis recovery component information is in a state without color. However, unless the white balance coefficient is 1: 1: 1, the color composition recovery component information mixed by 1/3 has a color. That is, the risk of false color generation is not set to 0 (zero).

  Therefore, as shown in FIG. 14A, white balance processing is performed on the input image. This white balance process is a process of normalizing the level of the signal value by performing a process corresponding to the division of the signal value of each color component by the white balance coefficient. By normalizing, the color composition recovery component information can be generated with a mixing ratio corresponding to the color composition ratio adjustment coefficient ω. Then, the white balance of the image is returned to the original state by performing reverse white balance processing before generating the secondary recovery image. This inverse white balance process is a process of performing a process corresponding to multiplication of the signal value of each color component with a white balance coefficient to restore the level of the normalized signal value. As another method, in FIG. 14B, the white balance is not considered in the image as in FIG. 14A, but the adjustment parameter of the recovery degree is corrected according to the white balance coefficient. The adjustment parameters are a color composition ratio adjustment coefficient ω and a recovery intensity adjustment coefficient μ.

  As described above, the preferred context of each processing step and the processing to be considered have been described. However, when there is a restriction from another viewpoint on the order of the processing steps, the present invention is not limited to this. It may be determined according to the required image quality. In addition, as long as the object of the present invention can be achieved, the order of the processing steps may naturally be changed.

  As mentioned above, although the Example regarding the imaging device using the image processing method of this invention was shown, various deformation | transformation and change are possible for the imaging device of this invention within the range of the summary.

  However, even if the actual image captured using the above-described image restoration technique, for example, if the degree of restoration is different from that assumed for each of R, G, B color components, an unexpected false color occurs, There are cases where a high-quality recovered image cannot be obtained. Here, the false color is a color that does not appear in an ideal recovered image.

101 Image restoration processing means 102 Difference information acquisition means 103 Adjustment coefficient setting means 104 Magnification chromatic aberration correction means 105 Recovery adjustment image generation means

Claims (6)

Image restoration processing means for restoring an input image having a plurality of color components;
Difference information acquisition means for acquiring difference information between the input image and the recovered image subjected to the recovery process;
Adjustment coefficient setting means for setting an adjustment coefficient for adjusting the degree of recovery in the recovery process;
Recovery adjustment image generation means for generating a recovery adjustment image by combining the difference information according to the adjustment coefficient with respect to the input image;
An image processing apparatus, comprising: a magnification chromatic aberration correction unit that adjusts a correction amount of magnification chromatic aberration with respect to the input image or the recovery adjustment image according to the adjustment coefficient. The difference information is obtained by synthesizing a difference amount for each color component of the input image and the restored image according to a color synthesis ratio adjustment coefficient indicating a mixing ratio of the color components,
The image processing apparatus according to claim 1, wherein the adjustment coefficient is set for each of the color components.   The image processing apparatus according to claim 1, wherein the magnification chromatic aberration correction unit is the image restoration processing unit. An image sensor that photoelectrically converts a subject image formed by the imaging optical system to generate a captured image; and
An imaging apparatus comprising: the image processing apparatus according to claim 1, wherein the captured image is processed as an input image. An image recovery process for recovering an input image having a plurality of color components;
A difference information acquisition step for acquiring difference information between the input image and the recovered image that has undergone recovery processing;
An adjustment coefficient setting step for setting an adjustment coefficient for adjusting the degree of recovery in the recovery process;
A recovery adjustment image generation step of generating a recovery adjustment image by combining the difference information according to the adjustment coefficient with respect to the input image;
An image processing program for causing an information processing apparatus to execute a magnification chromatic aberration correction step of adjusting a correction amount of magnification chromatic aberration with respect to the input image or the recovery adjustment image according to the adjustment coefficient. Recovering an input image having a plurality of color components;
Obtaining difference information between the input image and the recovered image after the recovery process;
Setting an adjustment coefficient for adjusting the degree of recovery in the recovery process;
Generating a recovery adjustment image by combining the difference information according to the adjustment coefficient with respect to the input image;
An image processing method comprising adjusting a correction amount of chromatic aberration of magnification with respect to the input image or the recovery adjustment image according to the adjustment coefficient.

Images