JP2018088587A - Image processing method and image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method and an image processing apparatus capable of correcting optical deterioration of a captured image with high accuracy.SOLUTION: An image processing method includes an input data acquisition step of acquiring a captured image having multiple color components, and the optical characteristics of an imaging optical system used for shooting of the captured image, a selection step of selecting the reference color component from the color component, by using at least one of the optical characteristics or the edge information of each color component of the captured image, a reference acquisition step of acquiring a reference image based on the reference color component, and a correction step of correction optical deterioration in the color component, so as to reduce difference of the width of edge in the color component and the width of edge in the reference image, by using the reference image and the optical characteristics.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing method and an image processing apparatus.

  When the light emitted from one point of the subject passes through the image pickup optical system, it cannot converge to one point due to the influence of diffraction and aberration of the image pickup optical system, and has a minute spread. A distribution having such a minute spread is called a point spread function (PSF). Since the captured image generated by imaging using the imaging optical system is an image in which PSF is convolved with an ideal subject image, optical degradation (blur) occurs in the captured image.

  Non-Patent Document 1 discloses a technique for correcting optical deterioration of a captured image by image processing. In the image processing method disclosed in Non-Patent Document 1, image correction processing is performed using PSF as an optimization problem for obtaining a solution that minimizes the loss function L and the regularization function R in the following equation (1).

  The loss function L and the regularization function R are represented by the following formulas (2) and (3), respectively.

In Expressions (2) and (3), i is a column vector whose component is each pixel of an image without optical degradation, j is a column vector whose component is each pixel of the captured image, and B is a convolution of PSF. It is a matrix. c is a component of the image (image component). In a general color image, c is three components of RGB (Red, Green, Blue). l represents an image component (reference color component) having a small optical deterioration among components constituting the captured image. H _{1 and 2} are primary differentials (vertical and horizontal directions), H ₃ , _{4 and 5} are secondary differentials (vertical and horizontal directions and their cross terms), and λ and β are weighting factors. In i _c · i _l , represents a Hadamard product.

Is the L2 norm,

Is the L1 norm. As vector x = (x ₁ , x ₂ ,..., X _n ),

L2 norm when the absolute value is

And L1 norm

Are defined by the following equations (4) and (5), respectively.

In Expression (2), the difference between the deteriorated image Bi given the PSF known to the original image i and the actual captured image j is taken. When there is no noise, the value of Expression (2) is 0 if the original image i is accurately estimated. The first term of the expression (3) calculates the L1 norm of the primary differential value of the original image i, so that the ratio of the edge to the entire natural image obtained by imaging a natural (general) subject is calculated. Reflects the characteristics of sparseness. The second term of Expression (3) is a regularization term that decreases in value when the position and width of the edge of the image component l and the image component c to be corrected match. In the second term of Equation (3), H _a i _c · i _l which is the product of the pixel gradient value obtained by the first derivative of the image component c and the pixel value of the image component l, and the first order of the image component l The difference between the pixel gradient value obtained by differentiation and the pixel value of the image component c is calculated as H _a i _l · _ic . A method for estimating the original image i that minimizes the expression (1) is described in detail in Non-Patent Document 1.

  Patent Document 1 discloses a method of correcting edge information of an R component and a B component using edge information of a G component (reference color component) for noise removal.

Japanese Patent No. 5198192

Felix Heide, et al. , “High-Quality Computational Imaging Through Simple Lenses”, ACM Transactions on Graphics (presented at SIGGRAPH 2013)

  However, in the methods disclosed in Non-Patent Document 1 and Patent Document 1, the performance relationship between color components changes in the imaging optical system due to changes in image height, aperture, focal length, etc., so that a sufficient correction effect is obtained. It may not be obtained. For example, when the same color component is used as the reference color component for the entire photographed image, the image height (position of the photographed image) in which the color component having a greater optical deterioration than the color component to be corrected becomes the reference color component May exist. When the correction process is performed using a color component having a larger optical deterioration than the color component to be corrected as a reference color component, the correction effect is reduced.

  In view of such a problem, an object of the present invention is to provide an image processing method and an image processing apparatus capable of correcting optical degradation of a captured image with high accuracy.

  An image processing method according to one aspect of the present invention includes an input data acquisition step for acquiring a captured image having a plurality of color components, and optical characteristics of an imaging optical system used for capturing the captured image, and the optical characteristics. Or a selection step of selecting a reference color component from the color component using at least one of edge information for each color component of the photographed image, and a reference acquisition step of acquiring a reference image based on the reference color component; A correction step of correcting optical deterioration in the color component so that a difference between an edge width in the color component and an edge width in the reference image is reduced using the reference image and the optical characteristic; It is characterized by having.

  An image processing apparatus according to another aspect of the present invention includes an input data acquisition unit that acquires a captured image having a plurality of color components and optical characteristics of an imaging optical system used for capturing the captured image. A selection unit that selects a reference color component from the color component using at least one of the optical characteristics or edge information for each color component of the photographed image, and a reference that generates a reference image based on the reference color component Correction that corrects optical deterioration in the color component so that the difference between the edge width in the color component and the edge width in the reference image is reduced using the acquisition unit, the reference image, and the optical characteristics And a portion.

  ADVANTAGE OF THE INVENTION According to this invention, the image processing method and image processing apparatus which can correct | amend the optical degradation of a picked-up image with high precision can be provided.

1 is a block diagram of an imaging apparatus according to Embodiment 1. FIG. 3 is a flowchart illustrating a correction process according to the first exemplary embodiment. 6 is an explanatory diagram of selection of a reference color component in Embodiment 1. FIG. FIG. 3 is an external view of an image processing system according to a second embodiment. 6 is a block diagram of an image processing system according to Embodiment 2. FIG. 10 is a flowchart illustrating a correction process according to the second embodiment. 6 is a lens cross-sectional view of an image pickup optical system of Example 2. FIG. 6 is an MTF curve of the imaging optical system of Example 2. FIG. It is sectional drawing of PSF with respect to the white light source of the imaging optical system of Example 2, and an image pick-up element. FIG. 10 is a cross-sectional view of R pixel values in a partial region of a captured image before and after correction according to the second exemplary embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

First, terms used in the description of the present embodiment will be described.
"Optical degradation"
The optical degradation is image degradation caused by the imaging optical system, and is image blur caused by various aberrations of the imaging optical system or diffraction blur caused by the optical system having a finite diameter. The optical degradation can be expressed by a point image intensity distribution function (PSF) that is a function representing a response to a point light source of the imaging optical system, an optical transfer function (OTF: Optical Transfer Function) obtained by Fourier transform of the PSF, or the like.
"Shooting state"
The shooting state is, for example, a focal length, an aperture value, a focused subject distance, and the like during imaging by the imaging optical system.
"Optical characteristics of imaging optics"
The optical characteristics of the imaging optical system are OTF, PSF, and the like that indicate optical degradation caused by the imaging optical system, and differ depending on the shooting state. The optical characteristics are preferably calculated in advance by computer simulation from the design values of the imaging optical system. The calculation of the optical characteristics of the imaging optical system by computer simulation uses the spectral characteristics of the assumed light source and the spectral transmittance characteristics of the color filters of each pixel of the imaging device to obtain the same spectral transmittance characteristics (same color filters). This is preferably performed for each pixel. The optical characteristics of the imaging optical system are required by the number of combinations of the imaging state and the spectral transmittance characteristics of the filter of the imaging element. Also, if aberrations and diffraction changes corresponding to the image height cannot be ignored even in the same shooting state and the same spectral transmittance characteristic, it is necessary to generate optical characteristic information for each image height. Further, as the optical characteristics, a measurement result obtained from wavefront measurement or the like may be used instead of a computer simulation calculation result.
"Image edge information"
Image edge information is information about the position, strength, width, and the like of edges in an image. The edge information can be acquired by calculating a spatial differential value (difference value) of the image.

  Hereinafter, specific examples of the present embodiment will be described.

  With reference to FIG. 1, the structure of the imaging device 200 of a present Example is demonstrated. FIG. 1 is a block diagram of the imaging apparatus. The imaging apparatus 200 includes an imaging optical system 201, an imaging element 202, an A / D conversion unit 203, an image processing unit 204, a display unit 205, an imaging optical system control unit 206, a state detection unit 207, a storage unit 208, and an image recording medium 209. And a system controller 210. Note that the imaging optical system 201 may be detachably attached to the imaging device 200.

  The imaging optical system 201 forms an image of light from a subject on an imaging element 202 configured by a CCD sensor, a CMOS sensor, or the like. The imaging optical system 201 includes a diaphragm 201a. The aperture 201a adjusts the amount of light from the subject incident on the image sensor 202 by increasing or decreasing the aperture diameter of the aperture. Further, the imaging optical system 201 may have a focus function and a zoom function. The image sensor 202 has pixels with an RGB Bayer array. However, as long as it has a plurality of color components (spectral distribution, not limited to the visible light range), other configurations may be used. The A / D conversion unit 203 converts an analog signal from the image sensor 202 into a digital signal. The image processing unit 204 includes an input data acquisition unit 204a, a selection unit 204b, a reference acquisition unit 204c, and a correction unit 204d. The image processing unit 204 performs optical deterioration correction processing (image processing) on the A / D converted digital signal (captured image). The image processing unit 204 reads and uses the optical characteristics of the imaging optical system 201 from the storage unit 208 when performing correction processing. The corrected image is displayed on the display unit 205 or stored in the image recording medium 209. The display unit 205 includes a liquid crystal monitor and an organic EL monitor, and the image recording medium 209 is a recording medium such as a magnetic disk, an optical disk, and a flash memory. The imaging optical system control unit 206 performs control of the imaging optical system 201, for example, control of an aperture value, focus, and zoom. The state detection unit 207 acquires the shooting state of the imaging optical system 201 from the imaging optical system control unit 206. The shooting state acquired by the state detection unit 207 is used when the image processing unit 204 performs correction processing. The system controller 210 instructs each member to execute the above-described operation.

  Hereinafter, the optical deterioration correction process executed by the image processing unit 204 will be described with reference to FIG. FIG. 2 is a flowchart showing optical deterioration correction processing executed by the image processing unit 204. An image processing unit 204 as a computer and an image processing apparatus executes an image processing program by executing an image processing program that is a computer program stored in the storage unit 208. Further, the image processing unit 204 may be mounted as a hardware circuit, and image processing may be performed by the hardware circuit.

  In step S101, the input data acquisition unit 204a acquires a captured image (digital signal that has been A / D converted by the A / D conversion unit 203). The photographed image of the present embodiment is an RGB Bayer array image. Therefore, only a single color component exists in the pixels at the same position. Since the processing described below is performed on the premise that a plurality of color components exist in the same pixel, the information of the missing pixel is interpolated and generated by bilinear interpolation for each color component of RGB. A plurality of color components may be included in the same pixel by performing downsampling with a pixel size twice that of RGB.

  In step S102, the input data acquisition unit 204a divides the captured image into a plurality of partial areas, and acquires one of the partial areas. The partial area may be one pixel or a plurality of pixels.

  In step S103, the input data acquisition unit 204a determines whether the optical characteristic corresponding to the partial region acquired in step S102 can be acquired. If it is determined that acquisition is possible, the process proceeds to step S104. If it is determined that acquisition is impossible, the process proceeds to step S106. The optical characteristics corresponding to the partial area include the aperture value, focal length, shooting distance (focusing distance), color component, image height (position of the partial area with respect to the shot image) of the imaging optical system 201, and subject existing in the partial area. The relative distance to the in-focus distance is determined as a parameter. The storage unit 208 stores optical characteristics of the imaging optical system 201 with respect to a plurality of parameters. When the optical characteristics of the combination of parameters corresponding to the partial areas are stored in the storage unit 208, or when the optical characteristics corresponding to the partial areas can be generated by interpolation from a plurality of stored optical characteristics, the input data The acquisition unit 204a determines that the optical characteristic can be acquired. In other cases, the input data acquisition unit 204a determines that the optical characteristics cannot be acquired. In the present embodiment, the storage unit 208 stores only the optical characteristics on the in-focus surface. Therefore, if the partial area is an out-of-focus plane (the relative distance to the in-focus distance of the subject existing in the partial area is outside the range of the depth of field), the corresponding optical characteristics cannot be generated by interpolation and acquired. Determined impossible. A distance map of the subject space is used to determine the in-focus surface and the out-of-focus surface. The distance map is acquired, for example, by photographing a plurality of images. A plurality of images with different in-focus distances are taken, and for each position in the image, a distance map is determined by determining that an image having the highest contrast among the plurality of taken images is an image focused on that position. Can be generated. Further, the distance map may be acquired using a method of dividing the pupil of the imaging optical system 201. For example, the pupil of the imaging optical system 201 is divided by arranging a microlens array on the imaging element 202 so that the exit pupil of the imaging optical system 201 is imaged over a plurality of pixels in the light receiving unit of the imaging element 202. it can. Since a plurality of parallax images can be acquired by pupil division, a distance map can be acquired.

  Further, the input data acquisition unit 204a may consider a manufacturing error of the imaging optical system 201 when performing the determination in step S103. The optical characteristics of the imaging optical system 201 with respect to a plurality of parameters stored in the storage unit 208 are optical characteristics calculated from design values of the imaging optical system 201. Therefore, the optical characteristics of the actual imaging optical system 201 may be different from the optical characteristics stored in the storage unit 208 due to manufacturing errors. The magnitude of the manufacturing error can be estimated by the following method, for example. First, the chromatic aberration of magnification is obtained by detecting an edge for each position of the photographed image and calculating the relative position of the edge between the color components. Next, the magnitude of the manufacturing error can be determined by comparing the obtained lateral chromatic aberration with the lateral chromatic aberration obtained from the design value of the imaging optical system 201. Further, when the imaging optical system 201 is configured as a rotationally symmetric system with respect to the optical axis, a manufacturing error may vary depending on how much the lateral chromatic aberration obtained from the captured image is deviated from the rotational symmetry with respect to the image center. Can judge large and small. The input data acquisition unit 204a determines that the optical characteristic corresponding to the partial region can be acquired when the manufacturing error is equal to or smaller than the predetermined value, and determines that the optical characteristic corresponding to the partial region cannot be acquired when the manufacturing error is larger than the predetermined value. Note that if it is equal to or greater than a predetermined value, it may be determined that acquisition is impossible.

  In step S104, the input data acquisition unit 204a acquires optical characteristics corresponding to the partial region. In the present embodiment, the input data acquisition unit 204a acquires the optical characteristics from the storage unit 208, but may download and acquire it from the network. Since the captured image has RGB color components, the input data acquisition unit 204a acquires optical characteristics corresponding to RGB.

  In step S105, the selection unit 204b selects a reference color component using the optical characteristics and edge information of the partial area. The reference color component is a color component that is referred to when correcting optical deterioration of other color components, and it is desirable to select a color component having the highest imaging performance. However, when the color component having the highest imaging performance is always used as the reference color component, a false edge is generated by correction depending on the subject.

  FIGS. 3A and 3B respectively show the PSF (optical characteristics) of the imaging optical system 201 convoluted with a subject existing in the partial area (corresponding to the partial area), and pixel values in the partial area. FIG. The solid line represents the R color component, the alternate long and short dash line represents the G color component, and the broken line represents the B color component. In FIG. 3A, the imaging performance is high in the order of G, R, and B. However, as shown in FIG. 3B, the subject existing in the partial area is a subject with little change in the G pixel value. When G is selected as a reference color component and R and B images are corrected using G in the partial area as a reference image, the edges of R and B are not corrected well because G does not have an edge. May be blurred. Therefore, a color component having an edge in the partial region and having the best imaging performance may be selected as the reference color component. In the example of FIG. 3, R is a reference color component. However, if the G image is corrected using R as a reference image, a false edge occurs in G. Therefore, for a color component having higher imaging performance than the reference color component, processing for weakening the effect of matching with the reference image is performed in a later correction step.

  If no edge exists in all color components, the reference color component is determined from the imaging performance. As an index (performance value) representing imaging performance, for example, a PSF peak value, a half width of PSF, a dispersion of PSF, or a value of MTF (Modulation Transfer Function) at a predetermined spatial frequency may be used. Good. The higher the performance value, the larger the performance value. Therefore, when the PSF peak value or MTF value is used, the value as it is can be used as the performance value, and when the half width or dispersion of the PSF is used, for example, the reciprocal can be used as the performance value. Whether or not there is an edge in the partial area may be determined, for example, by taking a spatial differentiation in the vertical and horizontal directions of the partial area and determining whether there is a place where the absolute value of the differentiation is greater than or equal to a threshold value. Further, it may be determined whether the differential value whose absolute value is equal to or greater than the threshold value is further spatially arranged more than the second threshold value (whether the edge width is equal to or greater than the second threshold value). . Alternatively, each color component of the partial region may be subjected to Fourier transform, and it may be determined whether the intensity of a predetermined spatial frequency is greater than or equal to a third threshold value.

  In addition, when acquiring edge information, it is desirable that magnification chromatic aberration correction is performed in advance. If there is chromatic aberration of magnification, if it is determined that there is no edge in the partial area with the specified color component, the color component of the edge will be shifted outside the partial area due to chromatic aberration of magnification whether the subject actually has no edge information of the color component. I can't tell if it was. Therefore, in order to determine the reference color component with high accuracy, it is desirable to obtain edge information after correcting the chromatic aberration of magnification for a partial region (or a captured image before obtaining the partial region).

  In step S106, the input data acquisition unit 204a has an optical characteristic closest to the optical characteristic acting on the partial region among the optical characteristics stored in the storage unit 208 (or optical characteristics generated by interpolation). To get. In this embodiment, since only the optical characteristics of the in-focus surface are stored in the storage unit 208, the optical characteristics of the in-focus surface are acquired even when the partial area is a non-in-focus surface. Even when a manufacturing error occurs and the optical characteristics cannot be acquired, the optical characteristics of the same shooting state and image height obtained by the design values are acquired. Note that the process in step S106 may be executed anywhere before the process in step S109.

  In step S107, the selection unit 204b selects the reference color component only from the edge information of the partial area. The color component having the sharpest edge is used as a reference color component with the strength of the edge (absolute value of the spatial differentiation of the pixel value), the width of the edge, etc. as indices. When no edge exists in all color components, G is set as a reference color component. R or B may be a reference color component.

  In step S108, the reference acquisition unit 204c acquires a reference image based on the reference color component. The reference color component of the partial area may be used as it is as a reference image, or an image obtained by correcting optical degradation for the reference color component of the partial area may be used as the reference image. The optical deterioration correction used for generating the reference image can be executed by, for example, a Wiener filter or an optimization process represented by Expression (1). When using Expression (1), for example, Expression (2) may be used for the loss function, and the first term of Expression (3) may be used for the regularization function. The PSF (or OTF) used for correction is the reference color component of the optical characteristics acquired in step S104 or step S106. In addition, when an image subjected to optical degradation correction is used as a reference image, an image in which optical degradation of the entire captured image is corrected using optical characteristics may be generated in advance before performing the process of step S108. . Then, the reference image may be acquired by extracting a portion corresponding to the partial region from the generated image (only the reference color component is extracted).

  In step S109, the correction unit 204d corrects optical degradation for the color components excluding the reference color component in the partial region using the reference image and the optical characteristics. At this time, the correction is performed so that the difference between the edge width of the color component to be corrected and the edge width of the reference image is reduced. Specifically, the correction can be performed by solving the optimization problem of Expression (1) using the loss function of Expression (2) and the regularization function of Expression (3). The PSF (or OTF) used for correction corresponds to the color component that is the correction target of the optical characteristics acquired in step S104 or step S106.

  In addition, it is desirable to correct optical deterioration after correcting chromatic aberration of magnification for each color component. Since the correction is performed so as to match the edge of the reference image, if the edge position is shifted for each color due to the chromatic aberration of magnification, a false edge is generated.

  Further, the weight of the reference image for correction may be determined based on the reference color component and the optical characteristics of the color component to be corrected. When the reference color component is determined using edge information, and the performance value of the correction target color component is higher than the reference color component, the correction target color component is not originally blurred due to optical deterioration, and is originally This means that the subject has no edge of that color. Accordingly, when correction is performed in accordance with the reference image, a false edge is generated. Therefore, the weight of the reference image for correction may be lowered, or the weight may be zero (the reference image is not used during correction). For example, when correction is performed using the regularization function of Expression (3), β is the weight of the reference image for correction. Similarly, for a partial region determined to have no edge in all color components, the weight of the reference image for correction may be lowered, or the reference image may not be used for correction. Since the reference image is used for the purpose of improving the edge correction effect, it is not necessary to refer to the reference image in an area where there is no edge.

  If there is no edge in all the color components of the partial area, the optical degradation in each color component may be corrected without selecting the reference color component and acquiring the reference image.

  In step S110, the correction unit 204d determines whether or not all the predetermined partial areas have been corrected. The predetermined partial area is arbitrary, and may be the entire captured image, for example. If all the predetermined partial areas have been corrected, the process proceeds to step S111. If not corrected, the process returns to step S102 to acquire and correct a new partial area.

  In step S111, the correction unit 204d generates a color image. A color image is an image in which partial areas in which optical degradation is corrected are arranged. In step S110, when the predetermined partial area is the entire captured image, the color image is an image in which optical degradation is corrected in the entire captured image. The correction unit 104d generates a color image by performing processing such as demosaicing processing, white balance adjustment, shading correction, and γ correction as necessary.

  As described above, the imaging apparatus 200 according to the present embodiment can correct optical degradation of a captured image with high accuracy.

  FIG. 4 is an external view of an image processing system including an image processing apparatus 1003 that executes the optical deterioration correction process of the present invention. FIG. 5 is a block diagram of the image processing system.

  The configuration of the imaging apparatus 1001 is the same as the configuration of the imaging apparatus 200 of Example 1 described with reference to FIG. In the first embodiment, the image processing unit 204 built in the imaging apparatus 200 performs optical deterioration correction processing. On the other hand, in the present embodiment, the image processing apparatus 1003, which is an apparatus different from the imaging apparatus 200, executes optical deterioration correction processing. The image processing apparatus 1003 includes a storage unit 1003a, an input data acquisition unit 1003b, a selection unit 1003c, a reference acquisition unit 1003d, and a correction unit 1003e. The image processing apparatus 1003 acquires a captured image from the imaging apparatus 1001 and executes optical deterioration correction processing using the optical characteristics of the imaging optical system of the imaging apparatus 1001 stored in the storage unit 1003a. A display unit 1002 displays the captured image corrected by the image processing apparatus 1003.

  Hereinafter, with reference to FIG. 6, the optical deterioration correction process executed by the image processing apparatus 1003 will be described. FIG. 6 is a flowchart showing optical deterioration correction processing executed by the image processing apparatus 1003. The description of the same processing as in the first embodiment is omitted.

  In step S <b> 201, the input data acquisition unit 1003 b acquires a captured image from the imaging device 1001. The captured image has a plurality of color components.

  In step S202, the input data acquisition unit 1003b acquires a partial region from the captured image.

  In step S203, the input data acquisition unit 1003b acquires the optical characteristics corresponding to the partial area from the storage unit 1003a. The input data acquisition unit 1003b specifies the optical characteristics corresponding to the partial area based on the shooting state of the imaging device 1001 at the time of shooting that can be acquired from the Exif information attached to the shot image and the position of the partial area with respect to the shot image. .

  In step S204, the selection unit 1003c selects a reference color component based on the optical characteristics. The selection unit 1003c selects the color component having the highest performance value as the reference color component. In this embodiment, the optical characteristic is OTF, and the selection unit 1003c acquires the performance value from the MTF that is the absolute value of the OTF. FIG. 7 is a lens cross-sectional view of the imaging optical system of the imaging apparatus 1001, and FIG. 8 is a graph plotting the MTF of the imaging optical system against the spatial frequency. The G MTF is higher at all frequencies than the R and B MTFs. Therefore, the selection unit 1003c selects G as a reference color component. The selection unit 1003c may determine a performance value from the PSF. FIG. 9 is a cross-sectional view of the PSF with respect to the white light source of the imaging optical system and the imaging device. 9A to 9C are cross-sectional views of the PSF with respect to R, G, and B, respectively. Based on the PSF peak value and half-value width in these figures, it can be determined that G has the highest performance.

  In step S205, the reference acquisition unit 1003d acquires a reference image. The reference acquisition unit 1003d may acquire the G component of the partial region as it is as a reference image, or may acquire an image in which optical degradation is corrected using G optical characteristics as a reference image.

  In step S206, the correction unit 1003e corrects optical degradation for color components other than the reference color component in the partial region using the reference image and the optical characteristics. FIG. 10 is a cross-sectional view of an R pixel value in a partial region of a captured image before and after correction. A photographed image in which a PSF for R of the imaging optical system shown in FIG. 9A is convoluted with respect to a subject (correct image) without optical deterioration is a photographed image before correction. By performing the correction using the reference image, the pixel value is corrected as in the corrected captured image shown in FIG.

  In step S207, the correction unit 1003e determines whether or not all the predetermined partial areas have been corrected. If all the predetermined partial areas have been corrected, the process proceeds to step S208. If not corrected, the process returns to step S202 to acquire and correct a new partial area.

  In step S208, the correction unit 1003e generates a color image by arranging the corrected partial areas.

  As described above, the image processing system of the present embodiment can correct optical degradation of a captured image with high accuracy.

  The optical deterioration correction processing described in the first and second embodiments can be performed by the system of the second embodiment and the imaging apparatus of the first embodiment, respectively.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

204 Image processing unit (image processing apparatus)
204a Input data acquisition unit 204b Selection unit 204c Reference acquisition unit 204d Correction unit

Claims (15)

An input data acquisition step for acquiring a captured image having a plurality of color components, and optical characteristics of an imaging optical system used for capturing the captured image;
A selection step of selecting a reference color component from the color component using at least one of the optical characteristics or edge information for each color component of the captured image;
A reference acquisition step for acquiring a reference image based on the reference color component;
Using the reference image and the optical characteristics, a correction step of correcting optical deterioration in the color component so that a difference between an edge width in the color component and an edge width in the reference image is reduced; An image processing method comprising:   The image processing method according to claim 1, wherein the selecting step selects the reference color component for each of a plurality of partial regions obtained by dividing the captured image.   The said selection process acquires the performance value for every said color component in the said imaging optical system from the said optical characteristic, The said reference color component is selected based on the said performance value, The Claim 1 or 2 characterized by the above-mentioned. Image processing method.   The image according to claim 3, wherein the performance value includes any one of a peak value, a half-value width, and a dispersion of a point spread function, or a value of a modulation transfer function at a predetermined spatial frequency. Processing method. The selection step selects the reference color component using the edge information,
5. The weight of the reference image for correction is determined based on a color component to be corrected and the optical characteristic of the reference color component in the correction step. An image processing method described in 1. The selection step selects the reference color component using the edge information,
The correction step acquires the performance value of the color component in the imaging optical system from the optical characteristics, and when the performance value of the color component to be corrected is larger than the performance value of the reference color component, The image processing method according to claim 1, wherein a weight is reduced or the reference image is not used for correction.   The image processing method according to claim 1, wherein the selection step uses the edge information that has been subjected to magnification chromatic aberration correction.   8. The image processing method according to claim 1, wherein the correction step corrects the optical deterioration with respect to the color component that has been subjected to magnification chromatic aberration correction. 9.   The image processing method according to claim 1, wherein the optical characteristic used in the correction step is a point image intensity distribution function of the imaging optical system corresponding to the color component. . The image processing method according to claim 1, wherein the edge information includes edge strength or width.   11. The optical characteristic according to claim 1, wherein the optical characteristic is determined using any one of a photographing distance when the photographed image is photographed, an aperture value of the imaging optical system, and a focal length. 2. The image processing method according to item 1. An input data acquisition unit that acquires a captured image having a plurality of color components, and optical characteristics of an imaging optical system used for capturing the captured image;
A selection unit that selects a reference color component from the color component using at least one of the optical characteristics or edge information for each color component of the captured image;
A reference acquisition unit that generates a reference image based on the reference color component;
A correction unit that corrects optical deterioration in the color component so that a difference between an edge width in the color component and an edge width in the reference image is reduced using the reference image and the optical characteristics; An image processing apparatus comprising: An imaging optical system that forms an image of the subject space;
An image sensor having a plurality of pixels having different color components and acquiring the image as a captured image;
An image processing apparatus comprising: the image processing apparatus according to claim 12.   An image processing program for causing a computer to execute the image processing method according to any one of claims 1 to 11. A computer-readable recording medium on which the image processing program according to claim 14 is recorded.