JP2012003454A - Image processing apparatus, imaging device and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of realizing a sufficient high-definition image under various imaging conditions.SOLUTION: An image processing device 110 comprises regional division means 1105 for dividing an input image generated by imaging systems 100, 102 into a plurality of regions having different evaluation values obtained using a response function of the imaging systems, and processing means 1104 which uses response functions respectively different for the divided regions to perform image processing to increase the image definition individually for the regions. The processing means performs the image processing using a response function which corresponds to an imaging condition when the input image is generated at the imaging system and which is read from storage means or generated by calculation.

Description

The present invention relates to a high-definition image generation technique for improving the resolution of an image acquired from an imaging apparatus.

A super-resolution technique is known as a technique for increasing the definition of an image generated by an imaging system. In the super-resolution technique classified as the reconstruction type, the degradation process in the imaging system is modeled as an observation model, and the resolution is increased as the inverse problem.

  As a method for solving the inverse problem, a frequency domain method using aliasing on a captured image and a MAP (Maximum a posteriori) method for estimating a high-definition image by optimizing the posterior probability of prior information have been proposed. In addition, a method of estimating by combining various prior information called POCS (Projection onto Convex Sets) method has been proposed.

  The observation model is composed of an imaging system response function obtained from the effects of the motion of the imaging system, deterioration due to the optical system, CF (Color Filter) and LPF (Low-Pass Filter), and sampling in the image sensor. It is known that the reproduction accuracy of the degradation process by the actual imaging system in this observation model greatly depends on the accuracy of high definition by the super-resolution technique, and various observation models have been proposed.

  Patent Document 1 discloses super-resolution processing using an observation model that uses an optical system PSF (Point Spread Function) that represents degradation due to an optical system as a response function of an imaging system.

JP 2005-095328 A

However, in the super-resolution processing using a single response function for the entire area of the image disclosed in Patent Document 1, the difference between the single response function and the response function of the original imaging system is large. In such a region, sufficient high definition cannot be performed.

  In addition, even though the imaging conditions such as aperture value, focal length, and wavelength of imaging light (imaging wavelength) have been changed, a good high-definition image can be obtained even when super-resolution processing is performed using the same response function. You may not be able to get it.

  The present invention provides an image processing apparatus and an image processing program capable of sufficiently high-definition of an image even under various imaging conditions.

An image processing apparatus according to an aspect of the present invention includes an area dividing unit that divides an input image generated by an imaging system into a plurality of areas having differences in evaluation values obtained using a response function of the imaging system, Each region has processing means for performing image processing for increasing image definition using a response function that is different for each region. The processing means is a response function corresponding to an imaging condition at the time of generation of the input image in the imaging system, and performs image processing using a response function read from the storage means or generated by calculation. And

  An imaging apparatus having an imaging system and the image processing apparatus also constitutes another aspect of the present invention.

  An image processing program according to still another aspect of the present invention provides a computer with an input image generated by an imaging system in a plurality of regions having differences in evaluation values obtained using the response function of the imaging system. A division step, and an image processing step for performing image processing for increasing the image definition using a response function different for each region. In the image processing step, image processing is performed using a response function corresponding to an imaging condition at the time of generating an input image in the imaging system, which is read from a storage unit or generated by calculation. And

  The storage medium storing the image processing program also constitutes another aspect of the present invention.

According to the present invention, it is possible to perform high-definition processing on an input image using a response function of an imaging system corresponding to an imaging condition at the time of generating the input image. Thereby, a good high-definition image can be generated even from an input image generated under various imaging conditions.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. 3 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment. 3 is a flowchart illustrating image region division processing in the imaging apparatus according to the first embodiment. 3 is a flowchart illustrating high-definition image processing in the imaging apparatus according to the first embodiment. FIG. 3 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 2 of the present invention. 9 is a flowchart illustrating the operation of the imaging apparatus according to the second embodiment. The figure which shows PSF in the case where F value is small and large. FIG. 6 is a diagram showing a PSF generated by linear interpolation in the second embodiment. FIG. 10 is a diagram illustrating a Gaussian function used for PSF correction in the second embodiment. FIG. 6 is a diagram showing PSF corrected with a Gaussian function in the second embodiment. Schematic which shows the structure of the image processing apparatus which is Example 3 of this invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration of an imaging apparatus 1 that is Embodiment 1 of the present invention. The imaging apparatus 1 includes an imaging optical system 100, an exposure amount control member 101, an imaging element 102, an AD conversion unit 103, a zoom control unit 104, an exposure control unit 105, an AF control unit 106, and shake detection. Part 107. In addition, the imaging apparatus 1 includes a system control circuit 108, a memory 109, an image processing apparatus 110, a visual correction unit 111, a compression unit 112, and an image output unit 113. f is a subject imaged by the imaging apparatus 1.

  In FIG. 1, light from a subject f forms an image on an image sensor 102 through an imaging optical system 100 and an exposure amount control member 101. The exposure control member 101 includes a diaphragm and a shutter.

  The image sensor 102 is an image sensor such as a CCD sensor or a CMOS sensor that converts a subject image into an electrical signal. The AD conversion unit 103 converts the analog electric signal output from the image sensor 102 into digital data. The imaging optical system 100, the exposure amount control member 101, the imaging element 102, and the AD conversion unit 103 constitute an imaging system.

  The zoom control unit 104 controls zooming of the imaging optical system 100. The exposure control unit 105 controls the operation of the exposure control member 101. The AF control unit 106 controls focusing of the imaging optical system 100.

  The shake detection unit 107 detects the shake of the imaging device 1. The system control circuit 108 includes a microcomputer that controls the operation of the entire imaging apparatus 1.

  The memory 109 stores data for controlling operations in the system control circuit 108 and processing programs (including an image processing program described later).

  The image processing apparatus 110 increases the image definition of the captured image (input image) converted into digital data by the AD conversion unit 103, that is, performs image processing (high definition processing) for high definition. Generate high-definition images.

  The visual correction unit 111 performs processing for improving the image quality of the high-definition image generated by the image processing apparatus 110, for example, image correction processing such as tone curve (gamma) correction, saturation enhancement, hue correction, and edge enhancement.

  The compression unit 112 compresses the high-definition image output from the visual correction unit 111 by a compression method such as JPEG to reduce the image size for recording.

  The image output unit 113 displays the image compressed by the compression unit 112 on a monitor (not shown) or records it on a recording medium such as a semiconductor memory or an optical disk.

  Next, the configuration of the image processing apparatus 110 will be described. An imaging condition acquisition unit 1101 acquires imaging conditions such as an imaging magnification, an aperture value, and a focal length from the system control unit 108.

  An imaging system response function storage unit 1002 is an imaging system response function (hereinafter referred to as an imaging system response function), and includes a plurality of imaging system response functions corresponding to a plurality of imaging conditions (imaging system responses for each imaging condition). Function) is stored in advance.

  An imaging system response function selection unit 1103 corresponds to the imaging conditions at the time of imaging (generation) of a captured image as an input image from a plurality of imaging system response functions stored in the imaging system response function storage unit 1102. Select the imaging system response function.

  An image area dividing unit 1105 divides a captured image from the AD conversion unit 103 into a plurality of areas using an imaging system response function for area division. The imaging system response function for area division will be described later.

  Reference numeral 1104 denotes a high-definition image processing unit that uses the image region division information acquired by the image region dividing unit 1105 and the imaging system response function corresponding to the imaging conditions at the time of imaging of the captured image acquired from the imaging system response function selection unit 1103. Then, high definition processing is performed on the input image.

  Next, the operation of the imaging apparatus 1 will be described using the flowchart of FIG. FIG. 2 is a flowchart illustrating a process from capturing an image of the subject f by the image capturing apparatus 1 to generating a high-definition image from the captured image and outputting the image from the image output unit 113. This processing is executed by the system control circuit 108 according to a computer program (image processing program) stored in a memory (not shown). This is the same in other embodiments described later.

  In step S201, when a shooting start button (not shown) is operated by the user, the system control circuit 108 causes the exposure control unit 105 to determine an exposure condition (aperture value, exposure time, etc.) and causes the AF control unit 106 to perform focusing. To do. Imaging conditions such as the determined exposure condition, focus position, and zoom position are stored in the memory 109.

  In addition, when the system control circuit 108 detects that vibration such as camera shake of the imaging apparatus 1 is generated through the shake detection unit 107, the system control circuit 108 drives a shake correction optical system (not shown) in the imaging optical system 100. Then, image blur correction is performed.

  Then, the system control circuit 108 causes the image sensor 102 to photoelectrically convert the subject image. An analog signal corresponding to the brightness of the subject image is output from the image sensor 102. The analog signal is converted into digital data by the AD conversion unit 103 to generate a captured image. The captured image is transmitted to the image processing apparatus 110.

  Next, in step S202, the system control circuit 108 uses the imaging condition acquisition unit 1101 to acquire imaging conditions at the time of imaging a captured image that is a target for high definition processing. The memory 109 stores the imaging condition data at the time of imaging in the step S201, and the imaging condition data according to the request from the imaging condition acquisition unit 1101 is transmitted to the imaging condition acquisition unit 1101. As described above, the imaging condition may be at least one of the aperture value, the imaging magnification, and the focal length, or any other parameter that changes the imaging system response function.

  Next, in step S203, the system control circuit 108 causes the image area dividing unit 1105 to perform image area dividing processing for dividing the captured image into a plurality of areas as shift invariant areas. Here, the shift invariant region means a region that uses the same imaging system response function for high definition processing. In other words, when there are a plurality of shift invariant areas, high definition processing using a different imaging system response function is performed for each shift invariant area.

  The image area dividing process will be described with reference to the flowchart shown in FIG. First, in step S301, the image area dividing unit 1105 performs temporary area division of a certain target image area. The temporary area division is performed in order to calculate evaluation values of a plurality of areas (temporary divided areas) obtained by trying the area division. In accordance with the evaluation value, it is determined whether or not the temporary divided area is subsequently used as a formal divided area.

  In step S302, the image region dividing unit 1105 generates an imaging system response function for region division for a representative position of each temporary divided region (for example, a central pixel of each temporary divided region). Any method may be used to generate the imaging system response function for area division, for example, a method of calculating by ray tracing using design information of the imaging system, or an imaging system response for area division that has been measured and stored in advance. A method of reading a function may be used.

  Next, in step S303, the image region dividing unit 1105 uses the imaging system response function for region division of each temporary divided region generated in step S302 to evaluate the evaluation value in each temporary divided region (hereinafter referred to as a divided evaluation value). ) I is calculated.

  In this embodiment, the division evaluation value I is calculated by the following equation (1) using an imaging system MTF (Modulation Transfer Function) which is one of imaging system response functions.

Here, ξ is a spatial frequency, and MTF is an imaging system MTF. ξ ₁ and ξ ₂ are constants for setting the upper and lower limits of the spatial frequency. The division evaluation value I is obtained by integrating the imaging system MTF at the spatial frequency.

  The division evaluation value I expressed by the equation (1) is calculated for each temporary division region, and comparing these evaluation values is equivalent to evaluating the change (difference) in the imaging system response function for region division. . For this reason, as will be described below, it is possible to obtain a result of image region division corresponding to a change in the imaging system response function for region division.

  In step S304, the image area dividing unit 1105 compares the division evaluation values of the temporary divided areas calculated in step S303. Specifically, a difference between the division evaluation values of the temporary divided regions is obtained, and it is determined whether or not the absolute value of the difference is equal to or greater than a preset threshold value (predetermined value) α. When there is at least one provisional divided region combination having a divided evaluation value at which the absolute value of the difference is equal to or greater than the threshold value α, the process proceeds to step S305.

  In step S305, the image area dividing unit 1105 sets the temporary divided area as a formal divided area. Then, the image region dividing unit 1105 repeats the processing from step S301 to step S305 for each divided region obtained in this way (that is, for each divided region).

  On the other hand, in step S304, if there is no combination of temporary divided areas having a divided evaluation value at which the absolute value of the difference is greater than or equal to the threshold value α, the image area dividing unit 1105 sets the temporary divided area as a formal divided area. Not set. In this case, the target image area before being divided into temporary divided areas in step S301 is set as a shift invariant area.

  The image area dividing unit 1105 ends the image area dividing process for the divided areas that have become the shift invariant areas. Further, when all the divided image areas are shifted invariant areas, the image area dividing process is terminated.

  When the change of the imaging system response function for region division is small at the center of the entire image (all image regions) and is large toward the periphery, the final region division result of all the image regions 701 is as shown in FIG. become. In FIG. 3B, a division result is obtained in which the size of the divided region is changed in accordance with the change of the imaging response function for dividing the region.

  Returning to FIG. 2, in step S <b> 204, the system control circuit 108 acquires, in step S <b> 202, a plurality of imaging system response functions stored in the imaging system response function storage unit 1102 in the imaging system response function selection unit 1103. The imaging system response function corresponding to the imaging condition thus selected is selected.

  Here, a storage method of the imaging system response function in the imaging system response function storage unit 1102 and a selection method of the imaging system response function corresponding to the imaging condition will be described.

  The imaging system response function represents a deterioration process in the imaging system that occurs until light incident on the optical system is displayed and recorded as a digital image. As a representation method of the imaging system response function, there are a point spread function PSF (Point Spread Function), an optical transfer function OTF (Optical Transfer Function), and the like. In this embodiment, a case where PSF is used as an imaging system response function will be described, but the same processing can be applied in other cases.

  PSF expresses what kind of image is finally seen when a point light source is imaged by an imaging system as a three-dimensional shape of light intensity and spread. For this reason, the shape of the PSF varies greatly depending on the imaging conditions at the time of imaging. For example, if the F value of the aperture is increased during imaging, the PSF has a wide and gentle shape. Conversely, if the F value is decreased, the PSF has a narrow and steep shape. In this way, since the shape of the PSF changes depending on the imaging conditions, if only the PSF under a specific imaging condition or a PSF approximated by a Gaussian shape is stored in the imaging apparatus, all of the various imaging conditions are stored. A good high definition image cannot be generated.

  Therefore, in this embodiment, PSFs in all imaging conditions within the range that the imaging system can take are stored in the imaging system response function storage unit 1102 to generate a good high-definition image under any imaging conditions. Allow possible PSFs to be used. That is, the imaging system response function storage unit 1102 stores the PSF for each pixel of the captured image for each minimum unit for which imaging conditions can be set.

  The PSF for each imaging condition may be stored, for example, measured at the time of factory shipment of the imaging apparatus, or may be calculated and stored by ray tracing using optical design information. In addition, any storage method may be used, such as a user imaging a test pattern and calculating a PSF therefrom, or downloading and storing PSF data corresponding to the imaging apparatus via a network.

  Then, the imaging system response function selection unit 1103 selects a PSF corresponding to the imaging condition acquired in step S202 from the PSFs stored in the imaging system response function storage unit 1102. Here, the imaging conditions used for selecting the PSF include the wavelength of light (imaging light) from the subject f in addition to the above-described aperture value, imaging magnification, and focal length. When PSFs for each wavelength are stored, for example, if the imaging apparatus is a three-plate color camera, the PSF corresponding to each wavelength of RGB is selected from the imaging system response function storage unit 1102. Further, when shooting with a color filter attached to a black and white camera, a PSF at a wavelength corresponding to the spectral characteristics of the color filter may be selected.

  As described above, the PSF corresponding to the imaging condition selected by the imaging system response function selection unit 1103 is sent to the high-definition image processing unit 1104.

  In step S205 of FIG. 2, the system control circuit 108 causes the high-definition image processing unit 1104 to perform high-definition processing on the captured image. Details of the processing in this step will be described with reference to the flowchart of FIG. Here, reconfigurable super-resolution processing is performed as high definition processing. The reconstruction type super-resolution processing is to reconstruct a high-resolution image before the deterioration process by modeling the deterioration process of the imaging system and solving the inverse problem.

  First, the image for which super-resolution processing is performed is used as a reference image, and multiple images (viewpoint images) obtained by imaging in which the viewpoint position with respect to the subject is slightly changed with respect to the same shooting scene as the reference image are subjected to image processing calculation To the unit 1103. In the following description, a plurality of images having different viewpoint positions from the reference image are all referred to as captured images, and these are collectively referred to as a captured image group. The number of captured images is N. At this time, the number of images included in the captured image group is appropriately set according to the performance of the imaging system, the magnification of the super-resolution processing, and the use of the image after super-resolution processing (super-resolution image).

  In step S401, the high-definition image processing unit 1104 enlarges the input reference image according to the set enlargement magnification, and generates an initial estimated high-definition image (hereinafter referred to as an initial estimated high-definition image). As the enlargement processing, any method such as simple linear interpolation or spline interpolation may be used.

  Next, in step S402, the high-definition image processing unit 1104 obtains the result of the image region division processing (image region division information) in step S203, the imaging system response function for region division corresponding to each divided region, and the initial estimated high-definition. The deterioration process by the imaging system is calculated using the image. Thereby, a low resolution image is generated. For the calculation of the deterioration process by the imaging system, an observation model that models the deterioration process is used. The observation model is given as follows using PSF (Point Spread Function) which is one of the imaging system response functions.

However, X is a high-definition image and Y ′ _k is the k-th low-resolution image after the deterioration process. * Indicates a convolution operation, and n indicates additional noise. PSF _k represents an imaging system response function for the k-th captured image in the captured image group. For the calculation of the deterioration process, the first term on the right side of Equation (2) is used. Note that the observation model is not limited to the one represented by the above calculation formula, and may be appropriately determined according to the characteristics of the imaging system and the algorithm of the super-resolution processing.

  When calculating the degradation process, Expression (2) is calculated using the image area division information and using the imaging system response function corresponding to the divided area including the pixel to be calculated.

  Next, in step S403, the high-definition image processing unit 1104 calculates an evaluation value (hereinafter referred to as a super-resolution evaluation value) J by using the low-resolution image group obtained by calculating the degradation process processing according to the following equation (3). calculate.

Here, Y _k represents the k-th captured image, Y ′ _k represents the k-th low-resolution image, and α represents the constraint parameter. Further, c is shown a matrix representing prior information for estimated high resolution image, ‖‖ ₂ shows the L2 norm.

  The first term of the super-resolution evaluation value J is a term for evaluating the estimated high-definition image by comparing the captured image with a low-resolution image created by degrading the estimated high-definition image. The second term of the super-resolution evaluation value J is a term for evaluating the likelihood of the estimated high-definition image, and the evaluation is performed using known information and properties of the estimated high-definition image. The constraint parameter α is appropriately set according to the image and the imaging system. For example, a high-pass filter is used for the matrix c.

The super-resolution evaluation value J is evaluated using a bilateral filter described in the following document A by Sina Farsiu et. Or may be evaluated.
[Reference A] “Multiframe Demosaicing and Super-Resolution of ColorImages” (IEEE TRANSACTION ON IMAGE PROCESSING, VOL.15, NO.1, JANUARY 2006)
In step S404, the high-definition image processing unit 1104 determines whether the super-resolution evaluation value J calculated in step S403 is smaller than a threshold value (predetermined value) β. The threshold value β is appropriately set according to the type of image, the performance of the imaging system, and the purpose of high definition processing. If J ≧ β, the process proceeds to step S405. If J <β, the high-definition processing ends, and the estimated high-definition image is set as a super-resolution image. Then, it progresses to step S206 of FIG.

  In step S405, the high-definition image processing unit 1104 updates the estimated high-definition image in a direction in which the super-resolution evaluation value J calculated in step S403 is optimized. As the update method, any method can be used. For example, using the steepest descent method,

You can update as Here, X ′ represents the high-definition image after update, and γ is a constant for setting the update step size. Thereafter, X ′ is assumed to be an estimated high-definition image, and the processing from step S402 is repeated until J <β to obtain a super-resolution image. Then, the process proceeds to step S206.

  In step S206 of FIG. 2, the system control circuit 108 performs processing for outputting the high-definition image generated in step S205. First, the visual correction unit 111 is caused to perform the above-described image correction processing for a high-definition image. Next, the compression unit 112 performs compression processing of the high-definition image after the image correction processing. Then, the compressed high-definition image is output from the image output unit 113.

  As described above, in this embodiment, the captured image is divided into a plurality of regions having differences in the divided evaluation values obtained using the imaging system response function for region division. In the high-definition processing for each region, an imaging system response function corresponding to the imaging condition at the time of imaging of the captured image is selected and used from the imaging system response function for each imaging condition stored in advance. As a result, it is possible to perform high definition processing using an imaging system response function suitable for the imaging conditions for each divided area, and a high-definition image better than when a single PSF is used. Can be generated.

FIG. 5 shows a configuration of an image pickup apparatus 1 ′ that is Embodiment 2 of the present invention. In this embodiment, an imaging system response function corresponding to an actual imaging condition is generated from the imaging system response functions stored discretely.

  In FIG. 5, components common to the components shown in FIG. 1 are assigned the same reference numerals as those in FIG. 1. The imaging apparatus 1 ′ according to the present exemplary embodiment includes an imaging system response function generation unit 4101 in addition to the configuration illustrated in FIG. Hereinafter, the operation of the image pickup apparatus of the present embodiment will be described with reference to the flowchart shown in FIG.

  Steps S501 to S503 are the same as steps S201 to S203 shown in FIG. Here, it is assumed that the imaging system response function storage unit 1102 according to the present embodiment discretely stores the imaging system response function at predetermined imaging condition intervals. In other words, the imaging system response function storage unit 1102 stores a plurality of imaging system response functions corresponding to a plurality of representative imaging conditions.

  In step S504, the system control circuit 108 causes the imaging system response function selection unit 1103 to select an imaging system response function corresponding to the imaging condition at the time of imaging acquired in step S502. However, in this embodiment, since the imaging system response functions are discretely stored in the imaging system response function storage unit 1102, there may be no imaging system response functions that directly correspond to the imaging conditions acquired in step S502. is there. In this case, the imaging system response function selection unit 1103 selects one or a plurality of imaging system response functions that are close to the imaging conditions at the time of imaging from among the discretely stored imaging system response functions, and the subsequent imaging system It is transmitted to the response function generation unit 4101.

  In step S504, the system control circuit 108 uses the imaging system response function close to the imaging condition at the time of imaging to the imaging system response function generation unit 4101 to generate an imaging system response function corresponding to the imaging condition at the time of imaging. Let

  One method for generating an imaging system response function corresponding to the imaging conditions at the time of imaging is interpolation processing using a plurality of imaging system response functions. Here, a method for performing linear interpolation processing will be described. Although the case where PSF is used as the imaging system response function will be described here, the same processing can be applied to other cases.

  In linear interpolation, a PSF existing between two PSFs is calculated assuming that the PSF linearly changes in accordance with a change in imaging conditions. It is assumed that the imaging system response function PSF_a corresponding to the aperture F value a, which is one of the imaging conditions, and the imaging system response function PSF_b corresponding to the F value b are selected by the imaging system response function selection unit 1103.

  FIG. 7A shows the shape of PSF_a, and FIG. 7B shows the shape of PSF_b. PSF_a indicates a PSF when the F value is small, and has a steep shape. On the other hand, PSF_b indicates a PSF when the F value is large, and has a wider shape than PSF_a. From these figures, it can be seen that the shape of the PSF varies greatly with different F values.

  For this reason, for example, even if high-definition processing is performed on an image having an F value b whose actual PSF shape is PSF_b using PSF_a corresponding to the F value a, the imaging conditions at the time of imaging are correctly handled. Since it is not a PSF, a good high-definition image cannot be obtained. Therefore, when performing high definition processing, it is necessary to use a PSF corresponding to the imaging condition. When there is no PSF corresponding to the imaging system response function storage unit 1102, a corresponding PSF is generated by interpolation processing. There is a need to.

  As an example of linear interpolation processing, an imaging system response function PSF_c corresponding to the F value c when the F value c of the aperture acquired in step S502 is between the F value a and the F value b is calculated. The calculation formula is as follows.

Here, α in the equation (5) represents a ratio between the difference between the F value a and the F value c and the difference between the F value b and the F value c, and from the F value c with respect to PSF_a and PSF_b. It represents that the weighting according to the difference is performed.

  FIG. 8 shows the result of applying equation (5) when the F value c is exactly halfway between the F value a and the F value b, that is, when α = 0.5. From FIG. 8, PSF_c has an intermediate shape between PSF_a and PSF_b, and it can be confirmed that a PSF corresponding to the F value c that is intermediate between the F value a and the F value b is generated by linear interpolation. .

  As described above, by performing linear interpolation on the two PSFs selected by the imaging system response function selection unit 1103, a PSF corresponding to the imaging condition at the time of imaging can be generated.

  In this embodiment, the case where linear interpolation processing is performed as the interpolation processing has been described, but other interpolation processing such as spline interpolation and parabolic interpolation may be performed.

  Further, as another example of a method for generating an imaging system response function that reflects imaging conditions at the time of imaging, a correction process for applying a correction function for correcting the selected imaging system response function to an imaging system response function at the time of imaging There is.

  Here, a Gaussian function will be described as an example of the correction function. The two-dimensional Gaussian function G (x, y) can be expressed by the following equation.

Here, G (x, y) has a convex shape as shown in FIG. 9, A is the magnitude of the peak value, and Bx and By are the peak positions in the x and y directions, respectively. This is the amount of deviation from the origin. C is a coefficient for adjusting the spread of the convex shape.

  Then, correction processing is performed on the PSF selected in step S504 using the Gaussian function G (x, y). A Gaussian function G (x, y) is applied to the imaging system response function PSF_IN (x, y) acquired in step S504 to generate an imaging system response function PSF_OUT (x, y) at the time of imaging. The calculation formula is as follows.

Here, the right side of Expression (7) indicates that multiplication is performed between values at the same coordinate position of PSF_IN (x, y) and G (x, y). Thereby, the shape of PSF_IN (x, y) can be corrected according to the shape of G (x, y).

  In this embodiment, the case where attention is paid to the F value of the diaphragm as an imaging condition will be described, but the same applies to other imaging conditions. Here, as an example, the PSF_b shown in FIG. 7B is corrected using the Gaussian function G shown in FIG. 9, thereby generating a PSF equivalent to the PSF_a shown in FIG.

  FIG. 9 shows the shape of the Gaussian function G when A = 1, Bx = By = 0, and C = 0.3 in equation (6). Then, PSF_b in FIG. 7B is PSF_IN, and the shape of the PSF generated by Expression (7) using the Gaussian function G in FIG. 9 is shown as PSF_OUT in FIG.

  From FIG. 10, it can be confirmed that the PSF equivalent to the PSF_a shown in FIG. 7A can be generated by performing the correction process using the Gaussian function G.

  Here, as a method of determining the coefficients A, Bx, By, and C of the Gaussian function used in the correction process, for example, an imaging condition corresponding to the PSF selected in step S504 and an imaging condition at the time of actual imaging are used. There is a method of determining linearly from the difference. Furthermore, as another example of a method for determining each coefficient of the Gaussian function, there is a method of calculating or measuring in advance how to set a coefficient and generate a PSF corresponding to what imaging condition. is there. Each coefficient obtained by these methods may be stored as a look-up table or stored as a multidimensional function fitted to the coefficient.

  In the present embodiment, the method using the Gaussian function as the correction function has been described, but there is also a method using a sinc function or the like. Further, in equation (7), a method of integrating the PSF and the correction function for each pixel position is shown as a correction method, but the same effect can be obtained even by performing convolution integration.

  Furthermore, as another example of a PSF generation method corresponding to an imaging condition at the time of imaging, there is a method of applying a correction coefficient for correcting to a PSF at the time of imaging with respect to a selected PSF.

  For example, the PSF correction target includes a peak value, a half-value width, and the like. As an example for acquiring a correction coefficient for such a correction target, a method of performing measurement in advance is given.

  First, one or a plurality of PSFs under a certain imaging condition are selected and set as reference PSFs. Then, the degree of difference between the PSF under imaging conditions other than the reference PSF and the reference PSF is obtained. The degree of difference may be calculated by, for example, the difference between the peak value and the full width at half maximum, the enlargement ratio, and the like, and a value that cancels the degree of difference becomes the interpolation coefficient. Each coefficient obtained at this time may be stored as a lookup table or may be stored as a multidimensional function fitted to the coefficient. Thus, only the coefficients and the reference PSF need be stored in the imaging system response function storage unit 1102.

  Then, correction processing is performed on the reference PSF selected in step S504 using the correction coefficient corresponding to the imaging condition acquired in step S502, thereby generating a PSF corresponding to the imaging condition at the time of imaging. It becomes possible.

  As described above, a large number of imaging system response functions for each imaging condition are not stored, but an imaging system response function corresponding to discrete imaging conditions having a certain interval is stored, and interpolation processing is performed on them. By performing the correction processing, an imaging system response function corresponding to the imaging condition at the time of imaging can be generated. As a result, it is possible to generate a good high-definition image using the imaging system response function corresponding to the imaging conditions while reducing the storage capacity required for the imaging system response function storage unit 1102.

  Steps S506 and S507 in FIG. 5 are the same as steps S205 and S206 shown in FIG.

In each of the above embodiments, the case where the image processing apparatus 110 is built in the imaging apparatus has been described. However, the embodiments of the present invention are not limited thereto.

  For example, as shown in FIG. 11, the captured image obtained by the imaging device 1801 is transmitted to the personal computer 1802. The transmission method may be either a cable method or a wireless method, and may be transmitted via the Internet or a LAN.

  Then, the personal computer 1802 may perform the image region division processing and the high definition processing described in the first and second embodiments on the received captured image. In this case, the personal computer functions as an image processing apparatus.

  Further, the image processing program described in the first and second embodiments may be installed in an imaging apparatus or a personal computer via a storage medium such as a semiconductor memory or an optical disk storing the program, the Internet, or a LAN. .

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

It is possible to provide an image processing apparatus that can perform sufficiently high-definition processing of a captured image under various imaging conditions.

DESCRIPTION OF SYMBOLS 1,1 'Imaging apparatus 100 Imaging optical system 102 Imaging element 108 System control circuit 110 Image processing apparatus 1102 Imaging system response function memory | storage part 1103 Imaging system response function selection part 1104 High-definition image processing part 1105 Image area division part

Claims (6)

Area dividing means for dividing the input image generated by the imaging system into a plurality of areas having differences in evaluation values obtained by using the response function of the imaging system;
Processing means for performing image processing for increasing the image definition using the response function that is different for each region for each region;
The processing means performs the image processing using the response function corresponding to an imaging condition at the time of generation of the input image in the imaging system and read out from a storage means or generated by calculation. An image processing apparatus. The image processing apparatus according to claim 1, wherein the imaging condition is at least one of an aperture value, an imaging magnification, a focal length, and a wavelength of imaging light of the imaging system. The storage means stores the response function corresponding to a plurality of the imaging conditions,
When the imaging condition at the time of generating the input image is different from the plurality of imaging conditions, the processing means is used for the image processing by interpolation processing or correction processing using the response function stored in the storage means. The image processing apparatus according to claim 1, wherein the response function is generated.
apparatus. An imaging system;
An image pickup apparatus comprising: the image processing apparatus according to claim 1. On the computer,
Dividing an input image generated by the imaging system into a plurality of regions having differences in evaluation values obtained using the response function of the imaging system;
An image processing step for performing image processing for increasing the image definition using the response function that is different for each region for each region,
In the image processing step, the image processing is performed using the response function corresponding to the imaging condition at the time of generating the input image in the imaging system, which is read from a storage unit or generated by calculation. An image processing program characterized by that. A storage medium storing the image processing program according to claim 5.