JP2015198380A - Image processing system, imaging apparatus, image processing program, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform highly accurate image processing by using information on an image height where an image is focused.SOLUTION: The image processing system includes: a storage section 308 for storing information on an optical transfer function or a point spread function of an imaging apparatus for each photographing condition for each image height where an image is focused; and image processing means for processing an image by using information on an optical transfer function or a point spread function for which a photographing condition of the image photographed through the imaging apparatus and an image height where an image is focused coincide with the photographing condition stored in the storage section and an image height where the image is focused.

Description

  The present invention relates to an image processing device, an imaging device, an image processing program, and an image processing method.

  In a lens with a large field curvature, the imaging position in the optical axis direction differs between the center of the screen and the periphery of the screen. When captured, the resolving power at the periphery of the screen is lower than that at the center of the screen. Similarly, when the captured image is captured using the MTF peak position of the screen periphery as the imaging surface by focusing on the screen periphery, the resolution of the screen periphery is improved compared to the case of focusing on the screen center. The resolving power at the center is reduced.

  Therefore, Patent Document 1 proposes a method of acquiring an image at an intermediate position between an image forming position at the center of the screen and an image forming position at the periphery of the screen, and recovering the image with a recovery filter on the image acquisition image plane. Patent Document 2 proposes a method of dividing a screen region based on subject distance information and recovering it using an image restoration filter corresponding to each divided region.

JP 2011-124692 A JP 2011-2111663 A

  However, in Patent Document 1, there is a problem that the position in the screen originally intended by the user cannot be focused.

  The present invention illustratively provides an image processing device, an imaging device, an image processing program, and an image processing method capable of performing highly accurate image processing using information on the image height that is in focus. Objective.

  The image processing apparatus according to the present invention includes a storage unit that stores correction data for each image height focused on a shooting condition of a shooting optical system, and a focus on the shooting condition of an image captured through the shooting optical system. And image processing means for processing the image using correction data that matches the shooting conditions stored in the storage means and the focused image height. And

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing program, and an image processing method capable of performing highly accurate image processing using information on the image height that is in focus. .

It is the schematic for demonstrating the concept of this invention. It is the schematic for demonstrating the point image intensity distribution function of FIG. It is a block diagram of the imaging device of the present invention. 3 is a flowchart of the present invention. (Example 1) 3 is a flowchart of the present invention. (Example 2) 3 is a flowchart of the present invention. (Example 3)

  FIG. 1 is a schematic diagram for explaining the concept of the present invention, in which the horizontal axis represents the defocus direction, and the vertical axis represents the position on the central MTF peak image plane. FIG. 1A shows a case where the user selects a focus detection point at the center of the screen, and FIG. 1B shows a case where the user selects a focus detection point at the periphery of the screen. The focus detection point is, for example, an area that is displayed on a photographing screen of a display unit 305 such as a liquid crystal display on the back as shown in FIG. In general, a plurality of focus detection areas are displayed on the screen and are selected by a user via a selection unit (not shown).

  FIG. 2 is a schematic diagram for explaining the point spread function (PSF) of FIG. 1, wherein the horizontal axis S represents the sagittal direction and the vertical axis M represents the meridional direction.

  An image obtained by imaging a subject with an imaging device such as a digital camera includes blur due to spherical aberration, coma aberration, field curvature, astigmatism, and the like of the photographing optical system. This blur occurs when the light beam emitted from one point of the subject should be collected again at one point on the imaging surface when there is no aberration and there is no influence of diffraction. Is represented by Further, this blur is different from the blur due to the focus shift.

  The color blur in the color image can also be said to be a difference in the blurring method for each wavelength of light with respect to the axial chromatic aberration, the spherical spherical aberration, and the color coma aberration of the photographing optical system. Also, the lateral color misregistration can be referred to as a positional misalignment or a phase misalignment due to a difference in imaging magnification for each wavelength of light when the lateral chromatic aberration is caused by the chromatic aberration of magnification of the photographing optical system.

  Conventionally, a technique for recovering an image deteriorated due to aberration of an optical system (hereinafter referred to as an image recovery process) is known. As an image restoration process, there is a method of using optical system OTF (Optical Transfer Function) or PSF information of the optical system that is related to Fourier transform.

  The OTF has a real part and an imaginary part, and is generally stored in the storage means as two-dimensional data. In general image restoration processing, OTF data is prepared for each of RGB, so the OTF data of one image height is the number of taps in the x direction × the number of taps in the y direction × 2 (real part, imaginary part) × 3 ( Color component). Further, OTF and PSF differ depending on photographing conditions including the image height of the image formed through the optical system, the focal length of the optical system, the F value, and the subject distance.

  OTF is aberration frequency component information and is represented by a complex number. The absolute value of the OTF, that is, the amplitude component is referred to as MTF (Modulation Transfer Function), and the phase component is referred to as PTF (Phase Transfer Function). MTF and PTF are frequency characteristics of an amplitude component and a phase component of image degradation due to aberration, respectively.

  When the phase component is a phase angle, the following equation is established. Here, Re (OTF) and Im (OTF) are a real part and an imaginary part of the OTF, respectively.

PTF = tan ⁻¹ (Im (OTF) / Re (OTF)) (1)
Thus, since the OTF of the photographing optical system deteriorates the amplitude component and the phase component of the image, the deteriorated image is in a state where each point of the subject is asymmetrically blurred like coma aberration.

  Further, lateral chromatic aberration is caused by shifting the imaging position due to the difference in imaging magnification for each wavelength of light, and acquiring this as, for example, RGB color components in accordance with the spectral characteristics of the imaging device. The spread of the image not only occurs due to the shift of the image formation position between RGB, but also spreads due to the shift of the image formation position for each wavelength within each color component, that is, the phase shift. Therefore, when the PSF is viewed in a one-dimensional section in each direction (azimuth direction) orthogonal to the principal ray (the ray passing through the center of the pupil of the imaging optical system), the phase deterioration component of the aberration generates asymmetry in the PSF. Let Further, the amplitude deterioration component affects the size of the PSF spread for each azimuth direction.

  The coordinate on the image in the real space is (x, y), the original image before being deteriorated by the optical system is f (x, y), the PSF is h (x, y), and the deteriorated image (input image) is g. Assuming (x, y), the following equation is established. However, * is convolution.

g (x, y) = h (x, y) * f (x, y) (2)
When Fourier transform is applied to Equation 2 to convert from real space (x, y) to frequency space (u, v), the following equation is established. Here, F (u, v) is the Fourier transform of f (x, y), G (u, v) is the Fourier transform of g (x, y), and H (u, v) is h (x, y). Fourier transform of OTF.

G (u, v) = H (u, v) · F (u, v) (3)
In order to obtain the original image from the degraded image, both sides are divided by H.

G (u, v) / H (u, v) = F (u, v) (4)
A restored image corresponding to the original image f (x, y) is obtained by performing inverse Fourier transform on F (u, v) and returning it to the actual surface. Here, ^assuming that the result of inverse Fourier transform of H ⁻¹ is R, the original image can be similarly obtained by performing convolution processing on the actual image as in the following equation.

g (x, y) * R (x, y) = f (x, y) (5)
R (x, y) is referred to as an image restoration filter. When the image is two-dimensional, the image restoration filter is usually a two-dimensional filter having taps (cells) corresponding to each pixel of the image. Generally, the recovery accuracy improves as the number of taps of the image recovery filter increases. Since the image restoration filter needs to reflect at least aberration characteristics, it is separated from the conventional edge enhancement filter (high-pass filter) of about 3 taps in the horizontal / vertical direction. Since the image restoration filter is created based on the OTF, both the amplitude component and the phase component can be corrected with high accuracy.

  FIG. 1A shows a case where a focus detection point at the center of the screen is selected and an image is captured on the central MTF peak image plane. 2A shows the PSF at the center of the screen indicated by A in FIG. 1A, and FIG. 2B shows the PSF at the periphery of the screen indicated by B in FIG. 1A.

  FIG. 1B shows a case where a focus detection point at the peripheral portion of the screen is selected and an image is captured on the peripheral MTF peak image plane. 2C is a PSF at the center of the screen indicated by C in FIG. 1B, and FIG. 2D is a PSF at the periphery of the screen indicated by D in FIG. 1B.

  When the focus detection point is located at the center of the screen, the image is captured in the shooting state shown in FIG. 1A, and the image is accurately obtained using the corresponding PSF shown in FIGS. 2A and 2B. Can be recovered.

  However, in recent years, since the focus detection point has been expanded to the periphery of the screen, the opportunity to capture an image in the shooting state shown in FIG. When conventional image restoration is performed on this, the PSF shown in FIGS. 2C and 2D is corrected by image processing using the PSF shown in FIGS. 2A and 2B. As a result, correction is performed using a PSF that is narrower than the actual PSF at the center of the screen, and correction is performed using a PSF that is wider than the actual PSF at the periphery of the image. descend.

  In the present embodiment, correction data used for image processing, for example, OTF or PSF information is changed according to the focus detection point (focused image height) selected by the user, thereby realizing high-accuracy image processing. To do. When a large amount of correction data is held in the defocus direction, the mounting load increases. Therefore, the correction data is used within the field curvature range, that is, the range where the field curvature amount is equal to or greater than the threshold value. In the case of a photographing optical system in which the amount of field curvature is less than a threshold value, the effect of changes in correction data based on the selection of a focus detection point is small, so that image processing accuracy is maintained without applying this embodiment. be able to.

  Another method for correcting image degradation due to the photographing optical system with high accuracy by image processing will be described below.

  First, the difference between the blurred image data and the original input image is obtained by applying an unsharp mask to the input image, and the difference data is added to or subtracted from the original input image to sharpen the image. There is an unsharp mask process to be realized. A filter for blurring an image such as a smoothing filter is used for the unsharp mask. The larger the difference between the blurred image and the input image, the sharper the image.

  Since the unsharp mask process uses a rotationally symmetric filter for the unsharp mask, a sharpened image cannot be sharpened correctly due to the influence of a PSF having a complicated shape such as asymmetric aberration or sagittal halo. As a result, if an attempt is made to correct an aberration in the azimuth direction where a large amount of aberration is generated, an undershoot occurs in the azimuth direction where the aberration is small. Conversely, if undershoot is suppressed, the aberration cannot be corrected sufficiently.

  Conventional unsharp mask processing cannot improve asymmetry in directions other than the image height direction, which is the meridional azimuth direction. Further, the asymmetry of the filter is adjusted by the number of minus tap coefficients, and the correction in the image height direction is different from the method of blurring the PSF of the optical system, so that it cannot be sufficiently sharpened.

  Therefore, the present embodiment provides highly accurate unsharp mask processing by using a PSF corresponding to the imaging conditions and the focused image height for the unsharp mask.

  As another image processing method, there is color blur processing for reducing color blur on a color image. Due to chromatic aberration of the photographing optical system, a color (color blur) that does not originally exist around a bright portion of an image may occur. In visible light imaging, color blur is likely to occur in a portion away from green, which is the central wavelength of the imaging optical system, and purple or blue artifacts in which blue, red, or both are mixed are blurred and are also called purple fringes.

  The chromatic aberration of the photographic optical system can be reduced to some extent by combining multiple lenses with different dispersions. However, due to the recent demands for higher resolution image sensors and smaller photographic optical systems, only the photographic optical system is sufficient. It is difficult to keep it down. Therefore, suppression of artifacts by image processing is required.

  Chromatic aberration is divided into lateral chromatic aberration (magnification chromatic aberration) and longitudinal chromatic aberration (axial chromatic aberration). Lateral chromatic aberration is a phenomenon in which the imaging position shifts in the direction along the image plane depending on the wavelength, and longitudinal chromatic aberration is a phenomenon in which the imaging position shifts in the direction along the optical axis due to the wavelength.

  In the case of a primary color digital imaging system, lateral chromatic aberration can be corrected by geometric transformation that adds different distortions to the R (red), G (green), and B (blue) color planes.

  In contrast to the image focused on the G plane that bears the center wavelength of the visible light region, the longitudinal chromatic aberration is unfocused on the subject on the R and B planes at the end of the visible light region, resulting in a blurred image (out-of-focus image). It cannot be corrected by geometric transformation such as lateral chromatic aberration. Color blur processing uses characteristics that occur mainly around whiteout (preset signal saturation area), searches for a saturated area in the G plane, integrates the pixels in the surrounding area, and corrects the amount Is calculated and corrected.

  Although color blurring occurs largely in the vicinity of whiteout, a color blur that is uncomfortable for an observer also occurs in an area where whiteout does not occur. An area in which the pixel value of the color plane in the photographed color image is monotonously increasing or monotonously decreasing may be determined as a color blur generation area and the color blur may be removed.

  Since color blur occurs due to aberrations of the photographing optical system, in a photographing optical system having asymmetric aberrations such as coma aberration, there are directions in which color blur occurs and does not occur with respect to the subject.

  When the color of the subject is similar to the color blur, there is a problem that the subject color is erroneously determined as a color blur even in a direction where the color blur is not originally generated, and the original color of the subject is removed.

  Therefore, in the present embodiment, when performing color fringing correction processing of a color image, the original color of the subject is determined by using the monotonic increase or monotonic decrease determination of the pixel value of each color plane and the information on the direction of color fringing of the photographing optical system. It is also possible to correct the color fringing process by reducing the harmful effects to be removed.

  In color blur processing, the amount of correction for reducing color blur is estimated for each area, the multiple correction amounts for the estimated areas are smoothed, and the smoothed correction amount is used to reduce color blur. Image processing may be performed. Here, the estimated correction amount is for correcting the estimated blur amount, and the estimated blur amount differs depending on the photographing condition and the focus detection point (focused image height) selected by the user. To do. For this reason, the estimated correction amount also differs depending on the image height in focus with the imaging conditions.

  The image processing method of each embodiment of the present invention will be described below. In the flowchart showing the image processing method of each embodiment, “S” represents a step. The image processing method of each embodiment is executed by a dedicated image processing apparatus, the imaging apparatus shown in FIG. 3, a personal computer (PC), a server on a network, or the like.

  The image processing apparatus has storage means and image processing means. The storage means stores correction data (OTF or PSF information of the photographing optical system and a correction amount for reducing color blur) for each image height focused on the photographing condition. The image processing means obtains correction data in which the image height in focus with the shooting conditions of the image taken via the shooting optical system matches the shooting conditions stored in the storage means and the focused image height. Use to process images.

  The image processing means may perform image restoration processing using h (x, y) or H (u, v) in Equations 2 and 3, or unsharp mask processing using the acquired PSF as an unsharp mask. May be performed. Alternatively, the image processing unit may reduce the color blur using the estimated correction amount (after smoothing).

  The image processing means can be realized by a computer (processor). In this case, each flowchart can be embodied as a program (image processing program) for causing a computer to realize the function of each step.

  FIG. 3 is a block diagram of an imaging apparatus that executes the image processing method of the present embodiment. The imaging device includes a digital still camera, a digital video camera, a digital microscope, an endoscope, and the like. The imaging apparatus includes an imaging optical system 301, an imaging element 302, an A / D converter 303, an image processing unit 304, a display unit 305, an imaging optical system control unit 306, a state detection unit 307, a storage unit 308, an image recording medium 309, A system controller 310 and selection means 311 are included.

  The imaging optical system 301 forms an optical image of a subject on the imaging surface. The image sensor 302 photoelectrically converts the optical image formed by the imaging optical system 301. The A / D converter 303 converts an analog electric signal output from the image sensor 302 into a digital signal.

  The image processing unit 304 applies the image processing method of the present embodiment to a digital signal, and includes an acquisition unit 304a and a correction unit 304b. The acquisition unit 304a acquires information acquired by the state detection unit 307. The correction unit 304b acquires correction data corresponding to the information acquired by the acquisition unit 304a from the storage unit 308, and performs image processing using the correction data. The image processing unit 304 has an image height that is in focus with the image capturing condition of the image captured through the image capturing optical system 301 and the image capturing condition that is stored in the storage unit 308 and the image height that is in focus. It functions as an image processing means for processing an image using the correction data.

  When the image height that is in focus with the image capturing condition of the image captured through the image capturing optical system does not match the image capturing condition stored in the storage unit 308 and the image height that is in focus, the image processing unit 304 Corresponding correction data is generated by interpolation.

  The display unit 305 is a liquid crystal display that displays the image processed by the image processing unit 304. The display unit 305 displays a plurality of focus detection points on the screen, and the user selects a focus detection point to be focused on by using the selection unit 311. The plurality of focus detection points are arranged on the screen so as to have different image heights and positions.

  The photographing optical system control unit 306 controls driving of a focus lens, a diaphragm, a zoom lens, and an image blur correction lens (not shown) of the photographing optical system 301 based on a command from the system controller 310.

  The state detection unit 307 receives shooting state information (focal length, F value, subject distance, etc.) from the shooting optical system control unit 306. The storage unit 308 functions as a storage unit that stores correction data used for image processing for each image height focused on the shooting condition.

  The image recording medium 309 records the image processed by the image processing unit 304. The system controller 310 controls the operation of each unit of the imaging apparatus, and instructs the imaging optical system control unit 306, for example, the aperture, zoom position, focus position, and the like.

  If the focused image height of the image captured through the imaging optical system does not match the focused image height stored in the storage unit 308, the image processing means interpolates the correction data by interpolation. It may be generated.

  FIG. 4 shows a flowchart of the first embodiment. In this embodiment, image restoration processing is used as image processing.

  First, the image processing means acquires an input image (S10). The input image may be acquired before the image processing is performed using the correction data, and the order is not limited. Next, the image processing means acquires shooting information and focusing point information of the shooting optical system at the time of shooting that acquired the input image (S12). The in-focus location information is information indicating where the focus detection point is in the screen (image height in focus). Next, the image processing means acquires correction data corresponding to the acquired photographing information and in-focus position information from the database of the storage means (S14). Next, the image processing means performs an image restoration process on the input image using the acquired correction data (S16). The filter used for the image restoration process may be created using an OTF of a photographing optical system as disclosed in Patent Document 2.

  FIG. 5 shows a flowchart of the second embodiment. In this embodiment, edge enhancement processing is used as image processing. Although the edge enhancement process is not as effective as the image restoration process, the amount of data required for storage is smaller than that of the image restoration process.

  First, the image processing means acquires an input image (S20). The input image may be acquired before image processing is performed using the correction data. Next, the image processing means acquires shooting information and focus location information of the shooting optical system at the time of shooting when the input image is acquired (S22). Next, the image processing means acquires correction data corresponding to the acquired photographing information and in-focus location information from the database of the storage means (S24). Next, the image processing means performs edge enhancement processing on the input image using the acquired correction data (S26).

  FIG. 6 shows a flowchart of the third embodiment. In this embodiment, color blur processing is used as image processing.

  First, the image processing means presses the shutter (S30). Next, the image processing means acquires shooting information and focus location information of the shooting optical system at the time of shooting (S32). Next, the image processing means acquires correction data corresponding to the acquired photographing information and in-focus location information from the database of the storage means (S34). Next, the image processing means acquires an input image (S36). Next, the image processing means performs a color blur process on the input image using the acquired correction data (S38).

  As mentioned above, although the present Example was described, this invention is not limited to a present Example, A various deformation | transformation and change are possible within the range of the summary.

  The image processing apparatus of the present invention can be applied to a use for recovering an image taken by an imaging apparatus such as a camera.

304: Image processing unit (image processing unit), 308: Storage unit (storage unit)

Claims (10)

Storage means for storing correction data for each image height in focus with the shooting conditions of the shooting optical system;
Use correction data in which the image height in focus with the shooting conditions of the image taken through the shooting optical system matches the shooting conditions and the in-focus image height stored in the storage means Image processing means for processing the image;
An image processing apparatus comprising:   When the image height in focus with the shooting condition of the image taken through the shooting optical system does not match the shooting condition stored in the storage unit and the focused image height, the image The image processing apparatus according to claim 1, wherein the processing unit generates correction data corresponding to an image height in focus with a shooting condition of an image taken through the shooting optical system by interpolation. .   When the field curvature amount of the photographing optical system is greater than or equal to a threshold, the image processing means processes the image, and when the field curvature amount of the photographing optical system is less than the threshold, the image processing means The image processing apparatus according to claim 1, wherein the image is not processed.   The image processing apparatus according to claim 1, wherein the photographing condition includes a focal length, an F value, and a subject distance. The correction data includes information on the optical transfer function or point spread function of the photographing optical system,
5. The image processing apparatus according to claim 1, wherein the image processing unit performs image restoration processing using the optical transfer function or the point spread function. The correction data includes information on the optical transfer function or point spread function of the photographing optical system,
5. The image processing apparatus according to claim 1, wherein the image processing unit performs an unsharp mask process using the point image intensity distribution function as an unsharp mask. 6. The image captured through the imaging optical system is a color image,
The correction data includes a correction amount for reducing color bleeding on the color image,
5. The image processing apparatus according to claim 1, wherein the image processing unit estimates the correction amount, and reduces color blur based on the estimated correction amount. 6. A photographing optical system for forming an optical image of a subject;
A selection means for selecting an image height to be focused;
Storage means for storing correction data for each image height in focus with the imaging conditions of the imaging optical system;
Correction that the imaging condition of the image captured through the imaging optical system and the image height selected by the selection unit match the imaging condition stored in the storage unit and the focused image height Image processing means for processing the image using data;
An imaging device comprising: In the storage means, correction data is stored for each image height in focus with the shooting conditions of the shooting optical system,
Use correction data in which the image height in focus with the shooting conditions of the image taken through the shooting optical system matches the shooting conditions and the in-focus image height stored in the storage means And processing the image. Computer
Means for storing correction data for each image height focused on the photographing conditions of the photographing optical system in the storage means;
Use correction data in which the image height in focus with the shooting conditions of the image taken through the shooting optical system matches the shooting conditions and the in-focus image height stored in the storage means Means for processing the image;
Image processing program to function as