JP5501069B2 - Image processing apparatus, imaging apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

  Patent Document 1 discloses an image processing method for restoring (recovering) an image captured by an imaging optical system based on a point spread function (PSF) of the imaging optical system. Patent Document 2 discloses an image processing method for selecting an optimum restoration filter for image restoration from a plurality of restoration filters.

Japanese Patent Laid-Open No. 62-127976 JP 2008-2111679 A

  Consider a case where the imaging screen includes a first subject at a focus position and a second subject at a defocus position that is within the depth of field but slightly deviated from the focus position. Then, in the conventional image restoration, since the restoration filter used for image restoration of the first subject is also used for image restoration of the second subject, the image of the second subject may not be restored well. It was. In general, if the PSF of the area to be restored (hereinafter referred to as “restoration area”) differs from the PSF actually used in the restoration area, the restored image may be deteriorated.

  On the other hand, according to Patent Document 2, an optimum filter can be obtained. However, Patent Document 2 performs a filtering process a plurality of times in order to select the optimum filter, and as a result, the image restoration time becomes longer. There was a problem. In particular, when the tap of the image restoration filter becomes large, the image restoration time becomes too long, and the image restoration disclosed in Patent Document 2 becomes unrealistic.

  Accordingly, an object of the present invention is to provide an image processing apparatus, an imaging apparatus, an image processing method, and a program that can perform image restoration in a short time and in a favorable manner.

  The image processing apparatus according to the present invention includes a plurality of restoration regions, each of which is a region to be restored, and a plurality of focus detection regions, each of which is a unit region to be subjected to focus detection. A deteriorated image acquisition unit that acquires a deteriorated image obtained by imaging with an imaging optical system focused on one of the focus detection areas, and a subject distance for one of the plurality of focus detection areas. Acquisition of information on the imaging state when the deteriorated image is captured, including the subject distance of each of the plurality of restoration regions obtained from the focal subject distance and the subject distance obtained for each of the plurality of focus detection regions An imaging state information acquisition unit, an optical transfer function acquisition unit that acquires an amount corresponding to an optical transfer function of the imaging optical system corresponding to the imaging state for each restoration region, and the degraded image It characterized by having a a restoration processing unit for restoring the deteriorated image in each restoration area using an amount corresponding to the optical transfer function.

  According to the present invention, it is possible to provide an image processing apparatus, an imaging apparatus, an image processing method, and a program that can perform image restoration in a short time and in a satisfactory manner.

FIG. 1 is a block diagram of the imaging apparatus of the present embodiment. FIG. 2 is a diagram of a screen having a plurality of restoration areas and a plurality of focus detection areas according to the present embodiment. FIG. 3 is a diagram showing an example of the restoration filter DB stored in the memory shown in FIG. FIG. 4 is a flowchart for explaining the image processing method of this embodiment. FIG. 5 is a diagram showing a modification of FIG.

  FIG. 1 is a block diagram of the image pickup apparatus of the present embodiment. The imaging apparatus includes an imaging optical system (imaging lens) 10, a system controller 15, an imaging optical system control unit 20, a subject distance acquisition unit, an imaging element 41, an image processing unit 50, a recording medium 60, and other units.

  The image pickup apparatus may be of any type such as a digital camera or a digital video camera. In this embodiment, the image pickup apparatus is a digital single-lens reflex camera capable of observing a subject with a finder (not shown). In addition, the imaging device includes a lens device (not shown) that houses the imaging optical system 10 and a camera body that houses components other than the imaging optical system 10 of FIG. 1, and the lens device is configured to be detachable from the camera body. Has been.

  In this embodiment, each pixel of the image has a signal value for each color component as R (red), G (green), and B (blue). However, the image processing method of the present invention uses two or more colors. The present invention can be applied as long as it forms an image made of

  The imaging optical system 10 forms an optical image of the subject. However, since the imaging optical system 10 has aberrations such as axial chromatic aberration, chromatic spherical aberration, and chromatic coma, the optical image of the subject is deteriorated due to the aberration. The imaging optical system 10 includes a lens 11, a focus lens 12 that moves during focus adjustment (focusing), and a diaphragm 13 for adjusting the amount of light. The lens 11 includes a lens other than the focus lens, such as a variable power lens (zoom lens) that moves upon zooming, and a fixed lens on the front side. In FIG. 1, OA indicated by a one-dot chain line is an optical axis.

  The variable power lens and the focus lens 12 are configured to be movable along the optical axis direction by driving means (not shown) such as a step motor, and the driving means is controlled by the imaging optical system control means 20. In FIG. 1, the lens 11 and the focus lens 12 are simplified as a single lens, but are actually composed of a plurality of lenses.

  The system controller 15 is connected to the imaging optical system control unit 20, the subject distance acquisition unit, the image processing unit 50, the recording medium 60, and the like, functions as a control unit that controls the operation of each unit of the imaging apparatus, and is a microcomputer (processor). Consists of The system controller 15 controls an image processing method executed by an image processing unit 50 described later.

  The imaging optical system control unit 20 includes a focus lens control unit 21, a variable magnification lens control unit 22, and an aperture control unit 23.

  The focus lens control unit 21 controls the operation (drive timing, drive amount, drive direction) of a drive unit (not shown) that drives the focus lens 12 in the optical axis direction. The zoom lens control unit 22 controls the operation of a driving unit (not shown) that drives the zoom lens in the optical axis direction. The aperture controller 23 controls the aperture diameter of the aperture 13.

  The subject distance acquisition unit acquires subject distances of a plurality of focus detection areas set on the screen. The subject distance acquisition unit includes a sub mirror 31, a focus detection unit 32, a focus detection information holding unit 34, and a subject distance calculation unit 35.

  The sub mirror 31 is disposed on the optical path that has passed through the imaging optical system 10 and reflects the light beam. FIG. 1 shows a state during non-imaging, and during imaging, the sub mirror 31 is retracted from the optical path and guides the light flux to the image sensor 41.

  In this embodiment, focus detection is performed using a phase difference detection method. The focus detection unit 32 calculates a pupil division optical system that separates the light beam from the sub-mirror 31 into two light beams, a pair of area sensors that capture the two separated images, and a positional deviation amount (phase difference) between the two images. And a phase difference calculation unit.

  FIG. 2 is a diagram of a screen on which a plurality of restoration areas and a plurality of focus detection areas are arranged according to the present embodiment. Each of the plurality of restoration areas indicates a unit area to be restored by the image processing unit 50 (image restoration). In this embodiment, as an example, the screen is divided into a plurality of rectangular restoration areas 201 to 235 of 5 columns and 7 rows.

  A plurality of focus detection areas (AF ranging areas) are unit areas to be focus-detected, and when one of the plurality of focus detection areas is automatically or manually selected to match the main subject, Focus adjustment is performed so that the focus detection area is in focus. In this embodiment, as an example, a plurality of rectangular focus detection areas 101 to 135 having 5 columns and 7 rows are set on the screen.

  In this embodiment, as shown in FIG. 2, one focus detection area is assigned to each restoration area, and the number of focus detection areas 101 to 135 is equal to the number of restoration areas 201 to 235.

  During AF, the area sensor of the focus detection unit 32 detects image signals of the plurality of focus detection regions 101 to 135, and the phase difference calculation unit of the focus detection unit 32 performs correlation calculation of the two image signals to detect a plurality of focus points. The phase difference between the areas 101 to 135 is obtained.

  Next, a focus detection area to be focused is selected from among the plurality of focus detection areas 101 to 135. In the manual selection mode, the focus detection area selected by the photographer is selected, and in the automatic selection mode, the focus detection area determined as the main subject is selected. Next, the focus detection unit 32 sends the phase difference of the selected focus adjustment region to the focus lens control unit 21.

  In response to this, the focus lens control unit 21 (or the system controller 15) moves the focus lens 12 to obtain a movement amount necessary for obtaining focus from the phase difference and the position of the focus lens 12 at the time of focus detection. Adjust the focus.

  The focus detection information holding unit 34 acquires and holds phase differences (focus detection information) of the plurality of focus detection regions 101 to 135 as detection results from the focus detection unit 32. The focus detection information holding unit 34 acquires and holds information about the position of the focus lens 12 at the time of focus detection and in-focus from the focus lens control unit 21.

  The subject distance calculation unit 35 is a distance (subject distance) from the phase difference between the plurality of focus detection areas 101 to 135 and the position of the focus lens 12 at the time of focus detection and focusing to the subject at the focus position (focal plane). And subject distance in each focus detection area. Since the position of the focus lens 12 at the time of focusing is known when the focus lens control unit 21 calculates the movement amount, the subject distance calculation unit 35 may perform this calculation while the focus lens 12 is moving.

  The low-pass filter (LPF) 40 is disposed in front of the image sensor 41 and guides the light flux via the image pickup optical system 10 to the image sensor 41. The LPF 40 suppresses aliasing components due to spatial sampling of the image sensor 41.

  The image sensor 41 is a CCD or CMOS that photoelectrically converts an optical image of a subject into an analog electrical signal. The A / D converter 42 converts the analog signal converted by the image sensor 41 into a digital signal.

  At the time of shooting, the sub mirror 31 is retracted from the optical path, and the light flux from the subject forms an image on the image sensor 41 via the image pickup optical system 10 and the LPF 40. The image sensor 41 is provided with a color mosaic filter composed of RGB, and can acquire signals of RGB color components.

  The image processing unit 50 performs predetermined processing including image restoration, and performs buffer memory 51, imaging state information acquisition unit 52, filter selection unit 53, memory 54, restoration filter interpolation generation unit 55, restoration processing unit 56, and post-processing unit. 58.

  The buffer memory 51 converts a deteriorated image (RAW image data) before image restoration in which an original image (image after image restoration) is deteriorated due to aberration of the imaging optical system 10 into an A / D converter. 42 is a deteriorated image acquisition unit that acquires and holds from 42.

  The imaging state information acquisition unit 52 acquires information on the imaging state of the imaging device when the deteriorated image is acquired. The imaging state information acquisition unit 52 acquires the imaging state of the imaging device from the system controller 15 or a lens controller (not shown) of the lens device.

  The imaging state includes imaging system design data (preferably with manufacturing error information), zoom position (focal length, zoom lens position), focus distance (distance to the in-focus subject, focus) Distance or focus lens position) and aperture value (F number, aperture diameter). The imaging state further includes the shooting distance (including the distance to the subject in the entire shooting region, and of course, the distance to the subject other than the focused subject), ISO sensitivity, white balance, and the like. The design data of the imaging system includes the design data of the imaging optical system 10, the size and shape of the diaphragm 13, the spatial frequency characteristics of the LPF 40, the pixel pitch and pixel aperture shape of the imaging device 41, the arrangement of color mosaic filters on the imaging device 41, and the like. It is. The imaging state includes the in-focus subject distance and the subject distances of the plurality of restoration regions 201 to 235 obtained from the plurality of focus detection regions 101 to 135.

  Note that the actual imaging state can be selected as necessary. In general, when restoring the blur of the captured image, more information on the imaging state is used, and when restoring the image based on aberration information obtained from the design data of the imaging system, ISO sensitivity and white balance are used. Is not used.

  The filter selection unit 53 selects one of a plurality of restoration filters held in the memory 54 for each light in each of the RGB wavelength ranges. Thus, in this embodiment, the restoration filter is used for image restoration.

  However, the present invention uses an optical transfer function (OTF) or an amount corresponding thereto (PSF, other aberration information of the imaging optical system 10, an OTF in a broad sense to be described later), and other functions in image restoration. In this case, the filter selection unit 53 functions as an optical transfer function acquisition unit, and OTF or the like is stored in advance in the restoration filter DB 54 or a memory (not shown) as an optical design value corresponding to the imaging state.

  The memory 54 stores a plurality of restoration filters (digital filters) used for image restoration and other information. In this embodiment, the system controller 15 creates three restoration filters corresponding to the RGB color components for the RGB color image. This is because the imaging optical system 10 has chromatic aberration, and the blur method differs for each color component. For this reason, the restoration filter for each color component has different characteristics (cross section) based on chromatic aberration.

  In order to create a restoration filter for each color component of RGB, the transmittance characteristic for each wavelength band is considered. This corresponds to the PSF weighting coefficient for each wavelength band, and includes the spectral transmittance characteristic of the imaging optical system 10, the spectral transmission characteristic of the color mosaic filter on the imaging element 41, the spectral sensitivity characteristic of the imaging element 41, and the like. When an infrared cut filter (not shown) is used, its spectral transmission characteristics are also taken into consideration.

  In this embodiment, the memory 54 discretely holds a plurality of restoration filters, and the restoration filter interpolation generation unit 55 generates a restoration filter corresponding to the intermediate value. As a result, the capacity of the memory 54 can be reduced.

  In the present embodiment, the memory 54 holds a restoration filter DB 54a. FIG. 3 shows an example of the restoration filter DB 54a.

  In FIG. 3A, the first row identifies the interchangeable imaging optical system (photographing lens) 10, the second row identifies the zoom position that is the position of the zoom lens, and the third row is the aperture 13. The F number is identified, and the fourth column identifies the restoration filter list number. In FIG. 3A, for convenience, the number of imaging states is reduced, and the actual restoration filter list is subdivided to include more imaging state parameters.

  FIG. 3B is a diagram showing a plurality of restoration filter tables included in the filter list in the fourth column in FIG. 3A (here, the filter list number is “lens A variable magnification 1 stop F2” as an example). It is. Each filter list includes a plurality of restoration filter tables, and the plurality of restoration filter tables are classified according to the in-focus subject distance. The in-focus subject distance is the distance to the subject at the focus position (focal plane).

  Each restoration filter table holds a plurality of restoration filters (specifically, determinants of filter coefficients constituting the restoration filter), and the plurality of restoration filters depend on the image height (restoration areas 201 to 235) and the subject distance. It is classified. This is because the restoration filter corresponding to each restoration area 201 to 235 is used because the PSF changes according to the angle of view (image height) of the imaging optical system 10 even in one photographing state.

  In image restoration, if the degraded image is g (x, y), the original image is f (x, y), and the point spread function (PSF) is h (x, y), the following relationship is established. Here, * is convolution (convolution), and (x, y) is coordinates on the image.

  When Formula 1 is Fourier transformed, it becomes a product format for each frequency as shown in the following formula. Here, H is OTF, and (u, v) is a coordinate on a two-dimensional frequency plane, that is, a frequency.

  An original image f (x, y) is obtained as a restored image by dividing both sides of Formula 2 by H and performing an inverse Fourier transform on F (u, v) to return to the real surface.

Here, when R is the result of inverse Fourier transform of H ⁻¹ , the following equation is established. For this reason, an original image can be similarly obtained by performing convolution on the actual image.

  R (x, y) is a restoration filter. A Wiener filter may be used as a restoration filter in consideration of a noise component included in an actual image. As a method for correcting deterioration of the color blur component of the image, there is a method in which the blur amount for each color component of the image is made uniform by correcting the blur component. Since the OTF varies depending on the photographing state such as the zoom position and the aperture diameter, the restoration filter is also changed accordingly.

  The OTF can include a factor that degrades the OTF with respect to an image input to the image processing unit. For example, the low-pass filter suppresses high-frequency components with respect to the frequency characteristics of the OTF, and the shape and aperture ratio of the pixel aperture of the image sensor 41, the spectral characteristics of the light source, and the spectral characteristics of various wavelength filters also affect the frequency characteristics. In this embodiment, a restoration filter is created based on a broad OTF including these.

  The restoration filter interpolation generation unit 55 performs an interpolation operation on a restoration filter of parameters (magnification position, aperture state, in-focus subject distance, subject distance information of each area) which are not held in the restoration filter DB 54 from the neighboring restoration filters. To generate. The restoration filter interpolation generation unit 55 is effective in reducing the capacity of the restoration filter of the restoration filter DB 54.

  In this embodiment, the restoration processing unit 56 restores an image using the selected restoration filter. The restoration filter may be a two-dimensional filter obtained by inverse Fourier transform of a function generated based on the inverse function of OTF. In this case, the restoration processing unit 56 convolves a restoration filter with respect to the deteriorated image.

  The restoration filter can determine the number of taps according to the aberration amount of the imaging optical system 10, and each tap is convoluted by image restoration corresponding to one pixel of the image. By convolving the restoration filter in the real space with respect to the input image, the image can be recovered without performing Fourier transform in the imaging apparatus. Note that the number of taps in the vertical and horizontal directions of the restoration filter need not be square.

  However, since the restoration filter may not always be used in the image restoration of the present invention, the restoration processing unit 56 determines the degraded image for each wavelength region based on the relationship between the degraded image and the amount corresponding to the OTF and the original image. It is sufficient to restore the original image. In addition, when using OTF of a broad sense for OTF, it may become a relationship different from the relationship shown to Numerical formula 1-3.

  For example, when the restoration processing unit 56 uses a function generated based on the inverse function of OTF instead of the restoration filter, the restoration processing unit 56 may deconvolve the Fourier transform of the degraded image in the frequency space.

  The restoration processing unit 56 has a first subject and a second subject in the screen, and when the first subject acquires an image at the focus position for light in each of the RGB wavelength ranges, The OTF of each light is used for image restoration of the first subject and the second subject. The effect of the present embodiment is particularly remarkable when the subject distances of the first subject and the second subject are different.

  An image composition unit is provided as necessary. The image combining unit generates a color image by combining a plurality of recovered original images in the RGB wavelength region.

  The post-processing unit 58 performs image processing (development processing) such as demosaicing processing, white balance processing, noise reduction processing, and γ processing on the restored RAW image data.

  The recording medium 60 is a memory card that can be attached to and detached from the camera body that stores the restored image. When the image processing apparatus is a PC or the like, the storage medium is a hard disk drive (HDD), a semiconductor memory, various storage disks, or the like.

  FIG. 4 is a flowchart for explaining an image processing method executed by the system controller 15, and “S” is an abbreviation for a step. The image processing method of the present embodiment can be embodied as a computer-executable program.

  First, the system controller 15 acquires the focused subject distance and subject distances of the plurality of focus detection areas 101 to 135 from the subject distance calculation unit 35 using the imaging state acquisition unit 52. In the present embodiment, as shown in FIG. 2, the focus detection area and the restoration area correspond one-to-one, so that the imaging state acquisition unit 52 causes the in-focus subject distance and the plurality of restoration areas 201 to 235 to be matched. The information of the imaging state including each subject distance can be acquired (S1). Other imaging information is as described above.

  Next, the system controller 15 confirms that the buffer memory (degraded image acquisition unit) 51 has acquired a deteriorated image (RAW image data) (S2). In addition, the time ahead of S1 and S2 is not ask | required.

  Next, the system controller 15 acquires a restoration filter for each restoration area (S3). Therefore, the system controller 15 searches for and acquires one of a plurality of restoration filters held in the memory 54 using the filter selection unit 53 according to the imaging state. Alternatively, the system controller 15 corrects (interpolates) the restoration filter selected using the restoration filter interpolation generation unit 55 to generate a restoration filter.

  Here, for the sake of simplicity, an example will be described in which the filter selection unit 53 searches the restoration filter DB 54a illustrated in FIG. 3 and acquires the restoration filter. The system controller 15 selects the restoration filter list shown in FIG. 3A from the type, magnification position, and F number of the imaging optical system 10 included in the imaging information, and then responds based on the focused subject distance. The restoration filter table shown in FIG. 3B is selected.

  Next, the system controller 15 uses the restoration processing unit 56 to select a restoration filter corresponding to each subject distance in the plurality of restoration areas 201 to 235 from the selected restoration filter table, and performs image restoration ( S4).

  Consider a case where a first subject at a focus position and a second subject at a defocus position within the depth of field but slightly deviated from the focus position are present on the imaging screen. In the conventional image restoration, the restoration filter used for image restoration of the first subject is also used for image restoration of the second subject. In this embodiment, a restoration filter suitable for each restoration area is used.

  In this embodiment, if the restoration area is in focus, the restoration area is restored by a restoration filter based on the PSF of the focused subject distance. If the restoration area is defocused, the restoration area is restored by a restoration filter based on the PSF of the corresponding subject distance. . As a result, this embodiment can provide good image restoration. Further, since the present embodiment uses the focus detection information, the restoration process is only required once, and the image restoration can be performed in a short time.

  Thereafter, if necessary, the image synthesis unit synthesizes the images for the respective colors, the post-processing unit 58 performs development processing on the restored original image, and the recording medium 60 stores the original image.

  In this embodiment, a color mosaic filter is arranged on the image sensor 41 to acquire RGB color component signals. However, the color separation optical system separates the light fluxes into the RGB color components, and separates the light fluxes into the light fluxes. On the other hand, an RGB image sensor may be arranged to acquire an image signal and restore and combine it.

  The image processing method of the present embodiment is not limited to that performed by the system controller 15 of the imaging apparatus, and may be performed on a personal computer (PC) to which the imaging apparatus is connected by a USB cable (not shown). Alternatively, the recording medium 60 may be connected to a PC. That is, the image processing apparatus that executes the image processing method of the present embodiment may be an imaging apparatus or a PC.

  Information on the image processing method and the restoration filter may be stored in a memory (not shown) of the PC or a storage device connected to the PC via a network (for example, an HDD of a server connected via the Internet). Note that the image processing method includes not only image restoration processing but also development and other image processing functions.

  For example, a PC (not shown) may acquire an image signal of each wavelength and information on a shooting state from an imaging device or a storage medium (or an external storage device). The shooting state information may be different from the image signal information or may be embedded in the image signal. Thereafter, the PC executes the image processing method.

  In the present embodiment, the number of restoration regions and the number of focus detection regions are the same, but they may not be the same. Hereinafter, a case where the number of focus detection areas is smaller than the number of restoration areas will be described.

  FIG. 5 is a diagram showing the relationship between a plurality of restoration areas and a plurality of focus detection areas in this case. In FIG. 5, the screen has a plurality of restoration areas 201 to 235 similar to those in FIG. 2 and a plurality of focus detection areas 109 to 113, 116 to 120, and 123 to 127 that are smaller than those in FIG. 2.

  At this time, in S <b> 1 of FIG. 4, the system controller 15 converts the subject distances of the plurality of restoration areas 209 to 213, 216 to 220, and 223 to 227 to the corresponding plurality of focus detection areas 109 to 113, 116 to 120, 123. Obtained as a subject distance of ~ 127.

  On the other hand, the system controller 15 acquires subject distances of the plurality of restoration areas 201, 202, and 208 as subject distances of the closest (or adjacent) focus detection area 109. In addition, the system controller 15 acquires subject distances of the plurality of restoration areas 203 to 205 as subject distances of the focus detection areas 110 to 112 adjacent to each other.

  Similarly, the system controller 15 acquires subject distances of the plurality of restoration areas 206, 207, and 214 as subject distances of the closest (or adjacent) focus detection area 113. Further, the system controller 15 acquires subject distances of the plurality of restoration areas 215 and 221 as subject distances of the focus detection areas 116 and 120 adjacent to each other.

  In addition, the system controller 15 acquires subject distances of the plurality of restoration areas 222, 229, and 230 as subject distances of the closest (or adjacent) focus detection area 123. Further, the system controller 15 acquires subject distances of the plurality of restoration areas 231 to 233 as subject distances of the focus detection areas 124 to 126 adjacent to each other. In addition, the system controller 15 acquires subject distances of the plurality of restoration areas 228, 234, and 235 as subject distances of the closest (or adjacent) focus detection area 127.

  As described above, the subject distance information of the focus detection region closest to the subject distance in the restoration region having no focus detection region is set. The same applies to S2 and subsequent steps in FIG. Note that the subject distance information of the focus detection area having the closest aberration characteristic to the restoration area may be set in the subject distance of the restoration area that does not have the focus detection area. For example, in the axially symmetric imaging optical system 10, the focus detection area where the image height from the optical axis OA is closest is the area where the aberration characteristic is closest.

  A plurality of focus detection areas may be assigned to the restoration area. In that case, a simple average or a weighted average of object distances obtained for a plurality of corresponding focus detection areas, or one of a plurality of focus detection areas is selected as the object distance of the restoration area and the selected focus is selected. The subject distance in the detection area may be set.

  As the focus detection method, other focus detection methods such as a contrast detection method or a hybrid method of a phase difference detection method and a contrast detection method may be used. Also, the light beam used for focus detection may be a light beam that does not pass through the imaging optical system 10 instead of a light beam that has passed through the imaging optical system 10 (that is, internal measurement).

  For example, when the contrast detection method is used, the position of the focus lens 12 corresponding to the peak position of the contrast is the in-focus position, and the subject distance in each focus detection area is calculated from the position of the focus lens 12 where the contrast reaches the peak. Can be sought. In addition, the subject distance in the focus detection region selected for focus adjustment when actually capturing an image may be set as the focused subject distance.

  In this embodiment, the focus lens 12 is driven, but the image sensor 41 may be driven in focus adjustment. In this case, the focus detection information holding unit 34 may acquire information on the position of the imaging element 41 (or the distance between the imaging optical system 10 and the imaging element 41) instead of the position of the focus lens 12.

  In this embodiment, the restoration filter (or the amount corresponding to OTF) is acquired from the imaging state of the captured image (degraded image), but the restoration filter (or the aberration information obtained from the design data of the imaging system) The amount corresponding to OTF) may be acquired.

  As mentioned above, although the preferable Example of this invention was described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

  The imaging device can be applied to use for imaging a subject.

10 Imaging Optical System 50 Image Processing Unit (Image Processing Device)
51 Buffer memory (degraded image acquisition unit)
52 Imaging state information acquisition unit 52 Filter selection unit (optical transfer function acquisition unit)
56 Restoration processing unit

Claims (6)

One of the plurality of focus detection areas in a screen having a plurality of restoration areas, each of which is a target area for image restoration, and a plurality of focus detection areas, each of which is a target area for which focus detection is performed. A deteriorated image acquisition unit that acquires a deteriorated image obtained by imaging with an imaging optical system in focus;
The subject distance of each of the plurality of restoration regions obtained from the focused subject distance that is the subject distance for one of the plurality of focus detection regions and the subject distance obtained for each of the plurality of focus detection regions. An imaging state information acquisition unit that acquires information on an imaging state when the degraded image is captured,
An optical transfer function acquisition unit for acquiring an amount corresponding to the optical transfer function of the imaging optical system corresponding to the imaging state for each restoration region;
A restoration processing unit that restores the degraded image for each restoration region using an amount corresponding to the degraded image and the optical transfer function;
An image processing apparatus comprising:   An imaging apparatus comprising the image processing apparatus according to claim 1. One of the plurality of focus detection areas in a screen having a plurality of restoration areas, each of which is a target area for image restoration, and a plurality of focus detection areas, each of which is a target area for which focus detection is performed. Acquiring a deteriorated image obtained by focusing and imaging the imaging optical system;
The subject distance of each of the plurality of restoration regions obtained from the focused subject distance that is the subject distance for one of the plurality of focus detection regions and the subject distance obtained for each of the plurality of focus detection regions. Including acquiring information on an imaging state when the degraded image is captured;
Obtaining an amount corresponding to an optical transfer function of the imaging optical system corresponding to the imaging state for each restoration region;
Restoring the degraded image for each restored region using the degraded image and an amount corresponding to the optical transfer function and the original image;
An image processing method comprising: The number of the plurality of restoration regions and the number of the plurality of focus detection regions are the same, and each restoration region includes one focus detection region,
The image processing method according to claim 3, wherein the step of acquiring information on the imaging state sets a subject distance of a corresponding focus detection region as a subject distance of each restoration region. The number of the plurality of focus detection areas is less than the number of the plurality of restoration areas,
The step of acquiring the information on the imaging state sets the subject distance of the closest focus detection region or the focus detection region having the closest aberration characteristic to the subject distance of each restoration region. Image processing method. Computer
One of the plurality of focus detection areas in a screen having a plurality of restoration areas, each of which is a target area for image restoration, and a plurality of focus detection areas, each of which is a target area for which focus detection is performed. Means for acquiring a deteriorated image obtained by imaging with the imaging optical system in focus;
The subject distance of each of the plurality of restoration areas obtained from the in-focus subject distance that is the subject distance for one of the plurality of focus detection areas and the subject distance obtained for each of the plurality of focus detection areas Means for acquiring information on an imaging state when the degraded image is captured,
Means for obtaining an amount corresponding to an optical transfer function of the imaging optical system corresponding to the imaging state for each restoration region;
Means for restoring the degraded image for each restored region using the degraded image and an amount corresponding to the optical transfer function and the original image;
Program to make it function.