WO2013005602A1

WO2013005602A1 - Image capture device and image processing device

Info

Publication number: WO2013005602A1
Application number: PCT/JP2012/066187
Authority: WO
Inventors: 高宏矢野; 一哉山中
Original assignee: オリンパス株式会社
Priority date: 2011-07-04
Filing date: 2012-06-25
Publication date: 2013-01-10

Abstract

This image capture device is provided with: a color image sensor (22) that generates at least an R image and a B image; an imaging optical system (9) that forms a subject image on the image sensor (22); a band-limiting filter (12) is disposed on an optical path of an imaging light beam, and limits a first band, whereby, with regard to light that attempts to pass through part of an aperture area of the imaging optical system (9), B light is blocked and R light is allowed to pass through, and limits a second band, whereby, with regard to light that attempts to pass through the other part of the aperture area, R light is blocked and B light is allowed to pass through; a distance computation unit (39) that computes the amount of phase difference between the R image and the B image; and a color image generation unit (37) which, on the basis of the computed amount of phase difference, corrects a deviation between the position of the center of gravity of blur of the R image and the position of the center of gravity of blur of the B image.

Description

Imaging device, image processing device

The present invention relates to an imaging apparatus and an image processing apparatus that can acquire distance information based on an image obtained from an imaging element.

The distance information is used for AF processing by an automatic focus adjustment mechanism (AF), for creating a stereoscopic image, or for image processing (for example, subject extraction processing, background extraction processing, or image processing processing for blur amount control). And the like can be used to realize various functions in the imaging apparatus.

Various methods for obtaining such distance information have been proposed. For example, an active distance measurement method in which illumination light is irradiated and reflected light from a subject is received to perform distance measurement, or a baseline length. The focus lens is driven so as to increase the contrast of images acquired by a plurality of image pickup devices (for example, stereo cameras) arranged in accordance with the principle of triangulation, or the image acquired by the image pickup device itself. Various methods such as a contrast AF method have been proposed.

However, the active distance measurement method requires a dedicated member for distance measurement such as a light projection device for distance measurement, and the triangular distance measurement method requires a plurality of image pickup devices. Will increase. On the other hand, since the contrast AF method uses an image acquired by the imaging apparatus itself, a dedicated distance measuring member or the like is not required. However, the contrast value peak is obtained by performing multiple imaging while changing the position of the focus lens. Therefore, it takes time to search for a peak corresponding to the in-focus position, and it is difficult to perform high-speed AF.

Under such a background, as a technology to acquire distance information while suppressing an increase in size and cost, a light beam that passes through the pupil of the lens is divided into a plurality of light beams, and a light beam that passes through one pupil region in the lens Has been proposed to obtain distance information to the subject by performing a correlation operation between the pixel signal obtained from the pixel signal and the pixel signal obtained from the light beam that has passed through another pupil region in the lens. Yes. As a technique for simultaneously acquiring such pupil division images, for example, there is a technique for disposing a light shielding plate (mask) on a pixel for distance detection.

As a technique for performing pupil division without using a light-shielding plate (mask), for example, Japanese Patent Laid-Open No. 2001-174696 includes a pupil color division filter having different spectral characteristics for each partial pupil in an imaging optical system. In addition, a technique is described in which a subject image from a photographic optical system is received by a color image sensor to perform pupil division by color. In other words, the image signal output from the color image sensor is color-separated, and the relative shift amount between the same subject on each color image is detected, so that it is shifted from the in-focus position to the short distance side or the long distance Two pieces of focusing information, that is, a focusing shift direction that is shifted to the side and a focusing shift amount that is a shift amount from the in-focus position in that direction, are acquired.

In Japanese Patent Laid-Open No. 4-251239, partial pupils having spectral characteristics (blue and orange) as shown in FIG. 7 of the publication are provided, and a plurality of images with parallax are acquired simultaneously from different partial pupils. A technique for creating a stereoscopic image is described.

Further, Japanese Patent Application Laid-Open No. 11-344661 describes an AF diaphragm in which different color filters (G filter and M filter) are respectively arranged in openings arranged at different eccentric positions with respect to the center of the AF lens. Yes. In imaging for AF, the image data of the AF area in the image frame corresponding to each light beam passing through each color filter of the AF stop is acquired for each color, and the cross-correlation calculating unit determines between these image data for each color. The cross-correlation is calculated, and the distance direction calculation unit calculates the distance and direction to the in-focus position of the AF lens based on the cross-correlation, and drives the AF lens to the in-focus position.

By the way, in Japanese Patent Laid-Open No. 2001-16611, the amount of deviation between the first image that has passed through the first opening and the second image that has passed through the second opening is acquired, and the lens formula is used. A technique for calculating distance information is described.

However, when techniques such as those described in the above publications are used, each color image obtained from the image sensor is misaligned, so that the use of the obtained image is limited to AF. It was inappropriate.

The present invention has been made in view of the above circumstances, and an imaging apparatus and image processing that can make an image captured by light that has passed through different pupil regions of the imaging optical system depending on the band into a more preferable image for viewing. The object is to provide a device.

In order to achieve the above object, an imaging device according to one embodiment of the present invention receives and photoelectrically converts at least first-band light and second-band light to each of a first image signal and a second image signal. A color imaging device that generates an image signal, an imaging optical system that forms a subject image on the imaging device, and an optical path of a photographic light flux from the subject side of the imaging optical system to the imaging device. The first band limitation for blocking the light in the first band and allowing the light in the second band to pass with respect to the light that attempts to pass through a part of the pupil region of the imaging optical system is performed. A band-limiting filter that performs a second band limitation that blocks the light in the second band and allows the light in the first band to pass with respect to light that is about to pass through another part of the pupil region; and the first image signal And calculating the phase difference amount based on the second image signal An image for correcting a positional deviation between the blur center position of the first image signal and the blur center position of the second image signal based on the calculation unit and the phase difference amount calculated by the calculation unit. And a correction unit.

An image processing apparatus according to another aspect of the present invention receives at least light in the first band and light in the second band and performs photoelectric conversion to generate a first image signal and a second image signal. A color imaging device, an imaging optical system that forms a subject image on the imaging device, and an optical path of a photographic light beam from the subject side of the imaging optical system to the imaging device. A first band restriction for blocking light in the first band and allowing light in the second band to pass through a part of the pupil area of the system; Processing an image obtained by an imaging device having a second band limiting filter that blocks the second band light and allows the first band light to pass with respect to light that is about to pass through a part An image processing apparatus for performing the first processing A calculation unit that calculates a phase difference amount based on an image signal and the second image signal; a center of gravity position of a blur of the first image signal based on the phase difference amount calculated by the calculation unit; And an image correction unit that corrects a positional deviation between the center of gravity position of the blur of the second image signal.

1 is a block diagram illustrating a configuration of an imaging apparatus according to Embodiment 1 of the present invention. FIG. 3 is a diagram for explaining a pixel array of the image sensor in the first embodiment. The figure for demonstrating one structural example of the band-limiting filter in the said Embodiment 1. FIG. In the said Embodiment 1, the top view which shows the mode of a subject light beam condensing when imaging the to-be-photographed object in a focus position. FIG. 3 is a plan view showing a state of subject light flux condensing when an image of a subject closer to the in-focus position is imaged in the first embodiment. FIG. 4 is a diagram showing a blur shape formed by light from one point on a subject that is closer to the in-focus side than the in-focus position in the first embodiment. In the said Embodiment 1, the figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object in the short distance side from a focus position for every color component. In the said Embodiment 1, the top view which shows the mode of a subject light beam condensing when imaging the to-be-photographed object from the in-focus position. FIG. 4 is a diagram showing a blur shape formed by light from one point on a subject that is farther than the in-focus position in the first embodiment. In the said Embodiment 1, the figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object on the far side from the focus position for every color component. In the said Embodiment 1, the figure which shows the mode of an image when the to-be-photographed object and the to-be-photographed object in the near distance and far distance are imaged. The figure which shows the 1st modification of the band-limiting filter in the said Embodiment 1. FIG. The figure which shows the 2nd modification of the band-limiting filter in the said Embodiment 1. FIG. The figure which shows the 3rd modification of the band-limiting filter in the said Embodiment 1. FIG. The figure which shows the 4th modification of the band-limiting filter in the said Embodiment 1. FIG. In the said Embodiment 1, the figure which shows the outline | summary of the blur shape after a colorization process. The figure which shows the shape of the R filter kernel applied with respect to the R image of the to-be-photographed object in the far side from the focus position in the colorization process of the said Embodiment 1. FIG. The figure which shows the shape of the filter kernel for B applied to the B image of the to-be-photographed object in the far distance side from the focus position in the colorization process of the said Embodiment 1. FIG. In the said Embodiment 1, the figure which shows the mode of the shift of the R image of the to-be-photographed object and B image in the far side from the focus position in the colorization process of another example. The figure which shows the shape of the filter applied with respect to R image and B image in the colorization process of the other example of the said Embodiment 1. FIG. In the said Embodiment 1, the figure which shows the shape of the elliptical Gaussian filter which enlarged the standard deviation of the horizontal direction. In the said Embodiment 1, the figure which shows the shape of the elliptical Gaussian filter which made the standard deviation of the vertical direction small. 5 is a flowchart showing colorization processing performed by a color image generation unit in the first embodiment. FIG. 2 is a block diagram illustrating a configuration of an image processing apparatus according to the first embodiment. The figure which shows the outline | summary of the colorization process performed by the color image generation part in Embodiment 2 of this invention. The figure which shows the partial area | region set to R image and B image when performing phase difference detection in the said Embodiment 2. FIG. FIG. 10 is a diagram illustrating a state in which a blur diffusion partial region of an original R image is copied and added to an R copy image in the second embodiment. FIG. 10 is a diagram showing a state in which the blur diffusion partial area of the original B image is copied and added to the B copy image in the second embodiment. In the said Embodiment 2, the diagram which shows the example which changes the size of a blur spreading | diffusion partial area | region according to phase difference amount. 9 is a flowchart illustrating colorization processing performed by a color image generation unit in the second embodiment. FIG. 10 is a diagram illustrating an outline of a PSF table for each color according to a phase difference amount in the third embodiment of the present invention. FIG. 10 is a diagram illustrating an outline of colorization processing performed by a color image generation unit in the third embodiment. In the said Embodiment 3, the figure which shows the outline | summary of the colorization process accompanying the blurring amount control performed by the color image generation part. 9 is a flowchart illustrating colorization processing performed by a color image generation unit in the third embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Embodiment 1]
1 to 24 show Embodiment 1 of the present invention, and FIG. 1 is a block diagram showing a configuration of an imaging apparatus.

The imaging apparatus of the present embodiment is configured as a digital still camera, for example. However, here, a digital still camera is taken as an example, but the imaging device may be any device that has a color imaging device and has an imaging function. A digital still camera, a video camera, a camera-equipped mobile phone, a camera-equipped personal digital assistant (PDA with camera), a camera-equipped personal computer, a surveillance camera, an endoscope, and the like.

The imaging apparatus includes a lens unit 1 and a body unit 2 that is a main body portion to which the lens unit 1 is detachably attached via a lens mount. Here, a case where the lens unit 1 is detachable will be described as an example, but of course, it may not be detachable.

The lens unit 1 includes an imaging optical system 9 including a lens 10 and a diaphragm 11, a band limiting filter 12, a lens control unit 14, and a lens side communication connector 15.

The body unit 2 includes a shutter 21, an imaging element 22, an imaging circuit 23, an imaging drive unit 24, an image processing unit 25, an image memory 26, a display unit 27, an interface (IF) 28, and a system controller. 30, a sensor unit 31, an operation unit 32, a strobe control circuit 33, a strobe 34, and a body side communication connector 35. In addition, although the recording medium 29 is also described in the body unit 2 of FIG. 1, this recording medium 29 is a memory card (smart media, SD card, xD picture card, etc.) which is detachable with respect to the imaging device. Since it is configured, the configuration may not be unique to the imaging apparatus.

First, the imaging optical system 9 in the lens unit 1 is for forming a subject image on the imaging element 22. The lens 10 of the imaging optical system 9 includes a focus lens for performing focus adjustment. Although the lens 10 is generally composed of a plurality of lenses in general, only one lens is shown in FIG. 1 for simplicity.

The diaphragm 11 of the imaging optical system 9 is for adjusting the brightness of the subject image formed on the imaging element 22 by regulating the passage range of the subject luminous flux that passes through the lens 10.

The band limiting filter 12 is disposed on the optical path of the photographing light beam from the subject side of the imaging optical system 9 to the imaging element 22 (preferably at the position of the diaphragm 11 of the imaging optical system 9 or in the vicinity thereof). The first band restriction for blocking light in the first band and allowing light in the second band to pass through a part of the pupil area of the system 9 is performed, and the other pupil area of the imaging optical system 9 It is a filter that performs second band limitation for blocking light in the second band and allowing light in the first band to pass through the light that is about to pass through the section. In particular, the band limiting filter 12 in the present embodiment blocks the first band light and the second band and the second band in the imaging light flux that attempts to pass through the first area that is a part of the pupil area of the imaging optical system 9. The first band limitation that allows light in the three bands to pass through and the second band light in the imaging light flux that attempts to pass through the second area that is another part of the pupil area of the imaging optical system 9 are blocked. This is a filter that performs the second band limitation that allows light in the first band and the third band to pass therethrough.

Here, FIG. 3 is a diagram for explaining a configuration example of the band limiting filter 12. In the band limiting filter 12 of the configuration example shown in FIG. 3, the pupil region of the imaging optical system 9 is divided into a first region and a second region. In other words, the band limiting filter 12 has a left half of the G (green) component and R (red) when the image pickup device is viewed from the image pickup device 22 in a standard posture (so-called posture in which the camera is held in a normal horizontal position). The RG filter 12r that passes the component and blocks the B (blue) component (that is, the B (blue) component is one of the first band and the second band), and the right half is the G component and the B component. And the R component is cut off (that is, the R component is the other one of the first band and the second band). Therefore, the band limiting filter 12 passes all the G components included in the light passing through the aperture 11 of the imaging optical system 9 (that is, the G component is the third band), and the R component is half of the aperture. Only the region is allowed to pass, and the B component is allowed to pass only for the remaining half of the aperture. If the RGB spectral transmission characteristics of the band limiting filter 12 are different from the RGB spectral transmission characteristics of an element filter (see FIG. 2) configured on-chip in the image sensor 22, the RG filter 12r and the GB filter 12b As will be described later, the accuracy of the position information acquired from the images obtained based on the difference in the spatial position is reduced, or a light amount loss due to a mismatch in spectral characteristics occurs. Therefore, it is desirable that the spectral transmission characteristic of the band limiting filter 12 is the same as or as close as possible to the spectral transmission characteristic of the element filter of the image sensor 22. Further, other configuration examples of the band limiting filter 12 will be described later with reference to FIGS.

The lens control unit 14 controls the lens unit 1. That is, the lens control unit 14 drives and focuses the focus lens in the lens 10 based on a command received from the system controller 30 via the lens-side communication connector 15 and the body-side communication connector 35, or the aperture 11 is moved. It is driven to change the aperture diameter.

The lens-side communication connector 15 enables communication between the lens control unit 14 and the system controller 30 by connecting the lens unit 1 and the body unit 2 with a lens mount and connecting to the body-side communication connector 35. Connector.

Next, the shutter 21 in the body unit 2 is an optical shutter for adjusting the exposure time of the image sensor 22 by regulating the passage time of the subject light beam reaching the image sensor 22 from the lens 10. Although an optical shutter is used here, an element shutter (electronic shutter) by the image sensor 22 may be used instead of or in addition to the optical shutter.

The imaging device 22 receives and photoelectrically converts the subject image formed by the imaging optical system 9 for each of a plurality of wavelength bands (for example, but not limited to RGB), and performs photoelectric conversion. A color image sensor that outputs a signal, for example, a CCD or CMOS. Here, the configuration of the color image sensor may be a single-plate image sensor provided with an on-chip element color filter, or a three-plate system using a dichroic prism that performs color separation into RGB color lights. However, it may be an image sensor of a method capable of acquiring RGB imaging information according to the position in the depth direction of the semiconductor at the same pixel position, or any imaging element that can acquire imaging information of a plurality of wavelength bands. It does n’t matter.

For example, with reference to FIG. 2, a configuration example of a single-plate color image sensor often used for a general digital still camera will be described. FIG. 2 is a diagram for explaining the pixel arrangement of the image sensor 22. In the present embodiment, the plurality of wavelength band lights transmitted through the element color filter mounted on-chip are R, G, and B. As shown in FIG. An image sensor is configured. Therefore, when the image pickup device 22 has the configuration shown in FIG. 2, only one color component is obtained per pixel, and therefore the image processor 25 performs a demosaicing process. A color image in which three colors of RGB are aligned per pixel is generated. Accordingly, the image sensor 22 receives and photoelectrically converts at least R light, for example, as light in the first band and B light, for example, as light in the second band, and performs R conversion as the first image signal. A signal and a B image signal as a second image signal are generated.

The image pickup circuit 23 performs A / D conversion when the image signal output from the image pickup device 22 is amplified (gain adjustment), or when the image pickup device 22 is an analog image pickup device and outputs an analog image signal. Or a digital image signal (hereinafter also referred to as “image information”) (when the image pickup device 22 is a digital image pickup device, it is already digital when it is input to the image pickup circuit 23). Therefore, A / D conversion is not performed). The imaging circuit 23 outputs an image signal to the image processing unit 25 in a format corresponding to the imaging mode switched by the imaging driving unit 24 as will be described later.

The imaging drive unit 24 supplies a timing signal and power to the imaging element 22 and the imaging circuit 23 based on a command from the system controller 30 to cause the imaging element to perform exposure, reading, element shuttering, and the like, and also to the imaging element 22. Control is performed to execute gain adjustment and A / D conversion by the imaging circuit 23 in synchronism with the above operations. The imaging drive unit 24 also performs control to switch the imaging mode of the image sensor 22.

The image processing unit 25 is a digital image such as WB (white balance) adjustment, black level correction, γ correction, defective pixel correction, demosaicing, image information color information conversion processing, image information pixel number conversion processing, and the like. The processing is performed. The image processing unit 25 further includes a color correction unit 36 and a color image generation unit 37 which is an image correction unit and is a setting unit.

As described above, since the band limiting filter 12 limits the pass region in the pupil region of the imaging optical system 9 by the band (color) of the light that passes through, the brightness of the light that passes through the band varies. become. The inter-color correction unit 36 is for correcting such a difference in brightness between bands (between colors). The brightness difference between bands (between colors) can be easily corrected according to the area of the pass region for each band, but the amount of light in the periphery is larger than the center of the image. Of course, more detailed correction according to the optical characteristics of the imaging optical system 9 may be performed in consideration of the tendency to decrease. At this time, the correction value is not limited to be calculated in the imaging apparatus, and the correction value may be held as table data or the like. As a specific example, when the band limiting filter 12 is configured as shown in FIG. 3, R and B have half the amount of light passing through G, so that the R color signal and the B color signal are 2 It is conceivable to perform the doubling process as a simple inter-color correction (similarly, in the case of the configuration shown in FIGS. 12 to 15 described later, correction is performed according to the area of the pass region for each band. become). By performing such processing, it becomes possible to handle the image as an image similar to that of the imaging device in which the band limiting filter 12 of FIG. 3 is not inserted in terms of light quantity balance, and various functions (for example, image generation processing, etc.) can be performed. It can be used.

The color image generation unit 37 performs a colorization process that is a digital process for forming color image information. When the band limiting filter 12 as shown in FIG. 3 is used, a spatial positional deviation may occur between the R image and the B image. Therefore, the spatial positional deviation is corrected. It is an example of the colorization process which the color image generation part 37 performs. Since this positional deviation occurs in the blurred portion (non-focused portion) and does not occur in the focused portion, more specifically, the color image generation unit 37 is calculated by a distance calculation unit 39, which will be described later, which is a calculation unit. Based on the phase difference amount, a process of correcting a positional deviation between a blur center of gravity of the first image signal (for example, R image signal) and a blur center of gravity of the second image signal (for example, the B image signal). Do.

First, the spatial positional shift of each color image will be described with reference to FIGS. FIG. 4 is a plan view showing a state of subject light flux condensing when imaging a subject at the in-focus position, and FIG. 5 is a subject light flux concentrating when imaging a subject closer to the in-focus position. FIG. 6 is a plan view showing the state of light, FIG. 6 is a diagram showing the shape of a blur formed by light from one point on the subject that is closer to the focus position, and FIG. 7 is a closer distance than the focus position. FIG. 8 is a diagram showing the shape of blur formed by light from one point on a subject on the side for each color component, and FIG. 8 is a diagram of subject light flux collection when imaging a subject farther than the in-focus position. FIG. 9 is a plan view showing the state, FIG. 9 is a diagram showing the shape of a blur formed by light from one point on the subject that is on the far side from the in-focus position, and FIG. 10 is on the far side from the in-focus position. FIG. 11 is a diagram showing the shape of a blur formed by light from one point on a subject for each color component. Location and than a diagram showing a state of image obtained by imaging the object at a short distance and long distance. In the description of the blur shape, the case where the aperture of the diaphragm 11 has a circular shape will be described as an example.

When the subject OBJc is at the in-focus position, the light emitted from one point on the subject OBJc has a G component that passes through the entire band limiting filter 12 as shown in FIG. Both the R component that passes only through the filter 12r and the B component that passes only through the other half of the GB filter 12b of the band limiting filter 12 are collected at one point on the image sensor 22 to form a point image IMGrgb. Although there is a difference in the amount of light according to the area of the passing region as described above, no positional deviation occurs between the colors. Therefore, when the subject OBJc at the in-focus position is imaged, a subject image IMGrgb having no color blur is formed as shown in FIG.

On the other hand, when the subject OBJn is, for example, closer to the in-focus position, the light emitted from one point on the subject OBJn is used for the G component as shown in FIGS. A subject image IMGg that forms a circular blur is formed, a subject image IMGr that forms a semicircular blur of the left half is formed for the R component, and a subject image IMGb that forms a semicircular blur of the right half is formed for the B component. Therefore, when the subject OBJn that is closer to the in-focus position is imaged, as shown in FIG. 11, the blur component image in which the R component subject image IMGr is shifted to the left and the B component subject image IMGb is shifted to the right. The left and right positions of the R component and B component in this blurred image are the R component transmission region (RG filter 12r) and B component transmission region (GB filter 12b) of the band limiting filter 12 when viewed from the image sensor 22. (In FIG. 11, the deviation is emphasized and illustration of the blurred shape actually generated is omitted (the same applies to the case of the far side below)). As the subject OBJn moves away from the in-focus position, the blur increases, and the distance between the center of gravity of the R component subject image IMGr and the center of gravity of the B component subject image IMGb increases. Become.

On the other hand, when the subject OBJf is on the far side from the in-focus position, for example, the light emitted from one point on the subject OBJf causes a circular blur for the G component as shown in FIGS. A subject image IMGg is formed, a subject image IMGr that forms a semicircular blur of the right half is formed for the R component, and a subject image IMGb that forms a semicircular blur of the left half is formed for the B component. Therefore, when the subject OBJf that is farther than the in-focus position is imaged, as shown in FIG. 11, the blur component image in which the R component subject image IMGr is shifted to the right side and the B component subject image IMGb is shifted to the left side. The left and right positions of the R component and B component in this blurred image are the R component transmission region (RG filter 12r) and B component transmission region (GB filter 12b) of the band limiting filter 12 when viewed from the image sensor 22. It is the opposite of the left and right position. As the subject OBJf moves away from the in-focus position, the blur increases, and the distance between the center of gravity of the R component subject image IMGr and the center of gravity of the B component subject image IMGb increases. Become.

The image memory 26 is a memory capable of high-speed writing and reading. The image memory 26 is composed of, for example, SDRAM (Synchronous Dynamic Random Access Memory), and is used as a work area for image processing. Also used as For example, the image memory 26 not only stores the final image processed by the image processing unit 25 but also appropriately stores each intermediate image in a plurality of processing steps by the image processing unit 25.

The display unit 27 includes an LCD or the like, and an image processed for display by the image processing unit 25 (an image read from the recording medium 29 and processed for display by the image processing unit 25 is also included). Display). Specifically, the display unit 27 performs live view image display, confirmation display when recording a still image, reproduction display of a still image or a moving image read from the recording medium 29, and the like.

The interface (IF) 28 is detachably connected to the recording medium 29, and transmits information to be recorded on the recording medium 29 and information read from the recording medium 29.

The recording medium 29 is for recording an image processed for recording by the image processing unit 25 and various data related to the image, and is configured as a memory card or the like as described above.

The sensor unit 31 is, for example, a camera shake sensor configured by an acceleration sensor or the like for detecting blur of the imaging device, a temperature sensor for measuring the temperature of the imaging device 22, and a brightness for measuring the brightness around the imaging device. Includes brightness sensor, etc. The detection result by the sensor unit 31 is input to the system controller 30. Here, the detection result by the camera shake sensor is used to drive the image pickup device 22 and the lens 10 to perform camera shake correction or to perform camera shake correction by image processing. The detection result by the temperature sensor is used to control the drive clock by the imaging drive unit 24 and to estimate the amount of noise in the image obtained from the image sensor 22. Furthermore, the detection result by the brightness sensor is used, for example, to appropriately control the luminance of the display unit 27 according to the ambient brightness.

The operation unit 32 changes a power switch for turning on / off the power of the image pickup device, a release button including a two-stage press button for inputting an image pickup operation such as a still image or a moving image, an image pickup mode, and the like. Mode buttons, cross keys used to change selection items, numerical values, and the like. A signal generated by the operation of the operation unit 32 is input to the system controller 30.

The strobe control circuit 33 controls the light emission amount and the light emission timing of the strobe 34 based on a command from the system controller 30.

The strobe 34 is a light source that emits illumination light to the subject under the control of the strobe control circuit 33.

As described above, the body side communication connector 35 is connected between the lens control unit 14 and the system controller 30 when the lens unit 1 and the body unit 2 are coupled by the lens mount and connected to the lens side communication connector 15. It is a connector that enables communication.

The system controller 30 controls the body unit 2 and also controls the lens unit 1 via the lens control unit 14, and is a control unit that integrally controls the imaging apparatus. The system controller 30 reads a basic control program of the image pickup apparatus from a non-illustrated non-volatile memory such as a flash memory, and controls the entire image pickup apparatus in accordance with an input from the operation unit 32.

For example, the system controller 30 controls the diaphragm adjustment of the diaphragm 11 via the lens control unit 14, controls and drives the shutter 21, or shake correction (not shown) based on the detection result of the acceleration sensor of the sensor unit 31. The mechanism is controlled to perform camera shake correction or the like. Furthermore, the system controller 30 sets the mode setting of the imaging device (a still image mode for capturing a still image, a moving image mode for capturing a moving image, a stereoscopic image) in response to an input from the mode button of the operation unit 32. 3D mode or the like for imaging).

Further, the system controller 30 includes a contrast AF control unit 38 and a distance calculation unit 39 as a calculation unit. The system controller 30 performs AF control by the contrast AF control unit 38, or uses the distance information calculated by the distance calculation unit 39. Based on this, the lens unit 1 is controlled to perform AF.

The contrast AF control unit 38 may output an image signal output from the image processing unit 25 (this image signal may be a G image having a high ratio including a luminance component, and color misregistration is corrected by a colorization process described later. A contrast value (also referred to as an AF evaluation value) is generated from the luminance signal image related to the image, and the focus lens in the lens 10 is controlled via the lens control unit 14. That is, the contrast AF control unit 38 extracts a high-frequency component by applying a filter, for example, a high-pass filter, to the image signal to obtain a contrast value. Then, the contrast AF control unit 38 acquires the contrast value by changing the focus lens position, moves the focus lens in the direction in which the contrast value increases, and further acquires the contrast value. By repeatedly performing such processing, control is performed so that the focus lens is driven to the focus lens position (focus position) at which the maximum contrast value is obtained.

Next, the distance calculation unit 39 calculates the phase difference amount between the first image in the first band and the second image in the second band obtained from the image sensor 22, and based on the calculated phase difference amount. The distance to the subject is calculated. Specifically, the distance calculation unit 39 calculates distance information according to the lens formula from the amount of deviation between the R component and the B component as shown in FIG. 7, FIG. 10, FIG.

That is, the distance calculation unit 39 extracts the R component and the B component from among the components of each RGB color obtained as the captured image. Then, by calculating the correlation between the R component and the B component, the direction of the deviation and the magnitude of the deviation occurring in the R component image and the B component image are calculated (however, the present invention is not limited to this). Alternatively, it is possible to calculate the direction of displacement and the size of the displacement occurring between the R image and the G image or between the B image and the G image).

Here, for example, when the band limiting filter 12 as shown in FIG. 3 is used, as described with reference to FIGS. 5 to 11, depending on whether the subject is closer or farther than the in-focus position, The direction of deviation between the R component and the B component is reversed. Therefore, the direction of deviation is information for determining whether the subject of interest is closer or farther than the in-focus position.

Also, the further away from the in-focus position, the closer to the near side or the far side, the greater the difference between the R component and the B component. Therefore, the magnitude of the deviation is information for determining how far the subject of interest is away from the in-focus position.

In this way, the distance calculation unit 39 calculates how far the subject of interest is away to the side closer to or farther from the in-focus position based on the calculated direction of deviation and magnitude of deviation. It is like that.

The distance calculation unit 39 is based on an input from the operation unit 32 or based on a basic control program of the imaging apparatus, and a distance calculation area determined by the system controller 30 (for example, the entire area of the captured image or distance information in the captured image). The distance calculation as described above is performed for a part of the region where it is desired to acquire the For example, a technique described in Japanese Patent Application Laid-Open No. 2001-16611 can be used as such a technique for obtaining the deviation amount and calculating the distance information.

The distance information acquired by the distance calculation unit 39 can be used for, for example, autofocus (AF).

That is, the distance calculation unit 39 acquires distance information based on the difference between the R component and the B component, and the system controller 30 drives the focus lens of the lens 10 via the lens control unit 14 based on the acquired distance information, that is, AF. Phase difference AF can be performed. Thereby, high-speed AF is possible based on one captured image.

However, since the distance calculation unit 39 and the contrast AF control unit 38 are provided in the system controller 30 as described above, AF control may be performed based on the calculation result by the distance calculation unit 39. It may be performed by the contrast AF control unit 38.

Here, contrast AF by the contrast AF control unit 38 has high focusing accuracy, but it requires a plurality of captured images, and therefore there is a problem that the focusing speed cannot be said to be fast. On the other hand, since the calculation of the subject distance by the distance calculation unit 39 can be performed based on one captured image, the focusing speed is fast, but the focusing accuracy may be inferior to the contrast AF.

Therefore, the AF assist control unit 38a provided in the contrast AF control unit 38 may perform AF by combining the contrast AF control unit 38 and the distance calculation unit 39. That is, the distance calculation unit 39 performs a distance calculation based on the deviation between the R component and the B component of the image acquired through the band limiting filter 12 to determine whether the subject is on the far side from the current focus position. Or it is acquired whether it exists in the short distance side. Alternatively, the distance that the subject is away from the current focus position is acquired. Next, the AF assist control unit 38a controls the contrast AF control unit 38 to drive the focus lens toward the acquired long distance side or the short distance side (by the acquired distance) and perform contrast AF. By performing such processing, it is possible to obtain high focusing accuracy at a fast focusing speed.

Also, the R image and B image obtained from the image sensor 22 can be used as, for example, a stereo stereoscopic image (3D image).

The 3D image only needs to be able to observe the image from the left pupil with the left eye and the image from the right pupil with the right eye. An anaglyph method has been conventionally known as such a 3D image observation method (see also Japanese Patent Laid-Open No. 4-251239 described above). This anaglyph method generally generates a red left-eye image and a blue right-eye image, displays both images, and arranges a red transmission filter on the left eye side and a blue transmission filter on the right eye side. By observing this image using blue glasses, a monochrome stereoscopic image can be observed.

Therefore, in the present embodiment, an R image and a B image obtained from the image sensor 22 with a standard posture (an image in which color processing for correcting the positional deviation between the R component and the B component in the color image generation unit 37 is not performed) are performed. If the anaglyph type red / blue glasses are used for observation, a stereoscopic view is possible as it is. That is, as described with reference to FIG. 3, when the imaging apparatus is in the standard posture, the RG filter 12r of the band limiting filter 12 is arranged on the left side when the subject is viewed from the imaging element 22, and the GB filter 12b is on the right side. It is comprised so that it may be arrange | positioned. Thus, when wearing red and blue glasses, the R component light transmitted through the left RG filter 12r is observed only by the left eye, and the B component light transmitted through the right GB filter 12b is observed only by the right eye. Is possible.

Next, modified examples of the band limiting filter 12 will be described with reference to FIGS.

First, FIG. 12 is a diagram showing a first modification of the band limiting filter 12.

In the band limiting filter 12 shown in FIG. 12, the imaging light beam in which the pupil region of the imaging optical system 9 receives both the first band limitation and the second band limitation between the first region and the second region. It is configured such that a passing area is sandwiched. That is, the band limiting filter 12 shown in FIG. 3 is the RG filter 12r on the left half and the GB filter 12b on the right half when viewed from the image pickup device 22 with the image pickup apparatus in the standard posture. On the other hand, in the band limiting filter 12 shown in FIG. 12, a G filter 12g that allows only the G component to pass is disposed between the left RG filter 12r and the right GB filter 12b.

Even in such a configuration, all of the G component contained in the light passing through the aperture of the diaphragm 11 of the imaging optical system 9 passes in the same way as the band limiting filter 12 in FIG. However, since the RG filter 12r and the GB filter 12b are arranged apart from each other, the positional deviation between the R component and the B component of the image of the subject at the out-of-focus position on the image sensor 22 is further clarified. Therefore, there is an advantage that the distance calculation accuracy by the distance calculation unit 39 is improved. Therefore, when it is desired to obtain distance information with high accuracy, it is preferable to use the band limiting filter 12 configured as shown in FIG.

FIG. 13 is a diagram showing a second modification of the band limiting filter 12.

The band limiting filter 12 shown in FIG. 13 has a configuration in which the pupil region of the imaging optical system 9 is sandwiched between the first region and the second region through which the imaging light flux that is not subjected to the band limitation passes. ing. In other words, the band limiting filter 12 is a filter in which a W filter 12w that passes components of all RGB colors (that is, white light W) is disposed between the left RG filter 12r and the right GB filter 12b. .

Even in such a configuration, all of the G component contained in the light passing through the aperture of the diaphragm 11 of the imaging optical system 9 passes in the same way as the band limiting filter 12 in FIG. In contrast, the R component passes through the RG filter 12r and the W filter 12w, but is blocked by the GB filter 12b. Similarly, the B component passes through the GB filter 12b and the W filter 12w, but is blocked by the RG filter 12r.

According to such a configuration, not only the G component but also the R component and the B component are transmitted through the portion of the W filter 12w, so that the light amounts of the R component and the B component that pass through the band limiting filter 12 increase. There is an advantage that a bright captured image can be acquired (however, the accuracy of the distance information calculated by the distance calculation unit 39 may be reduced). Accordingly, when the distance information is used when high accuracy is not necessary, a necessary effect can be obtained and, for example, a bright high-quality 3D image can be acquired.

Further, FIG. 14 is a diagram showing a third modification of the band limiting filter 12.

In the band limiting filter 12 shown in FIG. 14, the first region and the second region are arranged with different vertical and horizontal positions when the imaging apparatus is in the standard posture. Yes. That is, the band limiting filter 12 divides a circular filter into four crosses, for example, and arranges the RG filter 12r in the lower left (third quadrant in the graph) and the GB filter 12b in the upper right (first quadrant in the graph). In addition, the first G filter 12g1 is arranged in the upper left (second quadrant in the graph), and the second G filter 12g2 is arranged in the lower right (fourth quadrant in the graph).

In the case of the configuration of the band limiting filter 12 as shown in FIGS. 3, 12, and 13 described above, it is easy to obtain distance information of a subject having a vertical edge, but a subject having a horizontal edge. It becomes difficult to acquire distance information. On the other hand, if the configuration of the band limiting filter 12 as shown in FIG. 14 is adopted, the distance to both the subject having the vertical edge and the subject having the horizontal edge is determined. There is an advantage that information can be easily acquired. Thus, by making the horizontal position and the vertical position of the RG filter 12r and the GB filter 12b different, it is possible to improve the subject edge direction dependency (particularly, the horizontal / vertical direction dependency) in distance information acquisition. It becomes.

Even when the band limiting filter 12 is configured as shown in FIG. 12 to FIG. 14 and the like, the amount of deviation between the R image and the B image, or a blurred PSF (Point Spread Function: point spread) as will be described later. Since only the shape of the function) changes, a colorization process for correcting the positional deviation between the R component and the B component in the color image generation unit 37 as described below is similarly applied to correct the color deviation. Generated images can be generated.

Subsequently, FIG. 15 is a diagram showing a fourth modification of the band limiting filter 12.

Although the band limiting filter 12 shown in FIG. 3 and FIGS. 12 to 14 is a normal color filter, the band limiting filter 12 shown in FIG. 15 is configured by a color selective transmission element 18.

Here, the color selective transmission element 18 includes a member capable of rotating the polarization transmission axis corresponding to the color (wavelength) and a member capable of selectively controlling whether or not to rotate the polarization transmission axis such as an LCD. This element is realized by combining a plurality of elements, and can change the color distribution. A specific example of the color selective transmission element 18 is a color switch manufactured by Color Link as disclosed in “SID '00 Digest, Vol. 31, p. 92” in SID2000 in April 2000 AD. .

The color selective transmission element 18 has the left half as the first color selection transmission element 18L and the right half as the second color selection transmission element 18R when viewed from the imaging element 22 with the imaging apparatus in the standard posture (lateral position). It is configured. Here, the first color selective transmission element 18L can selectively take an RG state that passes the G component and the R component and blocks the B component, and a W state that passes all the RGB components (W). It has become. Further, the second color selective transmission element 18R can selectively take a GB state in which the G component and the B component are allowed to pass and an R component is blocked, and a W state in which all the RGB components (W) are allowed to pass. ing. The first color selection / transmission element 18L and the second color selection / transmission element 18R are independently driven by a color selection drive unit (not shown), and the first color selection / transmission element 18L is set in the RG state. In addition, by causing the second color selective transmission element 18R to assume the GB state, the same function as the band limiting filter 12 shown in FIG. 3 can be achieved.

Furthermore, if the configuration is changed to another configuration, the color selective transmission element 18 can perform the same function as the band limiting filter 12 shown in FIGS. The band limiting filter 12 can also be configured.

Next, a colorization process for correcting a shift (color shift) between the color images performed in the color image generation unit 37 will be described.

When the subject on the far side from the in-focus position is photographed using the band limiting filter 12 shown in FIG. 3, a blur as shown in FIG. 7 is formed, and the subject is in the near side from the in-focus position. As shown in FIG. 10, a blur is formed.

The colorization process is a process for correcting the blur having the shape shown in FIG. 7 or 10 to the blur having the shape shown in FIG. FIG. 16 is a diagram showing an outline of the blurred shape after the colorization process.

In the following, in order to simplify the description, the case where the subject is on the far side from the in-focus position (in the case shown in FIG. 10) will be described as an example, but the subject is closer to the in-focus side than the in-focus position. In this case, as can be seen by comparing FIG. 7 with FIG. 10, the shape and position of the blur of the R image and the blur of the B image are just opposite to each other. It is possible to apply similarly.

FIG. 17 is a diagram showing the shape of the R filter kernel applied to the R image of the subject on the far side from the in-focus position in the colorization processing, and FIG. 18 is far from the in-focus position in the colorization processing. It is a figure which shows the shape of the filter kernel for B applied with respect to the B image of the to-be-photographed object on the distance side.

In this colorization process, color misregistration correction and blur shape correction are performed by filtering processing (processing that convolves a filter kernel with an image). That is, by performing R filtering processing on the R image, the shape of the blur of the R image and the centroid position (Ch, Cr) (this centroid position (Ch, Cr) is determined in the example shown in FIG. The peak position of the filter coefficient of the filter kernel for R) is the ideal shape of circular blur and the center of gravity (Ch, Cg) (this center of gravity (Ch, Cg) is shown in FIG. 17 and FIG. In the example shown, the B filter is made close to the coordinate center of the R filter kernel and the B filter kernel), and the B filtering process is performed on the B image, whereby the blur shape and the gravity center position of the B image ( Ch, Cb) (this centroid position (Ch, Cb) is also the peak position of the filter coefficient of the B filter kernel in the example shown in FIG. 18). The shape and position of the center of gravity of the circular blur an ideal blur mentioned above (Ch, Cg) has become a process of close to.

First, it is assumed that an ideal blur center of gravity position for the first and second image signals is set by the color image generation unit 37 as setting means. Here, the setting of the ideal center of gravity of the blur shape by the color image generation unit 37 as the setting means can be set to a desired position. As a specific setting example, the blur shape of the R image can be set as a desired position. Some examples include an example in which the geometrical positional relationship between the center of gravity position and the center of gravity position of the blur shape of the B image is set constant, and an example of setting as the center of gravity position of the blur shape of the G image that is the standard image. Is considered. More specifically, an example in which the former geometric positional relationship is set to be constant is more specifically an inner division point (more than a predetermined ratio between the center of gravity position of the blur shape of the R image and the center of gravity position of the blur shape of the B image. Preferably, for example, setting as a midpoint) can be considered.

The GB filter 12b and the RB filter 12r in the band limiting filter 12 shown in FIG. 3 each occupy a half of the pupil region and have a line-symmetric shape as shown in the figure, so that an ideal blur is obtained. When the center of gravity position of the shape is set to the midpoint between the center of gravity position of the blur shape of the R image and the center of gravity position of the blur shape of the B image, it can be considered to be an appropriate setting.

The reason why the center of gravity position of the ideal circular blur shape is the center of gravity position of the blur shape of the G image that is the standard image is that the G image is an actually acquired image and the entire pupil of the band limiting filter 12 This is because a natural circular blur shape corresponding to the region is exhibited. Then, when matching the blurring shape of the R image and the B image with the blurring shape of the G image, the blurring of the R image and the B image becomes a shape exhibiting a circular shape equivalent to the blurring of the G image. Since the size of the image is larger than the size of the blur of the R image and the B image, advantages such as easier processing can be obtained by matching the blur shape of the R image and the B image with the blur shape of the G image. It is also for the purpose.

Next, the center of gravity (Ch, Cg) of the ideal circular blur shape set as described above becomes the filter center, and the center of gravity (Ch, Cr) of the blur shape of the R image or the blur shape of the B image. The filter kernel is determined so that the barycentric position (Ch, Cb) is the peak position of the filter coefficient.

The reason why the center of gravity positions of the R image and the B image are brought close is to correct the color shift, and the blur shape of the R image and the B image is approximated by correcting that the blur shape is different for each color. This is because the blur in the color image is made to have a natural shape.

The R filter kernel shown in FIG. 17 is for performing a convolution operation on the R image, and a Gaussian filter is arranged as an example of a blur filter. In this R filter kernel, the peak of the filter coefficient of the Gaussian filter (corresponding approximately to the position of the center of gravity of the filter coefficient) is the position of the center of the kernel (the horizontal line Ch that divides the top and bottom of the kernel into two equal parts) Between the image obtained from the ideal circular blur shape and the R image from the position (Ch, Cg) at which the vertical line Cg that passes through the center of gravity of the ideal circular blur shape intersects the vertical line equally dividing. The filter shape is at a position shifted in phase difference (a position (Ch, Cr) where the horizontal line Ch and the vertical line Cr passing through the center of gravity of the blur shape of the R image intersect).

The B filter kernel shown in FIG. 18 is for performing a convolution operation on the B image, and similarly to the R filter kernel, a Gaussian filter is arranged as an example of a blur filter. In the filter kernel for B, the peak of the filter coefficient of the Gaussian filter (corresponding substantially to the position of the center of gravity of the filter coefficient) is the position of the kernel center (position where the horizontal line Ch and the vertical line Cg intersect (Ch, Cg)). From the position obtained by shifting the phase difference between the image obtained from the ideal circular blur shape and the B image (the position where the horizontal line Ch and the vertical line Cb passing through the center of gravity of the blur shape of the B image intersect (Ch, Cb)).

By causing the R filter kernel and the B filter kernel having such a filter shape to act on the R image and the B image, respectively, it is possible to simultaneously perform color misalignment correction and blur shape correction. .

Next, another example of colorization processing will be described with reference to FIGS. 19 and 20. FIG. 19 is a diagram showing how the R image and B image of the subject located farther than the in-focus position in the color processing of another example, and FIG. 20 shows the color processing of another example. It is a figure which shows the shape of the filter applied with respect to R image and B image.

In this example, the colorization processing is performed by correcting the color misregistration by parallel movement (shift) and correcting the blurred shape by the filtering processing.

That is, first, by performing R shift processing corresponding to the phase difference on the R image shown in FIG. 10, the centroid position of the R image is brought close to the ideal circular blur-shaped centroid position as shown in FIG. In addition, by performing B shift processing corresponding to the phase difference on the B image shown in FIG. 10, the centroid position of the B image is brought close to the ideal circular blur-shaped centroid position as shown in FIG. Let

Thereafter, the same filtering process as shown in FIG. 20 is performed on the R image and the B image to approximate the blur shape of the R image and the B image to an ideal circular blur shape. (See FIG. 16).

Here, the filter kernel shown in FIG. 20 is for performing a convolution operation on the R image and the B image, and a Gaussian filter is arranged as an example of the blur filter. In this filter kernel, the filter coefficient peak of the Gaussian filter (corresponding to the position of the center of gravity of the filter coefficient) is at the kernel center position (position (Ch, Cg) where the horizontal line Ch and the vertical line Cg intersect). It has a shape.

However, since the band limiting filter 12 in which the first area and the second area are symmetric is assumed here as shown in FIG. 3, the same filtering process is applied to the R image and the B image. However, when the first region and the second region are asymmetric, it is needless to say that different filtering processes may be performed on the R image and the B image.

In the above description, a Gaussian filter (circular Gaussian filter) is taken as an example of a blur filter, but the present invention is not limited to this. For example, in the case of the filter shape shown in FIG. 3, as shown in FIGS. 7 and 10, blurring generated in the R image and the B image is semicircular in the vertical direction (that is, shorter in the horizontal direction than in the circular shape). Shape). Therefore, if an elliptical Gaussian filter as shown in FIG. 21, FIG. 22 or the like having the horizontal direction (more generally, the direction in which the phase difference is generated) as the major axis direction is used, the blur is more accurately formed into a circular shape. It is possible to perform a correction process to approach FIG. 21 is a diagram showing the shape of an elliptical Gaussian filter with a large standard deviation in the horizontal direction, and FIG. 22 is a diagram showing the shape of an elliptical Gaussian filter with a small standard deviation in the vertical direction. 21 and 22 show examples in which the peak of the filter coefficient (corresponding to the position of the center of gravity of the filter coefficient) is located at the center of the filter kernel corresponding to FIG. 20, but it corresponds to FIG. 17 and FIG. In this case, it goes without saying that the peak of the filter coefficient (which substantially corresponds to the position of the center of gravity of the filter coefficient) is shifted from the center of the filter kernel.

Further, the blur filter is not limited to the circular Gaussian filter and the elliptic Gaussian filter, and the blur shape of the R image or the B image can be approximated to an ideal circular blur shape. A filter can be widely applied.

FIG. 23 is a flowchart showing the colorization process performed by the color image generation unit 37.

(Step S1)
When this process is started, initialization is performed. In this initial setting, first, an RGB image to be processed (that is, an R image, a G image, and a B image) is read. Next, an R copy image that is a copy of the R image and a B copy image that is a copy of the B image are created.

(Step S2)
Subsequently, a target pixel for performing phase difference detection is set. Here, the target pixel is set to one of the R image and the B image, here, for example, the R image.

(Step S3)
A phase difference with respect to the target pixel set in step S2 is detected. This phase difference is detected by setting a partial area including the target pixel at the center position as an R image as a reference image, and setting a partial area of the same size as a B image as a reference image (see FIG. 26, etc.) The distance calculation unit 39 performs a correlation calculation between the reference image and the reference image while shifting the position of the image in the direction in which the phase difference is generated, and the reference image determined to have the highest correlation value is compared with the reference image. The amount of misalignment between them (note that the sign of the misregistration amount gives information on the direction of misalignment) is the phase difference amount.

The partial area can be set to an arbitrary size. However, in order to stably detect the phase difference, it is preferable to use partial areas of 30 [pixels] or more in the vertical and horizontal directions. As an example, 51 × 51 An area of [Pixel] is given.

Then, specifically, the correlation calculation in the distance calculation unit 39 is performed by a process such as a ZNCC calculation or an SAD calculation for an image that has been previously subjected to filter processing.

First, the correlation calculation by ZNCC is performed based on the following formula 1.

[Equation 1]

Here, I is a partial area of the R image, T is a partial area of the B image (partial area of the same size as I), I (bar) is an average value of I, T (bar) is an average value of T, and M is The horizontal width [pixel] of the partial area, and N is the vertical width [pixel] of the partial area. A ZNCC operation is performed based on Equation 1, and the absolute value | R _ZNCC | of the result is set as a correlation value obtained as a result of the correlation operation.

In addition, when performing SAD calculation on an image that has been previously filtered, filtering processing such as a differential filter represented by a Sobel filter or a bandpass filter such as a LOG filter is first applied to the R image and the B image. Give it. After that, the correlation calculation is performed by the SAD calculation shown in the following formula 2.

[Equation 2]

Here, I ′ is a partial region of the R image after the filtering process is performed, T ′ is a partial region of the B image after the filtering process is performed (partial region of the same size as I ′), and M is a partial region The horizontal width [pixel] of the area, and N is the vertical width [pixel] of the partial area. In this case, R _SAD is a correlation value obtained as a result of the correlation calculation.

(Step S4)
It is determined whether or not the phase difference detection processing for all the target pixels in the image is completed. Then, until the completion, the processing of step S2 and step S3 is repeated while shifting the position of the target pixel. Here, all the target pixels in the image indicate all pixels in the image that can detect the phase difference. It should be noted that pixels that cannot detect the phase difference may be left undetected, or the phase difference amount may be calculated by performing interpolation or the like based on the detection result of the surrounding pixel of interest.

(Step S5)
Next, the target pixel for performing the correction process is set for the pixel for which the phase difference amount is obtained. Here, the pixel position of the target pixel is the same (that is, common) pixel position in the R image and the B image. Hereinafter, as the correction process, as described with reference to FIGS. 17 and 18, a case will be described in which a colorization process using only a filter without translation (shift) is performed.

(Step S6)
In accordance with the phase difference amount of the pixel of interest, the shape of the R image filter for performing the filtering process so that the R image has an ideal circular blur shape is acquired. Here, the relationship between the phase difference amount and the filter shape for the R image is held in advance in the imaging apparatus as a table as shown in Table 1 below, for example. In the example shown in Table 1, the filter shape is determined by determining the size of the filter kernel, the deviation of the Gaussian filter from the center of the filter kernel, the standard deviation σ of the Gaussian filter (this standard deviation σ is the blur of the blur filter) It shows the degree of spread). Therefore, the shape of the R image filter can be acquired by referring to the table based on the phase difference amount of the target pixel.

[Table 1]

(Step S7)
Next, a filtering process is performed on a neighboring region including the target pixel and its neighboring pixels in the R image, and a filter output value at the target pixel is acquired. Then, the acquired filter output value is copied to the target pixel position of the R copy image, and the R copy image is updated.

(Step S8)
In accordance with the phase difference amount of the target pixel, the shape of the B image filter for performing the filtering process so that the B image has an ideal circular blur shape is acquired. Here, the relationship between the phase difference amount and the filter shape for the B image is held in advance in the imaging apparatus as a table as shown in Table 2 below, for example. Also in the example shown in Table 2, the filter shape is determined by the size of the filter kernel, the deviation of the Gaussian filter from the center of the filter kernel, and the standard deviation σ of the Gaussian filter. Therefore, the shape of the B image filter can be acquired by referring to the table based on the phase difference amount of the target pixel.

[Table 2]

(Step S9)
Next, a filtering process is performed on a neighboring region including the target pixel and its neighboring pixels in the B image, and a filter output value at the target pixel is acquired. Then, the acquired filter output value is copied to the target pixel position of the B copy image, and the B copy image is updated.

(Step S10)
It is determined whether or not the filtering process for all the target pixels in the image has been completed. Until the processing is completed, the processing in steps S5 to S9 is repeated while shifting the position of the target pixel.

Thus, when it is determined that the filtering process for all the target pixels has been completed, the R copy image and the B copy image are output as corrected images for the R image and the B image, and the colorization process is terminated.

In step S6 or step S8, instead of using the circular Gaussian filter, when using an elliptical Gaussian filter as shown in FIG. 21 or FIG. 22, the standard deviation value of the elliptical Gaussian filter is set to x. Since the direction and the y direction may be set separately, the parameter table relating to the filter shape held in the imaging apparatus is as shown in Table 3 below, for example, excluding the deviation from the filter kernel center.

[Table 3]

Although not described here, the deviation from the filter kernel center when the elliptical Gaussian filter is used in step S6 is the deviation from the filter kernel center when the elliptical Gaussian filter is used in step S8, for example, as in Table 1. For example, it may be the same as in Table 2.

Furthermore, in the above description, an example in which the shape of the filter corresponding to the phase difference amount is acquired using the table with reference to Table 1 and Table 2 is not limited thereto. For example, the correspondence between each parameter for determining the filter shape and the phase difference amount is held as, for example, an equation, the phase difference amount is substituted into the equation, and the filter shape is determined by calculation or the like. It doesn't matter if you do.

Note that the processing for detecting the phase difference from the image captured through the band limiting filter 12 and the colorization processing for correcting the color misregistration do not necessarily have to be performed in the form of the imaging device, and the image processing device is separate from the imaging device. 41 (for example, a computer that executes an image processing program) may be performed.

FIG. 24 is a block diagram showing a configuration of the image processing apparatus 41.

The image processing apparatus 41 includes a lens mount (not shown), a shutter 21, an image sensor 22, an image pickup circuit 23, an image pickup drive unit 24, and a lens unit 1, which are not shown in FIG. Contrast AF control unit 38 (including AF assist control unit 38a) related to AF control, body side communication connector 35 related to communication with the lens unit 1, strobe control circuit 33 and strobe 34 related to subject illumination, and state of the imaging device The sensor unit 31 for obtaining the information is removed, and a recording unit 42 for recording information input from the interface (IF) 28 is further provided. The recording unit 42 is configured to output information input and recorded via the interface 28 to the image processing unit 25. Information can be recorded from the image processing unit 25 to the recording unit 42. The recording unit 42 is controlled by the system controller 30A (the reference number of the system controller is 30A in accordance with the removal of the contrast AF control unit 38) together with the interface 28. In addition, since a release button and the like for imaging are not necessary, the reference numeral of the operation unit is 32A.

Further, a series of processing in the image processing apparatus 41 is performed as follows, for example. First, an image is captured using an imaging device including the band limiting filter 12 and is recorded on the recording medium 29 as a RAW image output from the imaging circuit 23. Further, information related to the shape of the band limiting filter 12, lens data of the imaging optical system 9, and the like are also recorded on the recording medium 29.

Next, the recording medium 29 is connected to the interface 28 of the image processing apparatus 41, and images and various information are recorded in the recording unit 42. When recording is completed, the recording medium 29 may be removed from the interface 28.

Thereafter, the image and various types of information recorded in the recording unit 42 are read out, and the phase difference is calculated by the distance calculating unit 39 or the color shift by the color image generating unit 37 is performed in the same manner as the imaging device described above. Perform color correction processing.

Thus, the colorized image processed by the image processing device 41 is recorded in the recording unit 42 again. Further, the colorized image recorded in the recording unit 42 is displayed on the display unit or transmitted to an external device via the interface 28. Therefore, in the external device, the colorized image can be used for various purposes.

According to the first embodiment, the center of gravity positions of the R image and the B image are converted into an ideal circular blur shape by using the blur filter itself or by translating (shifting) it before using the blur filter. Since the position of the center of gravity of the R image, the B image, and the G image is corrected so that the positions of the center of gravity of the blur coincide with each other, the color misregistration is reduced. A more preferable image can be obtained for viewing.

At this time, since the blur shape of the R image and the B image is approximated to an ideal circular blur shape by the blur filter, a natural blur-shaped image more preferable for viewing can be obtained.

[Embodiment 2]
FIGS. 25 to 30 show the second embodiment of the present invention, and FIG. 25 is a diagram showing an outline of the colorization processing performed by the color image generation unit 37.

In the second embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

In the present embodiment, the colorization processing in the color image generation unit 37 is different from that in the first embodiment. In other words, in the first embodiment described above, correction of color misregistration is performed using a blur filter, but in the present embodiment, this is performed by image copy addition processing.

Referring to FIG. 25, the concept of colorization processing in the present embodiment when the subject is on the far side from the in-focus position will be described.

When the subject is on the far side from the in-focus position, as shown in FIG. 10, the blur of the R image has a shape in which the left half of the blur of the G image that is an ideal circular blur shape is missing. Yes, the blur of the B image is a shape in which the right half of the blur of the G image, which is an ideal circular blur shape, is missing. According to the color filter 12 (see FIG. 3) used in the second embodiment, an ideal circular blur shape can be obtained for the G image. Therefore, for easy explanation, the ideal circular blur shape is a G image. In this embodiment, the ideal circular blur shape is a circular blur shape that compensates for the half blurring shape lacking the R image and the B image.

Therefore, as shown in FIG. 25, in order to approximate the blur of the R image to an ideal blur shape, the blur of the R image is copied and added to the missing portion of the blur of the R image, and the blur of the B image is Copy addition is added to the missing part of the blurred image.

It is desirable that the shape of the partial area to be copied and added matches the shape of the missing portion of the blur in the R image and the B image, but here, in order to simplify the processing, a rectangular (for example, square) area is used. Yes.

The blur diffusion partial region G1 and the blur diffusion partial region G2 are shown in the circular blur diffusion region in the G image which is an ideal circular blur shape. Here, the blur diffusion partial region G1 and the blur diffusion partial region G2 are symmetrically positioned with respect to the vertical line Cg passing through the center of gravity of the blur diffusion region forming the circular shape of the G image which is an ideal circular blur shape. Are arranged as the same size. In addition, the sizes of the partial areas G1 and G2 are preferably about the size of the radius of the circular blur diffusion area in consideration of fulfilling the function of color misregistration correction.

On the other hand, the blur diffusion partial region R1 shown for the R image is a region having the same size and the same size as the blur diffusion partial region G2. In the blur diffusion region of the R image, the blur diffusion partial region R2 having the same size and the same position as the blur diffusion partial region G1 is insufficient.

Similarly, the blur diffusion partial region B1 shown for the B image is the same size and the same position as the blur diffusion partial region G1. The blur diffusion region B2 of the B image lacks the blur diffusion partial region B2 having the same size and the same position as the blur diffusion partial region G2.

Therefore, a movement amount that completely overlaps the blur diffusion partial region G1 when the blur diffusion partial region G2 is moved in the blur diffusion partial region R1 of the R image (for example, a movement amount about the radius of the circular blur diffusion region G1). The blur diffused partial area R2 of the R image is generated by moving and moving the image (to an R copy image that is a copy of the R image, as will be described later). The blur diffusion partial region R2 is a region corresponding to the blur diffusion partial region G1 (or the blur diffusion partial region B1 of the B image) of the blur diffusion region forming the circular shape of the G image which is an ideal circular blur shape.

Similarly, the blur diffusion partial region B1 of the B image is moved by a movement amount (same as above) that completely overlaps the blur diffusion partial region G2 when the blur diffusion partial region G1 is moved (as will be described later, B The blur diffusion partial area B2 of the B image is generated by performing copy addition (to the B copy image which is a copy of the image). This blur diffusion partial region B2 is a region corresponding to the blur diffusion partial region G2 (or the blur diffusion partial region R1 of the R image) of the blur diffusion region forming the circular shape of the G image which is an ideal circular blur shape.

By such processing, it is possible to compensate for the blur diffusion partial areas that are lacking in the R image and the B image, respectively, and as a result, it is possible to approximate the blur diffusion areas of the G image, the R image, and the B image. Accordingly, the center of gravity of the blur diffusion region of the R image is close to the center of gravity of the blur diffusion region of the G image having an ideal circular blur shape (the proximity includes matching), and the same applies to the B image. The center of gravity of the blur diffusion region is close to the center of gravity of the blur diffusion region of the G image having an ideal circular blur shape.

By performing such processing on the entire R image and B image, a color image with corrected color misregistration can be obtained.

Next, FIG. 26 is a diagram showing a partial region set in the R image and the B image when performing phase difference detection, and FIG. 27 is a diagram showing a state in which the blur diffused partial region of the original R image is copied and added to the R copy image. FIG. 28 is a diagram showing a state in which the blur diffusion partial area of the original B image is copied and added to the B copy image, and FIG. 29 is a diagram showing an example of changing the size of the blur diffusion partial area according to the phase difference amount. Reference numeral 30 denotes a flowchart showing the colorization processing performed by the color image generation unit 37. Description will be made along FIG. 30 with reference to FIGS. 26 to 29 as appropriate.

(Step S21)
When this process is started, initialization is performed. In this initial setting, first, an RGB image to be processed (that is, an R image, a G image, and a B image) is read. Here, when the input image is a Bayer image, it is assumed that the demosaicing process is performed in advance in the image processing unit 25 (however, when the accuracy of the phase difference to be acquired is not required to be high) Or, if you want to prioritize speed over accuracy, or if you want to reduce processing load, etc.), R image, G image, and B image (that is, demosaicing process) by simply separating the Bayer image into each color. Not each color image)).

Next, in order to correct the phase difference between the R image and the B image, color differences Cr and Cb are used instead of using the R image and the B image itself. This is because a general subject has an advantage that the correction process can be performed stably because the variation of the pixel value is smoother in the color difference image than in the RGB image (however, R The image and the B image may be used as they are.In this case, in the following processing, the reading of Cr → R and Cb → B may be performed).

That is, for example, based on the following Equation 3, the color difference amount between the R image and the G image is calculated to generate a Cr image that is a color difference image, and the color difference amount between the B image and the G image is calculated to calculate the color difference image. A certain Cb image is generated.

[Equation 3]
Cr = RG
Cb = BG
As other calculation methods for calculating the color difference signals Cr and Cb (and the luminance signal Y) from the RGB signals,
[Equation 4]
Y = 0.29900R + 0.58700G + 0.11400B
Cr = 0.50000R-0.41869G-0.0811B
Cb = −0.16874R−0.33126G + 0.50000B
Is widely known, instead of Equation 3, Equation 4 may be used.

Next, a Cr copy image Cr1 (see FIG. 27) that is a copy image of the original Cr image Cr0 and a Cb copy image Cb1 (see FIG. 28) that is a copy image of the original Cb image Cb0 are created. Further, a Cr count image and a Cb count image having the same size as the Cr copy image Cr1 and the Cb copy image Cb1 are generated (here, in these count images, the initial value of the pixel value is set to 1 for all pixels). .

(Step S22)
Subsequently, a partial region for performing phase difference detection is set. Here, the partial area is set to one of the R image and the B image, here, for example, the R image.

(Step S23)
A phase difference with respect to the partial region set in step S22 is detected. This phase difference is detected by using the partial area set in the R image as a standard image and the partial area of the B image having the same size as the standard image as a reference image as shown in FIG. By doing so, phase difference detection is performed between the R image and the B image.

(Step S24)
Based on the phase difference amount obtained by the processing in step S23, the radius of the circular blur of the G image corresponding to the ideal blur shape (or the radius of the semicircular blur of the R image and the B image) ( Here, the radius is given as an example, but it may be a diameter or any other amount as long as it can represent the ideal circular blur size. Here, the relationship between the phase difference amount and the blurring radius of the G image that is an ideal circular blur shape is held in advance in the imaging apparatus as a table, a mathematical expression, or the like. Therefore, the blur radius can be acquired by referring to a table or performing a calculation using a mathematical formula based on the phase difference amount. Note that the process of step S24 may be omitted, and a simple method using the phase difference amount acquired in step S23 in place of the blur radius may be applied. In this case, the relationship between the phase difference amount and the blur radius need not be held in the imaging apparatus in advance.

(Step S25)
Next, a partial region is read from the original Cr image Cr0, shifted by a predetermined amount corresponding to the phase difference detected in step S23, and then copied and added to the Cr copy image Cr1. Here, the predetermined amount for shifting the partial area is an amount including the shifting direction, and the size thereof is, for example, the blur radius acquired in step S24. The reason why the Cr copy image Cr1 is created in the above-described step S21 is that it is necessary to hold the original Cr image Cr0 separately from the Cr copy image Cr1 whose pixel value is changed by copy addition ( The same applies to the Cb copy image Cb1). However, it is not necessary to prepare a copy image when, for example, partial areas are processed in parallel operation instead of sequentially processing in the order of raster scanning.

(Step S26)
Subsequently, +1 is added to the region of “the position where the copy addition process has been performed in step S25” of the Cr count image so that the number of times of addition can be understood. This Cr count image is used to perform pixel value normalization processing in the subsequent step S30.

(Step S27)
Further, the partial area is read from the same position as “the position copied to the Cr copy image Cr1 in step S25” in the original Cb image Cb0, and the copy source from the original Cr image Cr0 in “step S25” of the Cb copy image Cb1. A copy is added to “the position from which data was acquired”. Thus, the predetermined amount for shifting the Cb image has the same absolute value as the predetermined amount for shifting the Cr image, but the direction is reversed.

(Step S28)
Then, the number of times of addition can be seen in the region of “the position where the copy source data was acquired from the original Cr image Cr0 in step S25” (that is, the position where the copy addition process was performed in step S27) of the Cb count image. Add +1 to. This Cb count image is also used to perform pixel value normalization processing in the subsequent step S30.

In step S25 and step S27 described above, the copy addition process is performed for each partial area of the image, but this partial area may be the same as the partial area for which the phase difference detection process was performed in step S23. On the other hand, it may be a partial area having a size different from that of the phase difference detection process.

In addition, the size of the partial area used for the copy addition process may be constant for the entire image (that is, global size), but may be different for each partial area set in the image (that is, local size). (Even if it is a big size).

For example, the size of the partial region used in steps S25 to S28 may be changed as shown in FIG. 29 according to the phase difference amount detected in step S23.

In the example shown in FIG. 29, when the phase difference amount is 0, the vertical size and the horizontal size of the partial region are both 1, and the partial region is 1 pixel. In this case, since the phase difference amount is 0, the above-described copy addition process is not performed, and no process is substantially performed. Therefore, the processing may be branched depending on whether or not the phase difference amount is 0, and no processing may be performed when the phase difference amount is 0.

In the example shown in FIG. 29, the size of the partial area is increased in proportion to the phase difference amount. Since the inclination of the straight line at this time is appropriately set according to the configuration of the optical system, a specific scale is not shown in FIG.

Note that FIG. 29 shows an example in which the relationship between the phase difference amount and the size of the partial area is proportional, but of course, the relationship is not limited to the proportional relationship. For example, for subjective image quality evaluation. Accordingly, the design may be made so that the size of the partial region with respect to the phase difference amount is appropriate.

Further, the amount of diffusion (a point spread amount of PSF (Point Spread Function)) when the light beam from the point light source is blurred is not necessarily uniform within the blurred area. For example, it is conceivable that the amount of diffusion is smaller (that is, the luminance is lower) in the peripheral portion of the blur than in the central portion. Therefore, when performing copy addition of partial areas as described above, a weighting factor corresponding to the amount of blur diffusion may be multiplied. For example, each pixel in the peripheral part of the partial region is multiplied by a weighting factor of 1/2, each pixel in the central part of the partial region is multiplied by one weighting factor, and then copy addition is performed. At this time, also in the count images in step S26 and step S28, 1/2 is added to the peripheral part of the partial area, and 1 is added to the central part of the partial area.

(Step S29)
Thereafter, it is determined whether or not the processing for all the partial areas in the image is completed. Until the processing is completed, the processing in steps S22 to S28 is repeated while shifting the position of the partial area. Here, an arbitrary value can be set as the step for shifting the partial area, but it is preferably a value smaller than the width of the partial area.

(Step S30)
Thus, if it is determined in step S29 that the processing for all the partial areas has been completed, the normalized Cr by dividing the pixel value of the Cr copy image by the pixel value of the Cr count image for each same pixel position. A copy image is obtained, and a normalized Cb copy image is obtained by dividing the pixel value of the Cb copy image by the pixel value of the Cb count image.

(Step S31)
Then, an R image, a B image, and a G image are generated using the G image (or Y image) and the Cr copy image and the Cb copy image normalized in step S30. Here, when Equation 3 is used to calculate the color difference image,
[Equation 5]
R = Cr + G
B = Cb + G
Are used to generate an R image and a B image (the G image in step S21 is used as it is).

In addition, when Equation 4 is used to calculate the color difference image,
[Equation 6]
R = Y + 1.40200Cr
G = Y-0.71414Cr-0.44414Cb
B = Y + 1.77200Cb
Is used to generate an R image, a B image, and a G image.

The RGB image calculated in step S31 is an image obtained by the colorization process in the color image generation unit 37.

According to the second embodiment, the same effects as those of the first embodiment can be obtained by correcting the color misregistration by the image copy addition process.

Further, when performing copy addition, there is an advantage that the processing load can be reduced as compared with the case of filtering processing. Therefore, the cost of the processing circuit and the power consumption can be reduced.

[Embodiment 3]
FIGS. 31 to 34 show Embodiment 3 of the present invention, and FIG. 31 is a diagram showing an outline of a PSF (Point Spread Function) table for each color according to the phase difference amount, and FIG. FIG. 33 is a diagram showing an outline of colorization processing performed by the color image generation unit 37, FIG. 33 is a diagram showing an overview of colorization processing with blur amount control performed by the color image generation unit 37, and FIG. 34 is a color image generation unit. 37 is a flowchart showing a colorization process performed by 37.

In the third embodiment, the same parts as those in the first and second embodiments are denoted by the same reference numerals and the description thereof is omitted as appropriate, and only different points will be mainly described.

In the present embodiment, the colorization process in the color image generation unit 37 is different from those in the first and second embodiments. That is, according to the present embodiment, the colorization process includes a restoration process (inverse filtering process) for restoring a blurred image to a non-blurred image, and a circular blur shape corresponding to the subject distance for the restored non-blurred image. This is performed by combining filtering processing to be obtained.

When the band limiting filter 12 as shown in FIG. 3 is used, the PSF (Point Spread Function) of each color of RGB changes as shown in FIG. 31 according to the phase difference amount. In FIG. 31, the dark part where the light from the point light source does not reach is hatched.

For example, PSFr, which is the PSF of the R image, shows a larger semicircular shape as the absolute value of the phase difference increases, and converges to one point when the phase difference is 0, that is, at the in-focus position. Further, when the distance is closer than the in-focus position, the shape is a left semicircular shape as shown in the upper half of FIG. 31, but when the distance is farther than the in-focus position, it is shown in the lower half of FIG. It becomes a right semicircle shape.

Similarly, PSFb, which is the PSF of the B image, shows a larger semicircular shape as the absolute value of the phase difference amount increases, and converges to 1 point at the phase difference amount of 0, that is, at the in-focus position. Further, when the distance is shorter than the in-focus position, the shape is a right semicircle as shown in the upper half of FIG. 31, but when the distance is farther than the in-focus position, the lower half of FIG. It becomes a left semicircle shape.

Furthermore, PSFg, which is the PSF of the G image, shows a larger circular shape as the absolute value of the phase difference amount increases, and converges to one point when the phase difference amount is 0, that is, at the in-focus position. Further, the PSFg is always an ideally defocused circular shape regardless of whether it is closer or farther than the in-focus position except when it converges to one point.

A PSF table for each color corresponding to the phase difference amount as shown in FIG. 31 is assumed to be stored in advance in, for example, a non-illustrated nonvolatile memory in the color image generation unit 37 (however, in FIG. 1). In the case of the lens-interchangeable image pickup apparatus as shown, of course, the table stored in the lens control unit 14 may be received and used by communication).

The restoration processing and filtering processing using such PSF will be described with reference to FIG. FIG. 32 shows an example of the case where the subject is farther than the in-focus position, but the same processing can be applied when the subject is closer than the in-focus position.

First, by performing the inverse PSFr operation PSFr ⁻¹ on the R image, the right semicircular blur is converted into a non ^- blurred image in which the point light source converges to one point (either the restored first image or the restored second image). Either). Similarly, by performing the inverse PSFb operation PSFb ⁻¹ on the B image, the left semicircular blur is converted into a blur-free image in which the point light source converges to one point (the restored first image and the restored second image Convert to either one). By such processing, restoration processing of the R image and the B image is performed.

Next, a PSF for generating an ideal circular blurred shape is applied to the restored R image and the restored B image. Specifically, the PSF to be actuated here is PSFg, which is a PSF for a G image corresponding to an ideal circular blurred shape. Thereby, a circular blur similar to the G image is generated so as to compensate for the semicircular shape lacking the R image and the B image.

The concrete processing is performed as follows.

First, it is assumed that the blur PSF of the R image at a certain pixel position is Pr1, the blur PSF at the same pixel position of the B image is Pb1, and the blur PSF at the same pixel position of the G image is Pg1. As shown in FIG. 31, the blurred PSF differs depending on the phase difference between the R image and the B image, and the phase difference basically differs for each pixel position. Therefore, the PSF is different for each pixel position. To be determined. Further, it is assumed that the PSF is defined for a partial region including a plurality of neighboring pixels with the pixel position of interest at the center (see FIG. 31).

When the processing is started, the phase difference obtained for the pixel of interest is acquired, and by referring to a table as shown in FIG. 31, Pr1 which is a PSF centered on the pixel of interest in the R image and the B image Pb1 that is the PSF centered on the pixel of interest at the same position in Pb1 and Pg1 that is the PSF centered on the pixel of interest at the same position in the G image are acquired.

Next, for the obtained Pr1, Pb1, and Pg1, the partial region r centered on the target pixel in the R image, and the partial region b centered on the target pixel in the B image, the following Expression 7 is shown. As described above, the two-dimensional Fourier transform FFT2 is performed to obtain values PR1, PB1, PG1, R, and B after the respective transformations.

[Equation 7]
PR1 = FFT2 (Pr1)
PB1 = FFT2 (Pb1)
PG1 = FFT2 (Pg1)
R = FFT2 (r)
B = FFT2 (b)
Next, as shown in Equation 8 below, R is divided by PR1, B is divided by PB1, and restoration processing is performed, and each is multiplied by a value for generating an ideal blurred shape. The filtering process is performed by Specifically, here, the value for generating an ideal blurred shape is PG1 described above. Then, two-dimensional inverse Fourier transform IFFT2 is applied to the result, and an R image r ′ and a B image b ′ from which the same blur as the G image corresponding to the ideal blur shape is obtained are calculated.

[Equation 8]
r ′ = IFFT2 (R × PG1 / PR1)
b ′ = IFFT2 (B × PG1 / PB1)
In order to make the restoration process by the inverse filtering process of Expression 8 more stable, the restoration amount of the restoration process may be controlled as shown in Expression 9 below (Wiener filter method).

[Equation 9]
r ′ = IFFT2 (R × PG1 / PR1 × | PR1 | ² / (| PR1 | ² + Γ))
b ′ = IFFT2 (B × PG1 / PB1 × | PB1 | ² / (| PB1 | ² + Γ))
Here, Γ in Equation 9 is an arbitrary constant appropriately set according to the shapes of PR1 and PB1.

By adopting processing such as Equation 9, noise amplification during restoration processing is suppressed, and more preferable R and B images can be generated.

As a method for setting Γ, for example, the relationship between the absolute values | PR1 | and | PB1 | of the frequency coefficient of PR1 and the preferable values of Γ with respect to the absolute values | PR1 | and | PB1 | A method of designating Γ for each frequency coefficient based on this relationship can be considered. As another method, a configuration in which the amount of noise included in an image is estimated based on various parameters (ISO value, focal length, aperture value, etc.) of the imaging device, and Γ is changed according to the estimated amount of noise. (For example, Γ = 0.01 for ISO 200 and Γ = 0.04 for ISO 800) may be employed.

The restoration process and the filtering process as described above are performed for each partial area of the image. When the correction process is completed in a certain partial area, the partial area is designated by slightly shifting the position, and the designated partial area is again displayed. Similar restoration processing and filtering processing are performed. By repeatedly performing such processing, restoration processing and filtering processing are performed on the entire region of the image. Thereafter, with respect to the pixel position processed in duplicate, the sum of a plurality of corrected pixel values processed at the pixel position is divided by the number of corrections and averaged to obtain a normalized corrected image.

In addition, it is preferable that the size of the partial area on which the restoration process and the filtering process are performed is larger than the shape of the blur. Therefore, it is conceivable to adaptively change the size of the partial region in accordance with the size of the blur. Alternatively, when it is known in advance in which range the shape of the blur changes according to the phase difference, a partial region having a size larger than the maximum size of the blur may be used in a fixed manner.

Next, with reference to FIG. 34, the flow of colorization processing by the above-described restoration processing and filtering processing performed by the color image generation unit 37 will be described.

(Step S41)
When this process is started, initialization is performed. In this initial setting, first, an RGB image to be processed (that is, an R image, a G image, and a B image) is read. Next, an R copy image that is a copy of the R image and a B copy image that is a copy of the B image are created.

Subsequently, an R count image and a B count image having the same size as the R copy image and the B copy image are generated (here, in these count images, the initial value of the pixel value is set to 1 for all pixels).

(Step S42)
Subsequently, a partial region for performing phase difference detection is set. Here, the partial area is set to one of the R image and the B image, here, for example, the R image.

(Step S43)
A phase difference with respect to the partial region set in step S42 is detected. This phase difference is detected by using the partial area set in the R image as a standard image and the partial area of the B image having the same size as the standard image as a reference image, as shown in FIG. By performing the above, phase difference detection is performed between the R image and the B image.

(Step S44)
Based on the phase difference amount obtained by the processing in step S43, the radius of the circular blur of the G image corresponding to the ideal blur shape (or the radius of the semicircular blur of the R image and the B image) can be obtained. Obtained in the same manner as in step S24 described above.

(Step S45)
Next, the restoration process and the filtering process as described above are performed on the partial region specified in step S42 of the original R image. The processing result obtained in this way is copied and added to the same position as the partial area of the original R image in the R copy image. Here, an example in which the partial area for performing the colorization process is the same as the partial area for performing the phase difference detection will be described, but a different area may be set as a matter of course, as described above. Alternatively, a partial area having an adaptive size according to the detected phase difference may be used (the same applies to the B image described later).

(Step S46)
Subsequently, +1 is added to the “partial region designated in step S42” of the R count image so that the number of times of addition can be understood. This R count image is used to perform pixel value normalization processing in the subsequent step S50.

(Step S47)
Further, the restoration process and the filtering process as described above are performed on the partial region specified in the above-described step S42 of the original B image. The processing result obtained in this way is copied and added at the same position as the partial area of the original B image in the B copy image.

(Step S48)
Then, 1 is added to the “partial region designated in step S42” of the B count image so that the number of times of addition can be understood. This B count image is also used to perform pixel value normalization processing in the subsequent step S50.

(Step S49)
Thereafter, it is determined whether or not the processing for all the partial areas in the image is completed. Then, the processes in steps S42 to S48 are repeated while shifting the position of the partial area until the process is completed. Here, an arbitrary value can be set as the step for shifting the partial area, but it is preferably a value smaller than the width of the partial area.

(Step S50)
Thus, if it is determined in step S49 that the processing for all the partial areas has been completed, the R value normalized by dividing the pixel value of the R copy image by the pixel value of the R count image for each same pixel position. A copy image is obtained, and a normalized B copy image is obtained by dividing the pixel value of the B copy image by the pixel value of the B count image. The RGB image calculated in step S50 is an image obtained by the colorization process in the color image generation unit 37.

Thus, when the process of step S50 is completed, the process shown in FIG. 34 is terminated.

In the above description, restoration processing and filtering processing are performed after transforming from real space to frequency space using Fourier transform. However, the present invention is not limited to this, and restoration processing or filtering processing (for example, MAP estimation processing) in real space may be applied.

In the above description, the blurring shape of the R image and the B image is matched with the blurring shape of the G image, but in addition to this, the blur amount control may be performed.

In this case, as shown in FIG. 33, first, the inverse operation PSFr ^-1 of PSFr the R image, the inverse operation PSFB ^-1 of PSFB the B image, restoring the first image and the restored second image through each In addition, the inverse of the PSFg PSFg ⁻¹ is performed on the G image to convert the circular blur into a blur-free image in which the point light source converges to one point (restored third image). Thereby, the R image, the B image, and the G image are restored.

Next, PSF for generating ideal circular blur is applied to the restored R image, B image, and G image. Here, PSF′g, which is a desired PSF for the G image, is caused to act as a PSF for generating an ideal circular blur. Thereby, a circular blur having a desired size can be generated in the R image, the B image, and the G image, and the amount of blur can be controlled.

Specifically, by referring to a table as shown in FIG. 31, a desired Pg1 ′ is further acquired as a PSF centered on the pixel of interest at the same position in the G image.

Next, in addition to the processing of Equation 7 described above, two-dimensional Fourier is obtained as shown in Equation 10 below for the acquired Pg1 ′ and the partial region g centered on the pixel of interest in the G image. A conversion FFT2 is performed to obtain converted values PG1 ′ and G.

[Equation 10]
PG1 ′ = FFT2 (Pg1 ′)
G = FFT2 (g)
Then, as shown in Equation 11 below, R is divided by PR1, B is divided by PB1, G is divided by PG1, restoration processing is performed, and each is multiplied by PG1 ′ to perform filtering processing. The result is subjected to a two-dimensional inverse Fourier transform IFFT2 to calculate an R image r ″, a B image b ″, and a G image g ″ from which a desired amount of blur has been obtained.

[Equation 11]
r ″ = IFFT2 (R × PG1 ′ / PR1)
b ″ = IFFT2 (B × PG1 ′ / PB1)
g ″ = IFFT2 (G × PG1 ′ / PG1)
Note that, similarly to the above-described equation 9, a Wiener filter method as shown in the following equation 12 may be employed instead of the equation 11.

[Equation 12]
r ″ = IFFT2 (R × PG1 ′ / PR1 × | PR1 | ² / (| PR1 | ² + Γ))
b ″ = IFFT2 (B × PG1 ′ / PB1 × | PB1 | ² / (| PB1 | ² + Γ))
g ″ = IFFT2 (G × PG1 ′ / PG1 × | PG1 | ² / (| PG1 | ² + Γ))
Here, Γ is an arbitrary constant appropriately set according to the shapes of PR1, PB1, and PG1.

According to the third embodiment, the same effects as those of the first and second embodiments can be obtained by performing color shift correction by the restoration processing and filtering processing using PSF.

Furthermore, if a restoration process is performed for the G image, and then a filtering process using an arbitrary PSF for the G image is performed on the R image, the B image, and the G image, the amount of blur of the RGB color image is controlled as desired. It is also possible to do.

According to each embodiment of the present invention, since the positional deviation of the center of gravity position of the blur between the R image and the B image is corrected based on the phase difference amount between the R image and the B image, the R image and the B image are corrected. The color misalignment can be eliminated. At this time, by aligning the gravity center position of the blur of the R image and the B image with the gravity center position of the blur of the G image, the color deviation of the R image, the B image, and the G image can also be eliminated. In addition, since all blur shapes of the R image, the B image, and the G image are matched (identified), the color shift can be further improved. In addition, since all the blur shapes of the R image, the B image, and the G image are processed so as to have the same ideal circular shape, a natural and preferable image suitable for viewing can be obtained.

In all the embodiments described above, it is not necessary to set an ideal blur shape based on an actually acquired image such as a G image, and an ideal blur shape set virtually may be used. Absent.

Further, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

This application is filed on the basis of the priority claim of Japanese Patent Application No. 2011-148592 filed in Japan on July 4, 2011, and the above disclosure is disclosed in the present specification, claims, It shall be cited in the drawing.

Claims

A color imaging device that receives and photoelectrically converts at least light in the first band and light in the second band, and generates a first image signal and a second image signal;
An imaging optical system that forms a subject image on the imaging device;
It is arranged on the optical path of the imaging light flux from the subject side of the imaging optical system to the imaging device, and blocks the light in the first band with respect to light that attempts to pass through a part of the pupil region of the imaging optical system. The first band is limited to allow the second band light to pass through, and the second band light is blocked with respect to the light that is about to pass through another part of the pupil region of the imaging optical system. A band-limiting filter that performs a second band limitation that allows the light to pass through;
A calculation unit for calculating a phase difference amount based on the first image signal and the second image signal;
Based on the phase difference amount calculated by the calculation unit, an image correction unit that corrects a positional deviation between the blur center of gravity of the first image signal and the blur center of gravity of the second image signal;
An imaging apparatus comprising:
Further comprising setting means for setting a center of gravity position of an ideal blur shape for the first and second image signals;
The image correction unit corrects misalignment by bringing the center of gravity of the blur of the first image signal and the center of gravity of the blur of the second image signal close to the center of gravity of the ideal blur shape. The imaging apparatus according to claim 1, wherein:
The setting means has a constant geometric position relationship between the ideal gravity center position of the blur shape, the blur gravity center position of the first image signal, and the blur gravity center position of the second image signal. The imaging apparatus according to claim 2, wherein the imaging apparatus is set to be
The imaging device further receives a third band of light to generate a third image signal,
The band limiting filter allows the light of the third band to pass in the first band limitation and the second band limitation,
The imaging apparatus according to claim 2, wherein the setting unit sets a center of gravity position of the ideal blur shape to a center of gravity position of a blur of the third image signal.
The image correction unit
Causing a first filter kernel that approximates the blur shape of the first image signal to the ideal blur shape to act on the first image signal;
By causing the second image signal to act on a second filter kernel that approximates the blur shape of the second image signal to the ideal blur shape,
The imaging apparatus according to claim 2, wherein the blur shape is approximated.
The imaging apparatus according to claim 5, wherein the first filter kernel and the second filter kernel are configured as a blur filter.
The image correction unit
The filter coefficient of the blur filter is an amount corresponding to the amount of the phase difference in the first filter kernel in the direction of deviation of the center of gravity of the first image signal from the center of gravity of the ideal blur shape. The center of gravity position of is shifted from the position of the kernel center,
The filter coefficient of the filter is set to the second filter kernel by an amount corresponding to the phase difference amount in the direction of deviation of the center of gravity of the second image signal from the center of gravity of the ideal blur shape. By shifting the position of the center of gravity from the position of the kernel center,
The imaging apparatus according to claim 6, wherein the positional deviation is corrected.
The image correction unit
The first filter kernel and the second filter kernel are filters in which a centroid position of a filter coefficient of the blur filter is located at a kernel center;
Before the first filter kernel is applied to the first image signal, the first image kernel is set such that the position of the center of gravity of the blur of the first image signal matches the position of the center of gravity of the ideal blur shape. Shift the signal,
Before applying the second filter kernel to the second image signal, the second image signal is set so that the position of the center of gravity of the blur of the second image signal matches the position of the center of gravity of the ideal blur shape. By shifting the signal,
The imaging apparatus according to claim 6, wherein the positional deviation is corrected.
The image correction unit
Shifting the first image signal so that the position of the center of gravity of the blur of the first image signal is close to the position of the center of gravity of the ideal blur shape;
By shifting the second image signal so that the position of the center of gravity of the blur of the second image signal is close to the position of the center of gravity of the ideal blur shape,
The imaging apparatus according to claim 2, wherein the positional deviation is corrected.
The first band is one of a red (R) color band and a blue (B) color band, and the second band is the other of a red (R) color band and a blue (B) color band. ,
The image correction unit includes a first color difference image signal having a high contribution ratio of one of the R component and the B component, and a second color difference image signal having a high contribution ratio of the other of the R component and the B component. And
Instead of shifting the first image signal, shifting the first color difference image signal,
By shifting the second color difference image signal instead of shifting the second image signal,
The imaging apparatus according to claim 9, wherein the positional deviation is corrected.
The image correction unit
The inverse of PSF (Point Spread Function) corresponding to the phase difference of the imaging optical system with respect to the first image signal is performed on the first image signal, so that there is no blur or large blur. Create a restored first image signal with reduced size,
Reconstructed second image signal with no blur or reduced blur size by performing inverse calculation of PSF according to the phase difference of the imaging optical system with respect to the second image signal on the second image signal By creating
The imaging apparatus according to claim 2, wherein the positional deviation is corrected.
A color imaging device that receives and photoelectrically converts at least light in the first band and light in the second band, and generates a first image signal and a second image signal;
An imaging optical system that forms a subject image on the imaging device;
It is arranged on the optical path of the imaging light flux from the subject side of the imaging optical system to the imaging device, and blocks the light in the first band with respect to light that attempts to pass through a part of the pupil region of the imaging optical system. The first band is limited to allow the second band light to pass through, and the second band light is blocked with respect to the light that is about to pass through another part of the pupil region of the imaging optical system. A band-limiting filter that performs a second band limitation that allows the light to pass through;
An image processing device for processing an image obtained by an imaging device having
A calculation unit for calculating a phase difference amount based on the first image signal and the second image signal;
Based on the phase difference amount calculated by the calculation unit, an image correction unit that corrects a positional deviation between the blur center of gravity of the first image signal and the blur center of gravity of the second image signal;
An image processing apparatus comprising: