JP2013044806A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus with manufacturing cost rise suppressed, enabling easy confirmation of a focal part and a non-focal part by visual observation.SOLUTION: An imaging apparatus includes: a color image pick-up element 22 for generating an RGB image; an imaging optical system 9 for imaging a subject image in the image pick-up element 22; a band-limiting filter 12 which is arranged on an optical path of photographing luminous flux and which performs first band limitation for passing light of R and G therethrough by cutting off light of B trying to pass through a first partial pupil and second band limitation for passing the light of B and G therethrough by cutting off the light of R trying to pass through a second partial pupil; and a display part 27 which displays a phase difference image indicating phase difference between an R image and a B image.

Description

  The present invention relates to an imaging apparatus capable of confirming a focused state of a subject based on an image obtained from an imaging element.

  Conventionally, various techniques for measuring the distance from the imaging device to the subject have been proposed.For example, an active distance measurement method for performing distance measurement by irradiating illumination light and receiving reflected light from the subject, A focus lens is used so that the contrast of the image acquired by the imaging device itself is increased by a method of performing distance measurement based on the principle of triangulation from images acquired by a plurality of imaging devices (for example, stereo cameras) arranged with a base line length. Various methods such as a contrast AF method for driving have been proposed.

  However, the active distance measurement method requires a dedicated member for distance measurement such as a light projection device for distance measurement, and the triangular distance measurement method requires a plurality of image pickup devices. Will increase. On the other hand, since the contrast AF method uses an image acquired by the imaging apparatus itself, a dedicated distance measuring member or the like is not required. However, the contrast value peak is obtained by performing multiple imaging while changing the position of the focus lens. Therefore, it takes time to search for a peak corresponding to the in-focus position, and it is difficult to perform high-speed AF.

  Under such a background, as a technology to acquire distance information while suppressing an increase in size and cost, a light beam that passes through the pupil of the lens is divided into a plurality of light beams, and a light beam that passes through one pupil region in the lens Has been proposed to obtain distance information to the subject by performing a correlation operation between the pixel signal obtained from the pixel signal and the pixel signal obtained from the light beam that has passed through another pupil region in the lens. Yes.

  As a technique for simultaneously acquiring such pupil division images, for example, there is a technique for disposing a light shielding plate (mask) on a pixel for distance detection. For example, Japanese Patent Application Laid-Open No. 2009-147665 discloses a focus detection pixel by forming a light-shielding film that can perform pupil division on a part of pixels of an image sensor, and is based on an image obtained from the focus detection pixel. A technique for acquiring a phase difference, generating a split image, and also generating a planar image is described.

  As a technique for performing pupil division without using a light-shielding plate (mask), for example, Japanese Patent Laid-Open No. 2001-174696 includes a pupil color division filter having different spectral characteristics for each partial pupil in an imaging optical system. In addition, a technique is described in which a subject image from a photographic optical system is received by a color image sensor to perform pupil division by color. In other words, the image signal output from the color image sensor is color-separated, and the relative shift amount between the same subject on each color image is detected, so that it is shifted from the in-focus position to the short distance side or the long distance Two pieces of focusing information, that is, a focusing shift direction that is shifted to the side and a focusing shift amount that is a shift amount from the in-focus position in that direction, are acquired.

JP 2009-147665 A JP 2001-174696 A

  The technique described in the above Japanese Patent Application Laid-Open No. 2009-147665 needs to form a light-shielding film on the image sensor while aligning with the pixel using a fine processing technique, and uses a general-purpose image sensor as it is. Cannot be produced, resulting in an increase in manufacturing costs.

  Further, the technique described in Japanese Patent Application Laid-Open No. 2001-174696 describes only detection of the phase difference amount, and the user can determine which part of the subject is in focus and which part is out of focus. There is no disclosure of technology for visual confirmation.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an imaging apparatus with reduced manufacturing costs, in which a focused part and a non-focused part can be easily confirmed visually.

  In order to achieve the above object, an imaging device according to one embodiment of the present invention receives and photoelectrically converts light in a plurality of wavelength bands, and performs first conversion on the first image in the first band and the second in the second band. A color imaging device that generates a second image and a third image in the third band, an imaging optical system that forms a subject image on the imaging device, and imaging that reaches the imaging device via the imaging optical system The second band is disposed on the optical path of the light beam and blocks the light in the first band in the photographing light beam that attempts to pass through the first region that is a part of the pupil region of the imaging optical system. A first band limitation that allows light in the band and the third band to pass through, and a second band in the imaging light flux that attempts to pass through a second region that is another part of the pupil region of the imaging optical system And a second band limitation that blocks the light of the first band and allows the light of the first band and the third band to pass. Cormorant a band limiting filter, which includes a display unit for displaying a phase difference image illustrating the phase difference between the first image and the second image.

  According to the imaging apparatus of the present invention, the manufacturing cost is suppressed, and the in-focus portion and the out-of-focus portion can be easily confirmed visually.

The block diagram which shows the structure of the imaging device in 3D Example 1 which concerns on embodiment of this invention. The figure for demonstrating the pixel arrangement | sequence of the image pick-up element in 3D Example 1 which concerns on the said embodiment. The figure for demonstrating one structural example of the band-limiting filter in 3D Example 1 which concerns on the said embodiment. The top view which shows the mode of a subject light beam condensing when imaging the to-be-photographed object in 3D Example 1 which concerns on the said embodiment. The top view which shows the mode of a subject light beam condensing at the time of imaging the to-be-photographed object in 3D Example 1 which concerns on the said embodiment in the short distance side from an in-focus position. The figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object in the 3D Example 1 which concerns on the said embodiment in the short distance side from a focus position. The figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object for every color component in 3D Example 1 which concerns on the said embodiment for a short distance side. The top view which shows the mode of a subject light beam condensing when imaging the to-be-photographed object in the 3D Example 1 which concerns on the said embodiment in the far distance side from a focus position. The figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object in the 3D Example 1 which concerns on the said embodiment in the far distance side from a focusing position. The figure which shows the shape of the blur formed with the light from one point on the to-be-photographed object for every color component in 3D Example 1 which concerns on the said embodiment for a far distance side. In 3D Example 1 which concerns on the said embodiment, the figure which shows the mode of an image when the to-be-photographed object and the to-be-photographed object in the short distance and far distance are imaged. The figure which shows the 1st modification of the band-limiting filter in 3D Example 1 which concerns on the said embodiment. The figure which shows the 2nd modification of the band-limiting filter in 3D Example 1 which concerns on the said embodiment. The figure which shows the 3rd modification of the band-limiting filter in 3D Example 1 which concerns on the said embodiment. The figure which shows the 4th modification of the band-limiting filter in 3D Example 1 which concerns on the said embodiment. In 3D Example 1 which concerns on the said embodiment, the figure which shows the outline | summary of the blur shape after colorization process. The figure which shows the shape of the filter for R applied to the R image of the to-be-photographed object at the far side in the colorization process of 2D Example 1 of 3D Example 1 which concerns on the said embodiment. The figure which shows the shape of the filter for B applied in the colorization process of 2D Example 1 of 3D Example 1 which concerns on the said embodiment with respect to the to-be-photographed object's B image far from a focusing position. In 3D Example 1 which concerns on the said embodiment, the figure which shows the mode of the shift of the R image of the to-be-photographed object and B image in the far side from the focus position in the colorization process of the modification of 2D Example 1. FIG. The figure which shows the shape of the filter applied with respect to R image and B image in the colorization process of the modification of 2D Example 1 of 3D Example 1 which concerns on the said embodiment. The figure which shows the shape of the elliptical Gaussian filter which enlarged the standard deviation of the horizontal direction in 2D Example 1 of 3D Example 1 which concerns on the said embodiment. The figure which shows the shape of the elliptical Gaussian filter which made small the standard deviation of the vertical direction in 2D Example 1 of 3D Example 1 which concerns on the said embodiment. The flowchart which shows the colorization process performed by the color image generation part in 2D Example 1 of 3D Example 1 which concerns on the said embodiment. The block diagram which shows the structure of the image processing apparatus in 2D Example 1 of 3D Example 1 which concerns on the said embodiment. The figure which shows the outline | summary of the colorization process performed by the color image generation part in 2D Example 2 of 3D Example 1 which concerns on the said embodiment. The figure which shows the partial area | region set to R image and B image in 2D Example 2 of 3D Example 1 which concerns on the said embodiment, when performing a phase difference detection. The figure which shows a mode that copy addition of the blur diffusion partial area | region of an original R image is carried out to R copy image in 2D Example 2 of 3D Example 1 which concerns on the said embodiment. The figure which shows a mode that copy addition of the blur spreading | diffusion partial area | region of an original B image is carried out to B copy image in 2D Example 2 of 3D Example 1 which concerns on the said embodiment. The diagram which shows the example which changes the size of a blur spreading | diffusion partial area | region according to 2D Example 2 of 3D Example 1 which concerns on the said embodiment according to phase difference amount. The flowchart which shows the colorization process performed by the color image generation part in 2D Example 2 of 3D Example 1 which concerns on the said embodiment. The figure which shows the outline | summary of the table of PSF of each color according to phase difference amount in 2D Example 3 of 3D Example 1 which concerns on the said embodiment. The figure which shows the outline | summary of the colorization process performed by the color image generation part in 2D Example 3 of 3D Example 1 which concerns on the said embodiment. The figure which shows the outline | summary of the colorization process with the blur amount control performed by the color image generation part in 2D Example 3 of 3D Example 1 which concerns on the said embodiment. The flowchart which shows the colorization process performed by the color image generation part in 2D Example 3 of 3D Example 1 which concerns on the said embodiment. The figure which shows a mode that the 3D Example 1 which concerns on the said embodiment produces | generates a left eye image and a right eye image from the two-dimensional image after a colorization process. The flowchart which shows the production | generation process of the stereoscopic image by the stereo image production | generation part in 3D Example 1 which concerns on the said embodiment. The figure which shows the mode of the shift at the time of producing | generating a left eye image from the image | photographing of the to-be-photographed object of the far side rather than a focusing position in the production | generation process of the stereoscopic vision image of 3D Example 2 which concerns on the said embodiment. The figure which shows the mode of the shift at the time of producing | generating a right-eye image from the image | photographing of the to-be-photographed object of the far side from the focus position in the production | generation process of the stereoscopic vision image of 3D Example 2 which concerns on the said embodiment. The figure which shows the mode after performing the filter process which approximates the blurring shape of each color in the left-eye image after a shift in 3D Example 2 which concerns on the said embodiment. The figure which shows a mode after performing the filter process which approximates the blurring shape of each color in the right-eye image after a shift in 3D Example 2 which concerns on the said embodiment. FIG. 3 is a diagram illustrating an example of a phase difference image displayed on a display unit of the imaging apparatus according to the first embodiment of the present invention. 3 is a flowchart showing display processing in the imaging apparatus according to the first embodiment. The figure which shows the example of the phase difference image displayed on the display part of the imaging device of Embodiment 2 of this invention. 7 is a flowchart showing display processing in the imaging apparatus according to the second embodiment. The block diagram which shows the detailed structure of the image process part and system controller in the imaging device of Embodiment 3 of this invention. FIG. 10 is a diagram illustrating an example of a phase difference image displayed on a display unit of the imaging apparatus according to the third embodiment. 10 is a flowchart illustrating display processing in the imaging apparatus according to the third embodiment. The block diagram which shows the detailed structure of the image process part and system controller in the imaging device of Embodiment 4 of this invention. FIG. 6 is a diagram illustrating an example of a phase difference image displayed on a display unit of the imaging apparatus according to the fourth embodiment. 10 is a flowchart illustrating display processing in the imaging apparatus according to the fourth embodiment. The block diagram which shows the detailed structure of the image process part and system controller in the imaging device of Embodiment 5 of this invention. FIG. 10 is a diagram illustrating an example of a phase difference image displayed on a display unit of the imaging apparatus according to the fifth embodiment. 10 is a flowchart illustrating display processing in the imaging apparatus according to the fifth embodiment. In the said Embodiment 5, the diagram which shows an example of the color predetermined according to the magnitude | size of phase difference amount.

  Embodiments of the present invention will be described below with reference to the drawings.

First, a technique according to each embodiment of the present invention will be described with reference to FIGS. Here, 3D Example 1 related to a three-dimensional stereoscopic image will be described with reference to FIGS. 1 to 36, and 3D Example 2 related to a three-dimensional stereoscopic image will be described with reference to FIGS. . The planar image will be described as 2D Example 1 to 2D Example 3 in 3D Example 1.
[3D Example 1]

  FIG. 1 to FIG. 36 show a 3D example 1 according to the embodiment of the present invention, and FIG. 1 is a block diagram showing the configuration of the imaging apparatus.

  The imaging apparatus according to the 3D embodiment 1 is configured as a digital still camera, for example. However, here, a digital still camera is taken as an example, but the imaging device may be any device that has a color imaging device and has an imaging function. Digital still cameras, video cameras, mobile phones with cameras, personal digital assistants with cameras (PDAs with cameras), personal computers with cameras, surveillance cameras, endoscopes, digital microscopes (digital microscopes), and the like.

  The imaging apparatus includes a lens unit 1 and a body unit 2 that is a main body portion to which the lens unit 1 is detachably attached via a lens mount. Here, a case where the lens unit 1 is detachable will be described as an example, but of course, it may not be detachable.

  The lens unit 1 includes an imaging optical system 9 including a lens 10 and a diaphragm 11, a band limiting filter 12, a lens control unit 14, and a lens side communication connector 15.

  The body unit 2 includes a shutter 21, an imaging element 22, an imaging circuit 23, an imaging drive unit 24, an image processing unit 25, an image memory 26, a display unit 27, an interface (IF) 28, and a system controller. 30, a sensor unit 31, an operation unit 32, a strobe control circuit 33, a strobe 34, and a body side communication connector 35. In addition, although the recording medium 29 is also described in the body unit 2 of FIG. 1, this recording medium 29 is a memory card (smart media, SD card, xD picture card, etc.) which is detachable with respect to the imaging device. Since it is configured, the configuration may not be unique to the imaging apparatus.

  First, the imaging optical system 9 in the lens unit 1 is for forming a subject image on the imaging element 22. The lens 10 of the imaging optical system 9 includes a focus lens for performing focus adjustment. Although the lens 10 is generally composed of a plurality of lenses in general, only one lens is shown in FIG. 1 for simplicity.

  The diaphragm 11 of the imaging optical system 9 is for adjusting the brightness of the subject image formed on the imaging element 22 by regulating the passage range of the subject light flux passing through the lens 10.

  The band limiting filter 12 is disposed on the optical path of the imaging light beam that passes through the imaging optical system 9 and reaches the imaging device 22 (preferably at the position of the diaphragm 11 of the imaging optical system 9 or in the vicinity thereof). A first band limitation that blocks light in the first band in the photographic light flux that attempts to pass through the first area that is a part of the pupil area, and allows light in the second band and the third band to pass through. The light of the second band in the imaging light flux that attempts to pass through the second area that is another part of the pupil area of the imaging optical system 9 is blocked and the light of the first band and the third band is allowed to pass. And a second band limitation to be performed.

  FIG. 3 is a diagram for explaining a configuration example of the band limiting filter 12. In the band limiting filter 12 of the configuration example shown in FIG. 3, the pupil region of the imaging optical system 9 is divided into a first region and a second region. In other words, the band limiting filter 12 has a left half of the G (green) component and R (red) when the image pickup device is viewed from the image pickup device 22 in a standard posture (so-called posture in which the camera is held in a normal horizontal position). RG filter 12r that passes the component and blocks the B (blue) component (that is, the B (blue) component is one of the first band and the second band), the right half is the G component and The GB filter 12b is configured to pass the B component and block the R component (that is, the R component is the other of the first band and the second band). Therefore, the band limiting filter 12 passes all the G components contained in the light passing through the aperture 11 of the imaging optical system 9 (that is, the G component is the third band), and the R component is half of the aperture. And pass the B component only through the remaining half of the aperture. If the RGB spectral transmission characteristics of the band limiting filter 12 are different from the RGB spectral transmission characteristics of an element filter (see FIG. 2) configured on-chip in the image sensor 22, the RG filter 12r and the GB filter 12b As will be described later, the accuracy of the position information acquired from the images obtained based on the difference in the spatial position is reduced, or a light amount loss due to a mismatch in spectral characteristics occurs. Therefore, it is desirable that the spectral transmission characteristic of the band limiting filter 12 is the same as or as close as possible to the spectral transmission characteristic of the element filter of the image sensor 22. Other configuration examples of the band limiting filter 12 will be described later with reference to FIGS.

  The lens control unit 14 controls the lens unit 1. That is, the lens control unit 14 drives and focuses the focus lens in the lens 10 based on a command received from the system controller 30 via the lens-side communication connector 15 and the body-side communication connector 35, or the aperture 11 is moved. It is driven to change the aperture diameter.

  The lens-side communication connector 15 enables communication between the lens control unit 14 and the system controller 30 by connecting the lens unit 1 and the body unit 2 with a lens mount and connecting to the body-side communication connector 35. Connector.

  Next, the shutter 21 in the body unit 2 is an optical shutter for adjusting the exposure time of the image sensor 22 by regulating the passage time of the subject light beam reaching the image sensor 22 from the lens 10. Although an optical shutter is used here, an element shutter (electronic shutter) by the image sensor 22 may be used instead of or in addition to the optical shutter.

  The imaging device 22 receives and photoelectrically converts the subject image formed by the imaging optical system 9 for each of a plurality of wavelength bands (for example, but not limited to RGB), and performs photoelectric conversion. A color image sensor that outputs a signal, for example, a CCD or CMOS. Here, the configuration of the color image sensor may be a single-plate image sensor provided with an on-chip element color filter, or a three-plate system using a dichroic prism that performs color separation into RGB color lights. However, it may be an image sensor of a method capable of acquiring RGB imaging information according to the position in the depth direction of the semiconductor at the same pixel position, or any imaging element that can acquire imaging information of a plurality of wavelength bands. It does n’t matter.

  For example, with reference to FIG. 2, a configuration example of a single-plate color image sensor often used for a general digital still camera will be described. FIG. 2 is a diagram for explaining the pixel arrangement of the image sensor 22. In the present 3D embodiment 1, the plurality of wavelength band lights transmitted by the element color filter mounted on-chip are R, G, and B. As shown in FIG. A plate color imaging device is configured. Therefore, when the image pickup device 22 has the configuration shown in FIG. 2, only one color component is obtained per pixel, and therefore the image processor 25 performs a demosaicing process. A color image in which three colors of RGB are aligned per pixel is generated.

  The image pickup circuit 23 performs A / D conversion when the image signal output from the image pickup device 22 is amplified (gain adjustment), or when the image pickup device 22 is an analog image pickup device and outputs an analog image signal. Or a digital image signal (hereinafter also referred to as “image information”) (when the image pickup device 22 is a digital image pickup device, it is already digital when it is input to the image pickup circuit 23). Therefore, A / D conversion is not performed). The imaging circuit 23 outputs an image signal to the image processing unit 25 in a format corresponding to the imaging mode switched by the imaging driving unit 24 as will be described later.

  The imaging drive unit 24 supplies a timing signal and power to the imaging element 22 and the imaging circuit 23 based on a command from the system controller 30 to cause the imaging element to perform exposure, reading, element shuttering, and the like, and also to the imaging element 22. Control is performed to execute gain adjustment and A / D conversion by the imaging circuit 23 in synchronism with the above operations. The imaging drive unit 24 also performs control to switch the imaging mode of the imaging element 22.

  The image processing unit 25 is a digital image such as WB (white balance) adjustment, black level correction, γ correction, defective pixel correction, demosaicing, image information color information conversion processing, image information pixel number conversion processing, and the like. The processing is performed. The image processing unit 25 further includes a color correction unit 36, a color image generation unit 37, and a stereo image generation unit 40.

  As described above, since the band limiting filter 12 limits the pass region in the pupil region of the imaging optical system 9 by the band (color) of the light that passes through, the brightness of the light that passes through the band varies. become. The inter-color correction unit 36 is for correcting such a difference in brightness between bands (between colors). The brightness difference between bands (between colors) can be easily corrected according to the area of the pass region for each band, but the amount of light in the periphery is larger than the center of the image. Of course, more detailed correction according to the optical characteristics of the imaging optical system 9 may be performed in consideration of the tendency to decrease. At this time, the correction value is not limited to be calculated in the imaging apparatus, and the correction value may be held as table data or the like. As a specific example, when the band limiting filter 12 is configured as shown in FIG. 3, R and B have half the amount of light passing through G, so that the R color signal and the B color signal are 2 It is conceivable to perform the doubling process as a simple inter-color correction (similarly, in the case of the configuration shown in FIGS. 12 to 15 described later, correction is performed according to the area of the pass region for each band. become). By performing such processing, it becomes possible to handle the image as an image similar to that of the image pickup apparatus in which the band limiting filter 12 of FIG. 3 is not inserted in terms of light quantity balance, and various functions (for example, image generation processing) are performed. It can be used.

  The color image generation unit 37 performs colorization processing that is digital processing for forming color image information. When the band limiting filter 12 as shown in FIG. 3 is used, a spatial positional deviation may occur between the R image and the B image. Therefore, the spatial positional deviation is corrected. It is an example of the colorization process which the color image generation part 37 performs.

  First, the spatial positional shift of each color image will be described with reference to FIGS. FIG. 4 is a plan view showing a state of subject light flux condensing when imaging a subject at the in-focus position, and FIG. 5 is a subject light flux concentrating when imaging a subject closer to the in-focus position. FIG. 6 is a plan view showing the state of light, FIG. 6 is a diagram showing the shape of a blur formed by light from one point on the subject that is closer to the focus position, and FIG. 7 is a closer distance than the focus position. FIG. 8 is a diagram showing the shape of blur formed by light from one point on a subject on the side for each color component, and FIG. 8 is a diagram of subject light flux collection when imaging a subject farther than the in-focus position. FIG. 9 is a plan view showing the state, FIG. 9 is a diagram showing the shape of a blur formed by light from one point on the subject that is on the far side from the in-focus position, and FIG. 10 is on the far side from the in-focus position. FIG. 11 is a diagram showing the shape of a blur formed by light from one point on a subject for each color component. Location and than a diagram showing a state of image obtained by imaging the object at a short distance and long distance. In the description of the blur shape, the case where the aperture of the diaphragm 11 has a circular shape will be described as an example.

  When the subject OBJc is at the in-focus position, the light emitted from one point on the subject OBJc has a G component that passes through the entire band limiting filter 12 as shown in FIG. Both the R component that passes only through the filter 12r and the B component that passes only through the other half of the GB filter 12b of the band limiting filter 12 are collected at one point on the image sensor 22 to form a point image IMGrgb. Although there is a difference in the amount of light according to the area of the passing region as described above, no positional deviation occurs between the colors. Therefore, when the subject OBJc at the in-focus position is imaged, a subject image IMGrgb having no color blur is formed as shown in FIG.

  On the other hand, when the subject OBJn is closer to the in-focus position, for example, the light emitted from one point on the subject OBJn is used for the G component as shown in FIGS. A subject image IMGg that forms a circular blur is formed, a subject image IMGr that forms a semicircular blur of the left half is formed for the R component, and a subject image IMGb that forms a semicircular blur of the right half is formed for the B component. Therefore, when the subject OBJn that is closer to the in-focus position is imaged, as shown in FIG. 11, the blur component image in which the R component subject image IMGr is shifted to the left and the B component subject image IMGb is shifted to the right. The left and right positions of the R component and B component in this blurred image are the R component transmission region (RG filter 12r) and B component transmission region (GB filter 12b) of the band limiting filter 12 when viewed from the image sensor 22. (In FIG. 11, the deviation is emphasized and illustration of the blurred shape actually generated is omitted (the same applies to the case of the far side below)). As the subject OBJn moves away from the in-focus position, the blur increases, and the distance between the center of gravity of the R component subject image IMGr and the center of gravity of the B component subject image IMGb increases. Become. Note that the G component subject image IMGg is a blurred image extending over the R component subject image IMGr and the B component subject image IMGb, as shown in FIGS. Since the G component subject image IMGg has the same information as the blurred image when the band limiting filter 12 is not used, it can be used as a standard image in the colorization processing performed by the color image generation unit 37 described above. is there.

  On the other hand, when the subject OBJf is on the far side of the in-focus position, for example, the light emitted from one point on the subject OBJf causes a circular blur for the G component as shown in FIGS. A subject image IMGg is formed, a subject image IMGr that forms a semicircular blur of the right half is formed for the R component, and a subject image IMGb that forms a semicircular blur of the left half is formed for the B component. Therefore, when the subject OBJf that is farther than the in-focus position is imaged, as shown in FIG. 11, the blur component image in which the R component subject image IMGr is shifted to the right side and the B component subject image IMGb is shifted to the left side. The left and right positions of the R component and B component in this blurred image are the R component transmission region (RG filter 12r) and B component transmission region (GB filter 12b) of the band limiting filter 12 when viewed from the image sensor 22. It is the opposite of the left and right position As the subject OBJf moves away from the in-focus position, the blur increases, and the distance between the center of gravity of the R component subject image IMGr and the center of gravity of the B component subject image IMGb increases. Become. It is to be noted that the G component subject image becomes a blurred image straddling the R component subject image IMGr and the B component subject image IMGb (see FIGS. 9 and 10) as well on the far side. . On the far side, the G component subject image IMGg has the same information as the blurred image when the band limiting filter 12 is not used. Therefore, the standard image in the colorization process performed by the color image generation unit 37 described above. It can be used as

  Returning to the description of FIG. 1, the color image generation unit 37 performs, as a colorization process, a G component image having the largest number of pixels and the largest contribution to the luminance signal in the Bayer array as illustrated in FIG. 2 as a standard image. As described above, the process of correcting each color shift between the R component image and the B component image is performed as will be described in detail later. In this way, by generating an image based on the G color that can acquire information on the entire aperture of the diaphragm 11, it is possible to obtain a high-quality image that can withstand viewing.

  The stereo image generation unit 40 is based on a single image data acquired from the image sensor 22, and based on a phase difference amount calculated by a distance calculation unit 39 described later, the second image and the third image are blurred. The one-eye color image (one of the left-eye image and the right-eye image) is generated by moving the center-of-gravity position in the direction of the center-of-gravity position of the blur of the first image, and the first image and the third image By generating another one-eye color image (one of the left-eye image and the right-eye image) in which the position of the center of gravity of the image is moved in the direction of the position of the center of gravity of the second image. An image is generated. The processing of the stereo image generation unit 40 will be described in detail later.

  The image memory 26 is a memory capable of high-speed writing and reading, and is constituted by, for example, an SDRAM (Synchronous Dynamic Random Access Memory). The image memory 26 is used as a work area for image processing and is also a work area of the system controller 30. Also used as For example, the image memory 26 not only stores the final image processed by the image processing unit 25 but also appropriately stores each intermediate image in a plurality of processing steps by the image processing unit 25.

  The display unit 27 includes an LCD or the like, and an image processed for display by the image processing unit 25 (an image read from the recording medium 29 and processed for display by the image processing unit 25 is also included). Display). Specifically, the display unit 27 performs live view image display, confirmation display when recording a still image, reproduction display of a still image or a moving image read from the recording medium 29, and the like. In the 3D embodiment 1, the display unit 27 is configured to be able to display a stereoscopic image.

  The interface (IF) 28 is detachably connected to the recording medium 29, and transmits information to be recorded on the recording medium 29 and information read from the recording medium 29.

  The recording medium 29 records an image processed for recording by the image processing unit 25 and various data related to the image, and is configured as a memory card or the like as described above.

  The sensor unit 31 is, for example, a camera shake sensor configured by an acceleration sensor or the like for detecting blur of the imaging device, a temperature sensor for measuring the temperature of the imaging device 22, and a brightness for measuring the brightness around the imaging device. Includes brightness sensor, etc. The detection result by the sensor unit 31 is input to the system controller 30. Here, the detection result by the camera shake sensor is used to drive the image pickup device 22 and the lens 10 to perform camera shake correction or to perform camera shake correction by image processing. The detection result by the temperature sensor is used to control the drive clock by the imaging drive unit 24 and to estimate the amount of noise in the image obtained from the image sensor 22. Furthermore, the detection result by the brightness sensor is used, for example, to appropriately control the luminance of the display unit 27 according to the ambient brightness.

  The operation unit 32 changes a power switch for turning on / off the power of the image pickup device, a release button including a two-stage press button for inputting an image pickup operation such as a still image or a moving image, an image pickup mode, and the like. Mode buttons, cross keys used to change selection items, numerical values, and the like. A signal generated by the operation of the operation unit 32 is input to the system controller 30.

  The strobe control circuit 33 controls the light emission amount and the light emission timing of the strobe 34 based on a command from the system controller 30.

  The strobe 34 is a light source that emits illumination light to the subject under the control of the strobe control circuit 33.

  As described above, the body side communication connector 35 is connected between the lens control unit 14 and the system controller 30 when the lens unit 1 and the body unit 2 are coupled by the lens mount and connected to the lens side communication connector 15. It is a connector that enables communication.

  The system controller 30 controls the body unit 2 and also controls the lens unit 1 via the lens control unit 14, and is a control unit that integrally controls the imaging apparatus. The system controller 30 reads a basic control program of the image pickup apparatus from a non-illustrated non-volatile memory such as a flash memory, and controls the entire image pickup apparatus in accordance with an input from the operation unit 32.

  For example, the system controller 30 controls the diaphragm adjustment of the diaphragm 11 via the lens control unit 14, controls and drives the shutter 21, and shake correction (not shown) based on the detection result by the acceleration sensor of the sensor unit 31. The mechanism is controlled to perform camera shake correction or the like. Further, the system controller 30 sets the mode setting of the imaging device (a still image mode for capturing a still image, a moving image mode for capturing a moving image, a stereoscopic image in response to an input from the mode button of the operation unit 32. 3D mode or the like for imaging).

  Further, the system controller 30 includes a contrast AF control unit 38 and a distance calculation unit 39, and causes the contrast AF control unit 38 to perform AF control or based on the distance information calculated by the distance calculation unit 39. 1 is performed to perform AF.

  The contrast AF control unit 38 may output an image signal output from the image processing unit 25 (this image signal may be a G image having a high ratio including a luminance component, and color misregistration is corrected by a colorization process described later. A contrast value (also referred to as an AF evaluation value) is generated from the luminance signal image related to the image, and the focus lens in the lens 10 is controlled via the lens control unit 14. That is, the contrast AF control unit 38 extracts a high-frequency component by applying a filter, for example, a high-pass filter, to the image signal to obtain a contrast value. Then, the contrast AF control unit 38 acquires the contrast value by changing the focus lens position, moves the focus lens in the direction in which the contrast value increases, and further acquires the contrast value. By repeatedly performing such processing, control is performed so that the focus lens is driven to the focus lens position (focus position) at which the maximum contrast value is obtained.

  Next, the distance calculation unit 39 calculates the phase difference amount between the first image in the first band and the second image in the second band obtained from the image sensor 22 and calculates the calculated phase difference amount. Based on the above, the distance to the subject is calculated. Specifically, the distance calculation unit 39 calculates distance information according to the lens formula from the amount of deviation between the R component and the B component as shown in FIG. 7, FIG. 10, FIG.

  That is, the distance calculation unit 39 extracts the R component and the B component from the components of each of the RGB colors obtained as the captured image. Then, by calculating the correlation between the R component and the B component, the direction of the deviation and the magnitude of the deviation occurring in the R component image and the B component image are calculated (however, the present invention is not limited to this). In addition, it is not impossible to calculate the direction of displacement and the size of the displacement occurring between the R image and the G image or between the B image and the G image).

  Here, for example, when the band limiting filter 12 as shown in FIG. 3 is used, as described with reference to FIGS. 5 to 11, in the image, depending on whether the subject is closer or farther than the in-focus position. The direction of deviation between the R component and the B component is reversed. Therefore, the direction of deviation is information for determining whether the subject of interest is closer or farther than the in-focus position.

  Further, the larger the distance from the in-focus position to the short distance side or the far distance side, the larger the difference between the R component and the B component. Therefore, the magnitude of the deviation is information for determining how far the subject of interest is away from the in-focus position.

  In this way, the distance calculation unit 39 calculates how far the subject of interest is away to the side closer to or farther from the in-focus position based on the calculated direction of deviation and magnitude of deviation. It is like that.

  The distance calculation unit 39 is based on an input from the operation unit 32 or based on a basic control program of the imaging apparatus, and a distance calculation area determined by the system controller 30 (for example, the entire area of the captured image or distance information in the captured image). The distance calculation as described above is performed for a part of the region where it is desired to acquire the image. For example, a technique described in Japanese Patent Application Laid-Open No. 2001-16611 can be used as the technique for calculating the distance information by acquiring the deviation amount.

  The distance information acquired by the distance calculation unit 39 can be used, for example, for autofocus (AF).

  That is, the distance calculation unit 39 acquires distance information based on the difference between the R component and the B component, and the system controller 30 drives the focus lens of the lens 10 via the lens control unit 14 based on the acquired distance information, that is, AF. Phase difference AF can be performed. Thereby, high-speed AF is possible based on one captured image.

  However, since the distance calculation unit 39 and the contrast AF control unit 38 are provided in the system controller 30 as described above, AF control may be performed based on the calculation result by the distance calculation unit 39. It may be performed by the contrast AF control unit 38.

  Here, contrast AF by the contrast AF control unit 38 has high focusing accuracy, but requires a plurality of captured images, and thus there is a problem that the focusing speed cannot be said to be high. On the other hand, since the calculation of the subject distance by the distance calculation unit 39 can be performed based on one captured image, the focusing speed is fast, but the focusing accuracy may be inferior to the contrast AF.

  Therefore, the AF assist control unit 38a provided in the contrast AF control unit 38 may perform AF by combining the contrast AF control unit 38 and the distance calculation unit 39. That is, the distance calculation unit 39 performs a distance calculation based on the deviation between the R component and the B component of the image acquired through the band limiting filter 12 to determine whether the subject is on the far side from the current focus position. Or it is acquired whether it exists in the short distance side. Alternatively, the distance that the subject is away from the current focus position is acquired. Next, the AF assist control unit 38a drives the focus lens toward the acquired long distance side or the short distance side (by the acquired distance), and controls the contrast AF control unit 38 to perform contrast AF. By performing such processing, it is possible to obtain high focusing accuracy at a fast focusing speed.

  In addition, the R image and the B image obtained from the image sensor 22 can be used as, for example, a stereo stereoscopic image (3D image).

  The 3D image may be obtained by observing an image from the left pupil with the left eye and observing an image from the right pupil with the right eye. An anaglyph method is conventionally known as such a 3D image observation method (see also the above-mentioned Japanese Patent Laid-Open No. 4-251239). This anaglyph method generally uses red and blue glasses for anaglyphs that generate a red left-eye image and a blue right-eye image, display both, and arrange a red transmission filter on the left eye side and a blue transmission filter on the right eye side. By observing the lever image, a monochrome stereoscopic image can be observed.

  Thus, in the present 3D embodiment 1, an R image and a B image obtained from the image sensor 22 in the standard posture (an image in which color processing for correcting the positional deviation between the R component and the B component in the color image generation unit 37 is not performed). ) Can be viewed as it is by observing with the anaglyph type red-blue glasses. That is, as described with reference to FIG. 3, when the imaging apparatus is in the standard posture, the RG filter 12r of the band limiting filter 12 is arranged on the left side when the subject is viewed from the imaging element 22, and the GB filter 12b is on the right side. It is comprised so that it may be arrange | positioned. Thus, when wearing red and blue glasses, the R component light transmitted through the left RG filter 12r is observed only by the left eye, and the B component light transmitted through the right GB filter 12b is observed only by the right eye, allowing stereoscopic viewing. It becomes.

  In the 3D embodiment 1, as described above, the stereo image generation unit 40 can generate a color stereoscopic image that is not an anaglyph method.

  Next, modified examples of the band limiting filter 12 will be described with reference to FIGS.

  First, FIG. 12 is a diagram showing a first modification of the band limiting filter 12.

  In the band limiting filter 12 shown in FIG. 12, the imaging light beam in which the pupil region of the imaging optical system 9 receives both the first band limitation and the second band limitation between the first region and the second region. It is configured such that a passing area is sandwiched. That is, the band limiting filter 12 shown in FIG. 3 is the RG filter 12r on the left half and the GB filter 12b on the right half when viewed from the image pickup device 22 with the image pickup apparatus in the standard posture. On the other hand, in the band limiting filter 12 shown in FIG. 12, a G filter 12g that allows only the G component to pass is disposed between the left RG filter 12r and the right GB filter 12b.

  Even in such a configuration, all of the G component included in the light passing through the aperture of the diaphragm 11 of the imaging optical system 9 passes in the same way as the band limiting filter 12 in FIG. However, since the RG filter 12r and the GB filter 12b are arranged apart from each other, the positional deviation between the R component and the B component of the image of the subject at the out-of-focus position on the image sensor 22 is further clarified. Therefore, there is an advantage that the distance calculation accuracy by the distance calculation unit 39 is improved. Therefore, when it is desired to obtain distance information with high accuracy, it is preferable to use the band limiting filter 12 configured as shown in FIG.

  FIG. 13 is a diagram showing a second modification of the band limiting filter 12.

  The band limiting filter 12 shown in FIG. 13 has a configuration in which the pupil region of the imaging optical system 9 is sandwiched between the first region and the second region through which the imaging light flux that is not subjected to the band limitation passes. ing. In other words, the band limiting filter 12 is a filter in which a W filter 12w that passes components of all RGB colors (that is, white light W) is disposed between the left RG filter 12r and the right GB filter 12b. .

  Even in such a configuration, all of the G component included in the light passing through the aperture of the diaphragm 11 of the imaging optical system 9 passes in the same way as the band limiting filter 12 in FIG. In contrast, the R component passes through the RG filter 12r and the W filter 12w, but is blocked by the GB filter 12b. Similarly, the B component passes through the GB filter 12b and the W filter 12w, but is blocked by the RG filter 12r.

  According to such a configuration, not only the G component but also the R component and the B component are transmitted through the portion of the W filter 12w, so that the light amounts of the R component and the B component that pass through the band limiting filter 12 increase. There is an advantage that a bright captured image can be acquired (however, the accuracy of the distance information calculated by the distance calculation unit 39 may be reduced). Accordingly, when the distance information is used when high accuracy is not necessary, a necessary effect can be obtained and, for example, a bright high-quality 3D image can be acquired.

  Further, FIG. 14 is a diagram showing a third modification of the band limiting filter 12.

  In the band limiting filter 12 shown in FIG. 14, the first region and the second region are arranged with different vertical and horizontal positions when the imaging apparatus is in the standard posture. Yes. That is, the band limiting filter 12 divides a circular filter into four crosses, for example, and arranges the RG filter 12r in the lower left (third quadrant in the graph) and the GB filter 12b in the upper right (first quadrant in the graph). In addition, the first G filter 12g1 is arranged in the upper left (second quadrant in the graph), and the second G filter 12g2 is arranged in the lower right (fourth quadrant in the graph).

  Note that the direction of the phase difference acquired by the band limiting filter shown in FIG. 14 is a phase difference of 45 degrees obliquely. Using this phase difference information, the subsequent color image generation unit performs colorization processing, and then the stereo image generation unit performs stereo image generation. At that time, when color processing is performed in the color image generation unit, a normal band limiting filter (filter example other than the band limiting filter shown in FIG. 14) uses a horizontal phase difference. Colorization processing is performed in a phase difference direction of 45 degrees obliquely. When the stereo image generation process is performed in the stereo image generation unit, a stereo image generation process is performed using a normal horizontal phase difference (the horizontal phase difference is a 45 ° diagonal phase difference). If only the horizontal component is extracted, it can be easily obtained). By doing in this way, even when the phase difference acquired by the band limiting filter is in an oblique direction, it is possible to generate a stereo image having a phase difference in a natural horizontal direction when viewed by a person.

  In the case of the configuration of the band limiting filter 12 as shown in FIGS. 3, 12, and 13 described above, it is easy to obtain distance information of a subject having a vertical edge, but a subject having a horizontal edge. It becomes difficult to acquire distance information. On the other hand, if the configuration of the band limiting filter 12 as shown in FIG. 14 is adopted, the distance to both the subject having the vertical edge and the subject having the horizontal edge is determined. There is an advantage that information can be easily acquired. Thus, by making the horizontal position and the vertical position of the RG filter 12r and the GB filter 12b different, it is possible to improve the subject edge direction dependency (particularly, the horizontal / vertical direction dependency) in distance information acquisition. It becomes.

  Even when the band limiting filter 12 is configured as shown in FIG. 12 to FIG. 14 and the like, the amount of deviation between the R image and the B image and the shape of the blurred PSF as described later only change. Therefore, it is possible to generate an image with corrected color misalignment by similarly applying a colorization process for correcting the positional misalignment between the R component and the B component in the color image generating unit 37 as described below. .

  Next, FIG. 15 is a diagram illustrating a fourth modification of the band limiting filter 12.

  The band limiting filter 12 shown in FIGS. 3 and 12 to 14 is a normal color filter, but the band limiting filter 12 shown in FIG. 15 is configured by a color selective transmission element 18.

  Here, the color selective transmission element 18 includes a member capable of rotating the polarization transmission axis corresponding to the color (wavelength) and a member capable of selectively controlling whether the polarization transmission axis rotates or not, such as an LCD. This element is realized by combining a plurality of elements, and can change the color distribution. A specific example of the color selective transmission element 18 is a color switch manufactured by Colorlink as disclosed in SID2000 as “SID '00 Digest, Vol. 31, p. 92” in April 2000 AD. .

  The color selective transmission element 18 has the left half as the first color selection transmission element 18L and the right half as the second color selection transmission element 18R when viewed from the imaging element 22 with the imaging apparatus in the standard posture (lateral position). It is configured. Here, the first color selective transmission element 18L can selectively take an RG state that passes the G component and the R component and blocks the B component, and a W state that passes all the RGB components (W). It has become. Further, the second color selective transmission element 18R can selectively take a GB state in which the G component and the B component are allowed to pass and an R component is blocked, and a W state in which all the RGB components (W) are allowed to pass. ing. The first color selection / transmission element 18L and the second color selection / transmission element 18R are independently driven by a color selection drive unit (not shown), and the first color selection / transmission element 18L is set in the RG state. In addition, by causing the second color selective transmission element 18R to assume the GB state, the same function as the band limiting filter 12 shown in FIG. 3 can be achieved.

  Furthermore, if the configuration is changed to another configuration, the color selective transmission element 18 can perform the same function as the band limiting filter 12 shown in FIGS. The band limiting filter 12 can also be configured.

Next, a colorization process for correcting a shift (color shift) between the color images performed in the color image generation unit 37 will be described.
[2D Example 1]

  First, FIGS. 16-24 is a figure which shows 2D Example 1 of a colorization process.

  When the subject on the far side from the in-focus position is photographed using the band limiting filter 12 shown in FIG. 3, a blur as shown in FIG. 7 is formed, and the subject is in the near side from the in-focus position. As shown in FIG. 10, a blur is formed.

  The colorization process is a process for correcting the blur having the shape shown in FIG. 7 or 10 to the blur having the shape shown in FIG. FIG. 16 is a diagram showing an outline of the blurred shape after the colorization process.

  In the following, in order to simplify the description, the case where the subject is on the far side from the in-focus position (in the case shown in FIG. 10) will be described as an example, but the subject is closer to the in-focus side than the in-focus position. In this case, as can be seen by comparing FIG. 7 with FIG. 10, the shape and position of the blur of the R image and the blur of the B image are just opposite to each other. It is possible to apply similarly.

  FIG. 17 is a diagram showing the shape of the R filter applied to the R image of the subject that is farther than the in-focus position in the colorization processing of the 2D embodiment 1, and FIG. 18 is the color of the 2D embodiment 1. It is a figure which shows the shape of the filter for B applied with respect to the to-be-photographed object's B image in the far side rather than a focus position in a conversion process.

  In this colorization process, the correction of color misregistration and the correction of the blurred shape are performed by a filtering process (a process of convolving a filter kernel with an image). That is, by performing R filtering processing on the R image, the blur shape and the gravity center position of the R image are approximated to the blur shape and the gravity center position of the G image that is the standard image, and the B image By performing the filtering process for B, the blur shape and the gravity center position of the B image are approximated to the blur shape and the gravity center position of the G image that is the standard image. Here, the center of gravity position is approximated to correct color misregistration, and the blur shape is approximated to correct the blur shape different for each color, and the blur in the color image is naturally blurred. This is because.

  In particular, here, the blur shape of the R image and the B image is matched with the blur shape of the G image, which is the standard image. The reason is that the blur of the G image has a circular shape. Since it is blur and the blur size of the G image is larger than the blur size of the R image and the B image, it is easier to process the blur shape of the R image and the B image with the blur shape of the G image. There are some things.

  The R filter kernel shown in FIG. 17 is used for performing a convolution operation on an R image, and a Gaussian filter is disposed as an example of a blur filter. In this R filter kernel, the peak of the filter coefficient of the Gaussian filter (corresponding approximately to the position of the center of gravity of the filter coefficient) is the position of the center of the kernel (the horizontal line Ch that divides the top and bottom of the kernel into two equal parts) and the left and right of the kernel. A position where the phase difference between the G image and the R image is shifted (a horizontal line Ch and the R image) from the vertical line that equally divides and the vertical line Cg passing through the center of gravity of the blurred shape of the G image. This is a filter shape at a position where the vertical line Cr passing through the center of gravity of the blurred shape intersects.

  The B filter kernel shown in FIG. 18 is for performing a convolution operation on the B image, and, like the R filter kernel, a Gaussian filter is arranged as an example of a blur filter. In the B filter kernel, the peak of the filter coefficient of the Gaussian filter (corresponding approximately to the position of the center of gravity of the filter coefficient) starts from the position of the center of the kernel (the position where the horizontal line Ch and the vertical line Cg intersect). The filter shape is at a position where the phase difference between the images B and B is shifted (a position where the horizontal line Ch intersects the vertical line Cb passing through the center of gravity of the blur shape of the B image).

  By causing the R filter kernel and the B filter kernel having such a filter shape to act on the R image and the B image, respectively, it is possible to simultaneously perform color misalignment correction and blur shape correction. .

  Next, a modification of the colorization process will be described with reference to FIGS. 19 and 20. FIG. 19 is a diagram showing a shift state of the R image and the B image of the subject on the far side from the in-focus position in the colorization process of the modification of the 2D embodiment 1, and FIG. It is a figure which shows the shape of the filter applied with respect to R image and B image in the colorization process of the modified example.

  In this example, the color processing is performed by correcting the color misregistration by parallel movement (shift) and correcting the blurred shape by the filtering processing.

  That is, first, by performing an R shift process corresponding to the phase difference on the R image shown in FIG. 10, the center of gravity position of the R image becomes the center of gravity position of the blur of the G image that is the standard image as shown in FIG. 10 and performing the B shift processing corresponding to the phase difference on the B image shown in FIG. 10, the center of gravity of the B image is changed to the center of gravity of the G image as the standard image as shown in FIG. 19. To approximate.

  Thereafter, the same filtering process as shown in FIG. 20 is performed on the R image and the B image to approximate the blur shape of the R image and the B image to the blur shape of the G image that is the standard image. (See FIG. 16).

  Here, the filter kernel shown in FIG. 20 is for performing a convolution operation on the R image and the B image, and a Gaussian filter is arranged as an example of the blur filter. This filter kernel has a filter shape in which the peak of the filter coefficient of the Gaussian filter (corresponding to the position of the center of gravity of the filter coefficient) is at the position of the kernel center (position where the horizontal line Ch and the vertical line Cg intersect). .

  However, since the band limiting filter 12 in which the first area and the second area are symmetric is assumed here as shown in FIG. 3, the same filtering process is applied to the R image and the B image. However, when the first region and the second region are asymmetric, it is needless to say that different filtering processes may be performed on the R image and the B image.

  In the above description, a Gaussian filter (circular Gaussian filter) is taken as an example of the blur filter, but the present invention is not limited to this. For example, in the case of the filter shape shown in FIG. 3, as shown in FIGS. 7 and 10, blurring generated in the R image and the B image has a half-moon shape in the vertical direction (that is, a shape shorter in the horizontal direction than in the circular shape). ) Therefore, if an elliptical Gaussian filter as shown in FIG. 21, FIG. 22 or the like having the horizontal direction (more generally, the direction in which the phase difference is generated) as the major axis direction is used, the blur is more accurately formed into a circular shape. It is possible to perform a correction process to approach FIG. 21 is a diagram showing the shape of an elliptical Gaussian filter having a large horizontal standard deviation in 2D Embodiment 1, and FIG. 22 is a diagram of the elliptical Gaussian filter having a small vertical standard deviation in 2D Embodiment 1. It is a figure which shows a shape. 21 and 22 show examples in which the peak of the filter coefficient (corresponding to the position of the center of gravity of the filter coefficient) is located at the center of the filter kernel corresponding to FIG. 20, but it corresponds to FIG. 17 and FIG. In this case, it goes without saying that the peak of the filter coefficient (which substantially corresponds to the position of the center of gravity of the filter coefficient) is shifted from the center of the filter kernel.

  Further, the blur filter is not limited to the circular Gaussian filter and the elliptical Gaussian filter, and the blur filter that can approximate the blur shape of the R image or the B image to the blur shape of the G image. If so, it can be widely applied.

  Next, FIG. 23 is a flowchart illustrating the colorization processing performed by the color image generation unit 37 in the 2D embodiment 1.

(Step S1)
When this process is started, initialization is performed. In this initial setting, first, an RGB image to be processed (that is, an R image, a G image, and a B image) is read. Next, an R copy image that is a copy of the R image and a B copy image that is a copy of the B image are created.

(Step S2)
Subsequently, a target pixel for performing phase difference detection is set. Here, the target pixel is set to one of the R image and the B image, here, for example, the R image.

(Step S3)
A phase difference with respect to the target pixel set in step S2 is detected. This phase difference is detected by setting a partial area including the target pixel at the center position as an R image as a reference image, and setting a partial area of the same size as a B image as a reference image (see FIG. 26, etc.) The distance calculation unit 39 performs a correlation calculation between the reference image and the reference image while shifting the position of the image in the direction in which the phase difference is generated, and the reference image determined to have the highest correlation value is compared with the reference image. The amount of misalignment between them (note that the sign of the misregistration amount gives information on the direction of misalignment) is the phase difference amount.

  The partial area can be set to an arbitrary size. However, in order to stably detect the phase difference, it is preferable to use partial areas of 30 [pixels] or more in the vertical and horizontal directions. As an example, 51 × 51 An area of [Pixel] is given.

  The correlation calculation in the distance calculation unit 39 is specifically performed by a process such as a ZNCC calculation or an SAD calculation on an image that has been previously filtered.

ここに、ＩはＲ画像の部分領域、ＴはＢ画像の部分領域（Ｉと同一サイズの部分領域）、Ｉ（バー）はＩの平均値、Ｔ（バー）はＴの平均値、Ｍは部分領域の横幅［ピクセル］、Ｎは部分領域の縦幅［ピクセル］である。この数式１に基づきＺＮＣＣ演算を行い、その結果の絶対値｜ＲZNCC｜を相関演算の結果として得られた相関値とする。 First, the correlation calculation by ZNCC is performed based on Equation 1 below.
[Equation 1]

Here, I is a partial area of the R image, T is a partial area of the B image (partial area of the same size as I), I (bar) is an average value of I, T (bar) is an average value of T, and M is The horizontal width [pixel] of the partial area, and N is the vertical width [pixel] of the partial area. A ZNCC operation is performed based on Equation 1, and the absolute value | RZNCC | of the result is set as a correlation value obtained as a result of the correlation operation.

ここに、Ｉ’はフィルタリング処理が施された後のＲ画像の部分領域、Ｔ’はフィルタリング処理が施された後のＢ画像の部分領域（Ｉ’と同一サイズの部分領域）、Ｍは部分領域の横幅［ピクセル］、Ｎは部分領域の縦幅［ピクセル］である。この場合には、ＲSADが相関演算の結果得られた相関値である。 In addition, when performing SAD calculation on an image that has been previously filtered, filtering processing such as a differential filter represented by a Sobel filter or a bandpass filter such as a LOG filter is first applied to the R image and the B image. Apply. After that, the correlation calculation is performed by the SAD calculation shown in the following formula 2.
[Equation 2]

Here, I ′ is a partial region of the R image after the filtering process is performed, T ′ is a partial region of the B image after the filtering process is performed (partial region of the same size as I ′), and M is a partial region The horizontal width [pixel] of the area, and N is the vertical width [pixel] of the partial area. In this case, RSAD is a correlation value obtained as a result of the correlation calculation.

(Step S4)
It is determined whether or not the phase difference detection processing for all the target pixels in the image is completed. Then, until the completion, the processing of step S2 and step S3 is repeated while shifting the position of the target pixel. Here, all the target pixels in the image indicate all pixels in the image that can detect the phase difference. It should be noted that pixels that cannot detect the phase difference may be left undetected, or the phase difference amount may be calculated by performing interpolation or the like based on the detection result of the surrounding pixel of interest.

(Step S5)
Next, the target pixel for performing the correction process is set for the pixel for which the phase difference amount is obtained. Here, the pixel position of the target pixel is the same (that is, common) pixel position in the R image and the B image. Hereinafter, as the correction process, as described with reference to FIGS. 17 and 18, a case will be described in which a colorization process using only a filter without translation (shift) is performed.

(Step S6)
The shape of the R image filter for filtering the R image is acquired according to the phase difference amount of the target pixel. Here, the relationship between the phase difference amount and the filter shape for the R image is held in advance in the imaging apparatus as a table as shown in Table 1 below, for example. In the example shown in Table 1, the filter shape is determined by determining the size of the filter kernel, the deviation of the Gaussian filter from the center of the filter kernel, the standard deviation σ of the Gaussian filter (this standard deviation σ is the blur of the blur filter) It shows the degree of spread). Therefore, the shape of the R image filter can be acquired by referring to the table based on the phase difference amount of the target pixel.
[Table 1]

(Step S7)
Next, a filtering process is performed on a neighboring region including the target pixel and its neighboring pixels in the R image, and a filter output value at the target pixel is acquired. Then, the acquired filter output value is copied to the target pixel position of the R copy image, and the R copy image is updated.

(Step S8)
The shape of the B image filter for performing filtering processing on the B image is acquired according to the phase difference amount of the target pixel. Here, the relationship between the phase difference amount and the filter shape for the B image is held in advance in the imaging apparatus as a table as shown in Table 2 below, for example. Also in the example shown in Table 2, the filter shape is determined by the size of the filter kernel, the deviation of the Gaussian filter from the center of the filter kernel, and the standard deviation σ of the Gaussian filter. Therefore, the shape of the B image filter can be acquired by referring to the table based on the phase difference amount of the target pixel.
[Table 2]

(Step S9)
Next, a filtering process is performed on a neighboring region including the target pixel and its neighboring pixels in the B image, and a filter output value at the target pixel is acquired. Then, the acquired filter output value is copied to the target pixel position of the B copy image, and the B copy image is updated.

(Step S10)
It is determined whether or not the filtering process for all the target pixels in the image has been completed. Then, the processes in steps S5 to S9 are repeated while shifting the position of the target pixel until the process is completed.

  In this way, when it is determined that the filtering process for all the target pixels has been completed, the R copy image and the B copy image are output as corrected images for the R image and the B image, and the colorization process is ended.

In step S6 or step S8, instead of using the circular Gaussian filter, when using an elliptical Gaussian filter as shown in FIG. 21 or FIG. 22, the standard deviation value of the elliptical Gaussian filter is set to x. Since the direction and the y direction may be set separately, the parameter table relating to the filter shape held in the imaging apparatus is as shown in Table 3 below, for example, excluding the deviation from the filter kernel center.
[Table 3]

  Although not described here, the deviation from the filter kernel center when the elliptical Gaussian filter is used in step S6 is the deviation from the filter kernel center when the elliptical Gaussian filter is used in step S8, for example, as in Table 1. For example, it may be the same as in Table 2.

  Further, in the above description, an example in which the shape of the filter corresponding to the phase difference amount is acquired using the table with reference to Table 1 or Table 2 is not limited thereto. For example, the correspondence between each parameter for determining the filter shape and the phase difference amount is held as, for example, an equation, the phase difference amount is substituted into the equation, and the filter shape is determined by calculation or the like. It doesn't matter if you do.

  Note that the processing for detecting the phase difference from the image captured through the band limiting filter 12 and the colorization processing for correcting the color misregistration do not necessarily have to be performed in the form of the imaging device, and the image processing device is separate from the imaging device. 41 (for example, a computer that executes an image processing program) may be performed.

  FIG. 24 is a block diagram illustrating a configuration of the image processing apparatus 41 in the 2D embodiment 1.

  The image processing apparatus 41 includes, from the body unit 2 of the imaging apparatus illustrated in FIG. Contrast AF control unit 38 (including AF assist control unit 38a) related to AF control, body side communication connector 35 related to communication with the lens unit 1, strobe control circuit 33 and strobe 34 related to subject illumination, and state of the imaging device The sensor unit 31 for obtaining the information is removed, and a recording unit 42 for recording information input from the interface (IF) 28 is further provided. The recording unit 42 is configured to output information input and recorded via the interface 28 to the image processing unit 25. Information can be recorded from the image processing unit 25 to the recording unit 42. The recording unit 42 is controlled by the system controller 30A (the reference number of the system controller is 30A in accordance with the removal of the contrast AF control unit 38) together with the interface 28. In addition, since a release button and the like for imaging are not necessary, the reference numeral of the operation unit is 32A.

  In addition, a series of processing in the image processing apparatus 41 is performed as follows, for example. First, an image is captured using an imaging device including the band limiting filter 12 and is recorded on the recording medium 29 as a RAW image output from the imaging circuit 23. Further, information related to the shape of the band limiting filter 12, lens data of the imaging optical system 9, and the like are also recorded on the recording medium 29.

  Next, the recording medium 29 is connected to the interface 28 of the image processing apparatus 41, and an image and various types of information are recorded in the recording unit 42. When recording is completed, the recording medium 29 may be removed from the interface 28.

  Thereafter, the image and various types of information recorded in the recording unit 42 are read out, and the phase difference is calculated by the distance calculating unit 39 or the color shift by the color image generating unit 37 is performed in the same manner as the imaging device described above. Coloring processing to be corrected is performed, or stereoscopic image generation processing by the stereo image generation unit 40 as described later is performed.

  Thus, the colorized image and the stereoscopic image processed by the image processing device 41 are recorded in the recording unit 42 again. Further, the colorized image and stereoscopic image recorded in the recording unit 42 are displayed on the display unit or transmitted to an external device via the interface 28. Thus, in the external device, the colorized image and the stereoscopic image can be used for various purposes.

  According to the 2D embodiment 1 of such colorization processing, the barycentric positions of the R image and the B image can be determined by using the blur filter itself or by translating (shifting) it before using the blur filter. Since it is approximated to the position of the center of gravity of the G image, which is an image, it is possible to obtain a more preferable image for viewing with reduced color misregistration.

At this time, since the blur shape of the R image and the B image is approximated to the blur shape of the G image, which is the standard image, by the blur filter, a natural blur shape and a more preferable image for viewing are obtained. Can do.
[2D Example 2]

  25 to 30 show a 2D embodiment 2 of the colorization process, and FIG. 25 is a diagram showing an outline of the colorization process performed by the color image generation unit 37 in the 2D embodiment 2.

  In this 2D embodiment 2, the same parts as those of the above-mentioned 2D embodiment 1 are denoted by the same reference numerals, description thereof will be omitted, and only different points will be mainly described.

  In this 2D embodiment, the colorization processing in the color image generation unit 37 is different from the 2D embodiment 1 described above. That is, in the 2D embodiment 1 described above, correction of color misregistration is performed using a blur filter, but in the present 2D embodiment, image copy addition processing is performed.

  With reference to FIG. 25, the concept of the colorization process in the present 2D embodiment when the subject is on the far side from the in-focus position will be described.

  When the subject is on the far side from the in-focus position, as shown in FIG. 10, the blur of the R image has a shape in which the left half of the blur of the G image is missing, and the blur of the B image is The right half of the blur of the G image is missing.

  Therefore, as shown in FIG. 25, in order to approximate the blur of the R image to the blur of the G image, the blur of the R image is copied and added to the missing portion of the blur of the R image, and the blur of the B image is changed to the B image. Copy addition is added to the missing part of the blur.

  It is desirable that the shape of the partial area to be copied and added matches the shape of the missing portion of the blur in the R image and the B image, but here, in order to simplify the processing, a rectangular (for example, square) area is used. Yes.

  The blur diffusion partial region G1 and the blur diffusion partial region G2 are shown in the circular blur diffusion region in the G image. Here, the blur diffusion partial region G1 and the blur diffusion partial region G2 are arranged in the same size at positions symmetrical to the vertical line Cg passing through the center of gravity of the blur diffusion region forming the circular shape of the G image. ing. In addition, the sizes of the partial areas G1 and G2 are preferably about the size of the radius of the circular blur diffusion area in consideration of fulfilling the function of color misregistration correction.

  On the other hand, the blur diffusion partial region R1 shown for the R image is a region having the same size and the same size as the blur diffusion partial region G2. In the blur diffusion region of the R image, the blur diffusion partial region R2 having the same size and the same position as the blur diffusion partial region G1 is insufficient.

  Similarly, the blur diffusion partial region B1 shown for the B image is a region having the same size and the same size as the blur diffusion partial region G1. The blur diffusion region B2 of the B image lacks the blur diffusion partial region B2 having the same size and the same position as the blur diffusion partial region G2.

  Therefore, a movement amount that completely overlaps the blur diffusion partial region G1 when the blur diffusion partial region G2 is moved in the blur diffusion partial region R1 of the R image (for example, a movement amount about the radius of the circular blur diffusion region G1). The blur diffused partial area R2 of the R image is generated by moving and moving the image (to an R copy image that is a copy of the R image, as will be described later). The blur diffusion partial region R2 is a region corresponding to the blur diffusion partial region G1 of the G image (or the blur diffusion partial region B1 of the B image).

  Similarly, the blur diffusion partial region B1 of the B image is moved by a movement amount (same as above) that completely overlaps the blur diffusion partial region G2 when the blur diffusion partial region G1 is moved (as will be described later, B The blur diffusion partial area B2 of the B image is generated by performing the copy addition (to the B copy image which is a copy of the image). This blur diffusion partial area B2 is an area corresponding to the blur diffusion partial area G2 of the G image (or the blur diffusion partial area R1 of the R image).

  By such processing, it is possible to compensate for the blur diffusion partial region that is lacking in each of the R image and the B image, and as a result, it is possible to approximate the blur diffusion regions of the G image, the R image, and the B image. Thus, the center of gravity of the blur diffusion region of the R image is close to the center of gravity of the blur diffusion region of the G image, and the center of gravity of the blur diffusion region of the B image is close to the center of gravity of the blur diffusion region of the G image.

  By performing such processing on the entire R image and B image, a color image with corrected color misregistration can be obtained.

  Next, FIG. 26 is a diagram illustrating a partial region set in the R image and the B image when performing phase difference detection in the 2D embodiment 2, and FIG. FIG. 28 is a diagram showing a state of adding a copy to a copy image, FIG. 28 is a diagram showing a state of copying and adding a blur diffusion partial area of an original B image to a B copy image in 2D Example 2, and FIG. FIG. 30 is a flowchart illustrating a colorization process performed by the color image generation unit 37 in the 2D embodiment 2 in accordance with an example of changing the size of the blur diffusion partial area according to the amount. It demonstrates along FIG. 30, referring FIG. 26-FIG. 29 suitably.

(Step S21)
When this process is started, initialization is performed. In this initial setting, first, an RGB image to be processed (that is, an R image, a G image, and a B image) is read. Here, when the input image is a Bayer image, it is assumed that the demosaicing process is performed in advance in the image processing unit 25.

  Next, in order to correct the phase difference between the R image and the B image, color differences Cr and Cb are used instead of using the R image and the B image itself. This is because a general subject has an advantage that the correction process can be performed stably because the variation of the pixel value is smoother in the color difference image than in the RGB image (however, R The image and the B image may be used as they are.In this case, in the following processing, the reading of Cr → R and Cb → B may be performed).

That is, for example, based on the following Equation 3, the color difference amount between the R image and the G image is calculated to generate a Cr image that is a color difference image, and the color difference amount between the B image and the G image is calculated to calculate the color difference image. A certain Cb image is generated.
[Equation 3]
Cr = R-G
Cb = BG

As other calculation methods for calculating the color difference signals Cr and Cb (and the luminance signal Y) from the RGB signals,
[Equation 4]
Y = 0.29900R + 0.58700G + 0.11400B
Cr = 0.50000R-0.41869G-0.0811B
Cb = −0.16874R−0.33126G + 0.50000B
Is widely known, instead of Equation 3, Equation 4 may be used.

  Next, a Cr copy image Cr1 (see FIG. 27) which is a copy image of the original Cr image Cr0 and a Cb copy image Cb1 (see FIG. 28) which is a copy image of the original Cb image Cb0 are created. Further, a Cr count image and a Cb count image having the same size as the Cr copy image Cr1 and the Cb copy image Cb1 are generated (here, in these count images, the initial value of the pixel value is set to 1 for all pixels). .

(Step S22)
Subsequently, a partial region for performing phase difference detection is set. Here, the partial area is set to one of the R image and the B image, here, for example, the R image.

(Step S23)
A phase difference with respect to the partial region set in step S22 is detected. This phase difference is detected by using the partial area set in the R image as a standard image and the partial area of the B image having the same size as the standard image as a reference image as shown in FIG. By doing so, phase difference detection is performed between the R image and the B image.

(Step S24)
Based on the phase difference amount obtained by the process of step S23, the radius of the circular blur of the G image (or the radius of the semicircular blur of the R image and the B image) is acquired. Here, the relationship between the phase difference amount and the blur radius of the G image is held in advance in the imaging apparatus as a table, a mathematical expression, or the like. Therefore, the blur radius can be acquired by referring to a table or performing a calculation using a mathematical formula based on the phase difference amount. Note that the process of step S24 may be omitted, and a simple method using the phase difference amount acquired in step S23 in place of the blur radius may be applied. In this case, the relationship between the phase difference amount and the blur radius need not be held in the imaging apparatus in advance.

(Step S25)
Next, a partial region is read from the original Cr image Cr0, shifted by a predetermined amount corresponding to the phase difference detected in step S23, and then copied and added to the Cr copy image Cr1. Here, the predetermined amount for shifting the partial area is an amount including the shifting direction, and the size thereof is, for example, the blur radius acquired in step S24. The reason why the Cr copy image Cr1 is created in the above-described step S21 is that it is necessary to hold the original Cr image Cr0 separately from the Cr copy image Cr1 whose pixel value is changed by copy addition ( The same applies to the Cb copy image Cb1). However, it is not always necessary to prepare a copy image when, for example, partial areas are processed in parallel operation instead of sequentially processing in the order of raster scanning.

(Step S26)
Subsequently, +1 is added to the area of “the position where the copy addition process was performed in step S25” of the Cr count image so that the number of times of addition can be understood. This Cr count image is used to perform pixel value normalization processing in the subsequent step S30.

(Step S27)
Further, the partial area is read from the same position as “the position copied to the Cr copy image Cr1 in step S25” in the original Cb image Cb0, and the copy source from the original Cr image Cr0 in “step S25” of the Cb copy image Cb1. A copy is added to “the position from which data was acquired”. Thus, the predetermined amount for shifting the Cb image has the same absolute value as the predetermined amount for shifting the Cr image, but the direction is reversed.

(Step S28)
Then, the number of times of addition can be seen in the region of “the position where the copy source data was acquired from the original Cr image Cr0 in step S25” (that is, the position where the copy addition process was performed in step S27) of the Cb count image. Add +1 to. This Cb count image is also used to perform pixel value normalization processing in the subsequent step S30.

  In step S25 and step S27 described above, the copy addition process is performed for each partial area of the image, but this partial area may be the same as the partial area for which the phase difference detection process was performed in step S23. On the other hand, it may be a partial area having a size different from that of the phase difference detection process.

  In addition, the size of the partial area used for the copy addition process may be constant for the entire image (that is, global size), but may be different for each partial area set in the image (that is, local size). (Even if it is a big size).

  For example, the size of the partial region used in steps S25 to S28 may be changed as shown in FIG. 29 according to the phase difference amount detected in step S23.

  In the example shown in FIG. 29, when the phase difference amount is 0, the vertical size and the horizontal size of the partial region are both 1, and the partial region is 1 pixel. In this case, since the phase difference amount is 0, the above-described copy addition process is not performed, and no process is substantially performed. Therefore, the processing may be branched depending on whether or not the phase difference amount is 0, and no processing may be performed when the phase difference amount is 0.

  In the example shown in FIG. 29, the size of the partial area is increased in proportion to the phase difference amount. Since the inclination of the straight line at this time is appropriately set according to the configuration of the optical system, a specific scale is not shown in FIG.

  Note that FIG. 29 shows an example in which the relationship between the phase difference amount and the size of the partial area is proportional, but of course, the relationship is not limited to the proportional relationship. For example, for subjective image quality evaluation. Accordingly, the design may be made so that the size of the partial region with respect to the phase difference amount is appropriate.

  Further, the amount of diffusion when the light from the point light source is blurred (the amount of point spread of PSF (Point Spread Function)) is not necessarily uniform in the region where the blur occurs. For example, it is conceivable that the amount of diffusion is smaller (that is, the luminance is lower) in the peripheral portion of the blur than in the central portion. Therefore, when performing copy addition of partial areas as described above, a weighting factor corresponding to the amount of blur diffusion may be multiplied. For example, each pixel in the peripheral part of the partial region is multiplied by a weighting factor of 1/2, each pixel in the central part of the partial region is multiplied by one weighting factor, and then copy addition is performed. At this time, also in the count images in step S26 and step S28, 1/2 is added to the peripheral part of the partial area, and 1 is added to the central part of the partial area.

(Step S29)
Thereafter, it is determined whether or not the processing for all the partial areas in the image is completed. Then, the processes in steps S22 to S28 are repeated while shifting the position of the partial area until the process is completed. Here, an arbitrary value can be set as the step for shifting the partial area, but it is preferably a value smaller than the width of the partial area.

(Step S30)
Thus, if it is determined in step S29 that the processing for all the partial areas has been completed, the normalized Cr by dividing the pixel value of the Cr copy image by the pixel value of the Cr count image for each same pixel position. A copy image is obtained, and a normalized Cb copy image is obtained by dividing the pixel value of the Cb copy image by the pixel value of the Cb count image.

(Step S31)
Then, an R image, a B image, and a G image are generated using the G image (or Y image) and the Cr copy image and the Cb copy image normalized in step S30. Here, when Equation 3 is used to calculate the color difference image,
[Equation 5]
R = Cr + G
B = Cb + G
Are used to generate an R image and a B image (the G image in step S22 is used as it is).

In addition, when Equation 4 is used to calculate the color difference image,
[Equation 6]
R = Y + 1.40200Cr
G = Y−0.71414Cr−0.34414Cb
B = Y + 1.77200Cb
Is used to generate an R image, a B image, and a G image.

  The RGB image calculated in step S31 is an image obtained by the colorization process in the color image generation unit 37.

  According to the 2D embodiment 2 of such colorization processing, effects similar to those of the 2D embodiment 1 described above can be achieved by correcting the color misregistration by the image copy addition processing.

Further, when performing copy addition, there is an advantage that the processing load can be reduced as compared with the filtering process. Therefore, the cost of the processing circuit and the power consumption can be reduced.
[2D Example 3]

  FIGS. 31 to 34 show the 2D embodiment 3 of the colorization processing. FIG. 31 is a diagram showing an outline of the PSF table of each color according to the phase difference amount in the 2D embodiment 3. FIG. 32 shows the 2D embodiment. FIG. 33 is a diagram illustrating an outline of colorization processing performed by the color image generation unit 37 in the third embodiment, and FIG. 33 is a diagram illustrating an overview of colorization processing accompanied by blur amount control performed by the color image generation unit 37 in 2D Example 3. FIG. 34 is a flowchart showing the colorization processing performed by the color image generation unit 37 in the 2D embodiment 3.

  In this 2D embodiment 3, parts that are the same as those in the above-described 2D embodiments 1 and 2 are given the same reference numerals, description thereof is omitted, and only differences are mainly described.

  In this 2D embodiment, the colorization processing in the color image generation unit 37 is different from the 2D embodiments 1 and 2 described above. That is, in the 2D embodiment, the colorization process is a restoration process (inverse filtering process) for restoring a blurred image to a non-blurred image, and a circular blur shape corresponding to the subject distance for the restored non-blurred image. Is combined with a filtering process for obtaining the above.

  When the band limiting filter 12 as shown in FIG. 3 is used, the PSF of each color of RGB changes as shown in FIG. 31 according to the phase difference amount. In FIG. 31, the dark part where the light from the point light source does not reach is hatched.

  For example, PSFr, which is the PSF of the R image, shows a larger half-moon shape as the absolute value of the phase difference increases, and converges to one point when the phase difference is 0, that is, at the in-focus position. Further, when it is closer than the in-focus position, it has a left half moon shape as shown in the upper half of FIG. 31, but when it is far from the in-focus position, it is shown in the lower half of FIG. It becomes a right half moon shape.

  Similarly, PSFb, which is the PSF of the B image, shows a larger half-moon shape as the absolute value of the phase difference increases, and converges to one point when the phase difference is 0, that is, at the in-focus position. Further, when it is closer than the in-focus position, it has a right half moon shape as shown in the upper half of FIG. 31, but when it is far from the in-focus position, it is shown in the lower half of FIG. The left half-moon shape.

  Further, PSFg, which is the PSF of the G image, shows a larger full moon shape as the absolute value of the phase difference amount increases, and converges to one point when the phase difference amount is 0, that is, at the in-focus position. Further, PSFg is always in a full moon shape regardless of whether it is closer or farther than the in-focus position except when it converges to one point.

  A PSF table for each color corresponding to the phase difference amount as shown in FIG. 31 is assumed to be stored in advance in, for example, a non-illustrated nonvolatile memory in the color image generation unit 37 (however, in FIG. 1). In the case of the lens-interchangeable image pickup apparatus as shown, of course, the table stored in the lens control unit 14 may be received and used by communication).

  Such restoration processing and filtering processing using PSF will be described with reference to FIG. FIG. 32 shows an example of the case where the subject is farther than the in-focus position, but the same processing can be applied when the subject is closer than the in-focus position.

  First, by performing inverse calculation of PSFr on the R image (showing that -1 on the right shoulder is the inverse calculation, the same applies hereinafter), the right semicircular blur is converged to a single point light source. The image is converted into an image having no blur (either one of the restored first image and the restored second image). Similarly, by performing inverse calculation of PSFb on the B image, the left semicircular blur is converted into a non-blurred image in which the point light source converges to one point (the other one of the restored first image and the restored second image). ). By such processing, restoration processing of the R image and the B image is performed.

  Next, PSFg which is a PSF for the G image is applied to the restored R image and the restored B image. As a result, full-moon-shaped blur similar to the G image is generated in the R image and the B image.

  A specific process is performed as follows.

  First, it is assumed that the blur PSF of the R image at a certain pixel position is Pr1, the blur PSF at the same pixel position of the B image is Pb1, and the blur PSF at the same pixel position of the G image is Pg1. As shown in FIG. 31, the blurred PSF differs depending on the phase difference between the R image and the B image, and the phase difference basically differs for each pixel position. Therefore, the PSF is different for each pixel position. To be determined. Further, it is assumed that the PSF is defined for a partial region including a plurality of neighboring pixels with the pixel position of interest at the center (see FIG. 31).

  When the processing is started, the phase difference obtained for the pixel of interest is acquired, and by referring to a table as shown in FIG. 31, Pr1 which is a PSF centered on the pixel of interest in the R image and the B image Pb1 that is the PSF centered on the pixel of interest at the same position in Pb1 and Pg1 that is the PSF centered on the pixel of interest at the same position in the G image are acquired.

Next, for the obtained Pr1, Pb1, and Pg1, the partial region r centered on the target pixel in the R image, and the partial region b centered on the target pixel in the B image, the following Expression 7 is shown. As described above, the two-dimensional Fourier transform FFT2 is performed to obtain values PR1, PB1, PG1, R, and B after the respective transformations.
[Equation 7]
PR1 = FFT2 (Pr1)
PB1 = FFT2 (Pb1)
PG1 = FFT2 (Pg1)
R = FFT2 (r)
B = FFT2 (b)

Next, as shown in Equation 8 below, R is divided by PR1, B is divided by PB1, restoration processing is performed, and each is multiplied by PG1 to perform filtering processing. Two-dimensional inverse Fourier transform IFFT2 is performed to calculate an R image r ′ and a B image b ′ in which the same blur as the G image is obtained.
[Equation 8]
r ′ = IFFT2 (R × PG1 / PR1)
b ′ = IFFT2 (B × PG1 / PB1)

In order to make the restoration process by the inverse filtering process of Expression 8 more stable, the restoration amount of the restoration process may be controlled as shown in Expression 9 below (Wiener filter method).
[Equation 9]
r ′ = IFFT2 (R × PG1 / PR1 × | PR1 | ^ 2 / (| PR1 | ^ 2 + Γ))
b ′ = IFFT2 (B × PG1 / PB1 × | PB1 | ^ 2 / (| PB1 | ^ 2 + Γ))
Here, the symbol “^ 2” in Equation 9 represents the square (the same applies hereinafter), and Γ is an arbitrary constant appropriately set according to the shapes of PR1 and PB1.

  By adopting a process such as Equation 9, amplification of noise during the restoration process is suppressed, and more preferable R and B images can be generated.

  As a method for setting Γ, for example, the relationship between the absolute values | PR1 | and | PB1 | of the frequency coefficient of PR1 and the preferable values of Γ with respect to the absolute values | PR1 | and | PB1 | A method of designating Γ for each frequency coefficient based on this relationship can be considered. As another method, a configuration in which the amount of noise included in an image is estimated based on various parameters (ISO value, focal length, aperture value, etc.) of the imaging device, and Γ is changed according to the estimated amount of noise. (For example, Γ = 0.01 for ISO 200 and Γ = 0.04 for ISO 800) may be employed.

  The restoration process and the filtering process as described above are performed for each partial area of the image. When the correction process is completed in a certain partial area, the partial area is designated by slightly shifting the position, and the designated partial area is again displayed. Similar restoration processing and filtering processing are performed. By repeatedly performing such processing, restoration processing and filtering processing are performed on the entire region of the image. Thereafter, with respect to the pixel position processed in duplicate, the sum of a plurality of corrected pixel values processed at the pixel position is divided by the number of corrections and averaged to obtain a normalized corrected image.

  In addition, it is preferable that the size of the partial area for performing the restoration process and the filtering process is larger than the shape of the blur. Therefore, it is conceivable to adaptively change the size of the partial region in accordance with the size of the blur. Alternatively, when it is known in advance in which range the shape of the blur changes according to the phase difference, a partial region having a size larger than the maximum size of the blur may be used in a fixed manner.

  Next, with reference to FIG. 34, the flow of the colorization process performed by the color image generation unit 37 by the above-described restoration process and filtering process will be described.

(Step S41)
When this process is started, initialization is performed. In this initial setting, first, an RGB image to be processed (that is, an R image, a G image, and a B image) is read. Next, an R copy image that is a copy of the R image and a B copy image that is a copy of the B image are created.

  Subsequently, an R count image and a B count image having the same size as the R copy image and the B copy image are generated (here, in these count images, the initial value of the pixel value is set to 1 for all pixels).

(Step S42)
Subsequently, a partial region for performing phase difference detection is set. Here, the partial area is set to one of the R image and the B image, here, for example, the R image.

(Step S43)
A phase difference with respect to the partial region set in step S42 is detected. This phase difference is detected by using the partial area set in the R image as a standard image and the partial area of the B image having the same size as the standard image as a reference image, as shown in FIG. By performing the above, phase difference detection is performed between the R image and the B image.

(Step S44)
Based on the amount of phase difference obtained by the process of step S43, the radius of the circular blur of the G image (or the radius of the semicircular blur of the R image and the B image) is set in the same manner as in step S24 described above. Get.

(Step S45)
Next, the above-described restoration process and filtering process are performed on the original R image with respect to the partial region specified in step S42 described above. The processing result obtained in this way is copied and added to the same position as the partial area of the original R image in the R copy image. Here, an example in which the partial area for performing the colorization process is the same as the partial area for performing the phase difference detection will be described, but a different area may be set as a matter of course, as described above. Alternatively, a partial area having an adaptive size according to the detected phase difference may be used (the same applies to the B image described later).

(Step S46)
Subsequently, +1 is added to the “partial region designated in step S42” of the R count image so that the number of times of addition can be understood. This R count image is used to perform pixel value normalization processing in the subsequent step S50.

(Step S47)
Further, the restoration process and the filtering process as described above are performed on the original B image with respect to the partial region specified in step S42 described above. The processing result obtained in this way is copied and added at the same position as the partial area of the original B image in the B copy image.

(Step S48)
Then, 1 is added to the “partial region designated in step S42” of the B count image so that the number of times of addition can be understood. This B count image is also used to perform pixel value normalization processing in the subsequent step S50.

(Step S49)
Thereafter, it is determined whether or not the processing for all the partial areas in the image is completed. Then, the processes in steps S42 to S48 are repeated while shifting the position of the partial area until the process is completed. Here, an arbitrary value can be set as the step for shifting the partial area, but it is preferably a value smaller than the width of the partial area.

(Step S50)
Thus, if it is determined in step S49 that the processing for all the partial areas has been completed, the R value normalized by dividing the pixel value of the R copy image by the pixel value of the R count image for each same pixel position. A copy image is obtained, and a normalized B copy image is obtained by dividing the pixel value of the B copy image by the pixel value of the B count image. The RGB image calculated in step S50 is an image obtained by the colorization process in the color image generation unit 37.

  Thus, when the process of step S50 ends, the process shown in FIG. 34 ends.

  In the above description, restoration processing and filtering processing are performed after transforming from real space to frequency space using Fourier transform. However, the present invention is not limited to this, and restoration processing or filtering processing (for example, MAP estimation processing) in real space may be applied.

  In the above description, the blurring shape of the R image and the B image is matched with the blurring shape of the G image. However, in addition to this, the blur amount control may be performed.

  In this case, as shown in FIG. 33, first, the inverse operation of PSFr is performed on the R image, the inverse operation of PSFb is performed on the B image, and further, the inverse operation of PSFg is performed on the G image. The shape blur is converted into a blur-free image (restored third image) in which the point light source converges to one point. Thereby, the R image, the B image, and the G image are restored.

  Next, PSF′g, which is a desired PSF for the G image, is applied to the restored R image, B image, and G image. As a result, a full-moon-shaped blur having a desired size can be generated in the R image, the B image, and the G image, and the blur amount can be controlled.

  Specifically, by referring to a table as shown in FIG. 31, a desired Pg1 'is further acquired as a PSF centered on the target pixel at the same position in the G image.

Next, in addition to the processing of Equation 7 described above, two-dimensional Fourier is obtained as shown in Equation 10 below for the acquired Pg1 ′ and the partial region g centered on the pixel of interest in the G image. A conversion FFT2 is performed to obtain converted values PG1 ′ and G.
[Equation 10]
PG1 ′ = FFT2 (Pg1 ′)
G = FFT2 (g)

Then, as shown in Equation 11 below, R is divided by PR1, B is divided by PB1, G is divided by PG1, restoration processing is performed, and each is multiplied by PG1 ′ to perform filtering processing. The result is subjected to a two-dimensional inverse Fourier transform IFFT2 to calculate an R image r ″, a B image b ″, and a G image g ″ from which a desired amount of blur has been obtained.
[Equation 11]
r ″ = IFFT2 (R × PG1 ′ / PR1)
b ″ = IFFT2 (B × PG1 ′ / PB1)
g ″ = IFFT2 (G × PG1 ′ / PG1)

Note that, similarly to the above-described equation 9, a Wiener filter method as shown in the following equation 12 may be employed instead of the equation 11.
[Equation 12]
r ″ = IFFT2 (R × PG1 ′ / PR1 × | PR1 | ^ 2 / (| PR1 | ^ 2 + Γ))
b ″ = IFFT2 (B × PG1 ′ / PB1 × | PB1 | ^ 2 / (| PB1 | ^ 2 + Γ))
g ″ = IFFT2 (G × PG1 ′ / PG1 × | PG1 | ^ 2 / (| PG1 | ^ 2 + Γ))
Here, Γ is an arbitrary constant appropriately set according to the shapes of PR1, PB1, and PG1.

  According to the 2D embodiment 3 of such colorization processing, the same effects as the 2D embodiments 1 and 2 described above can be obtained by performing color shift correction by restoration processing using PSF and filtering processing. it can.

  Furthermore, if a restoration process is performed for the G image, and then a filtering process using an arbitrary PSF for the G image is performed on the R image, the B image, and the G image, the amount of blur of the RGB color image is controlled as desired. It is also possible to do.

  By performing any of the processes of the 2D embodiments 1 to 3 described above, a two-dimensional image without color misregistration is generated. The stereo image generation unit 40 of the 3D embodiment generates a color stereoscopic image (3D image) based on the two-dimensional image without color misregistration and the phase difference information.

  FIG. 35 is a diagram illustrating a state in which a left-eye image and a right-eye image are generated from a two-dimensional image after colorization processing, and FIG. 36 is a flowchart illustrating a stereoscopic image generation process by the stereo image generation unit 40.

(Step S61)
When this process is started, initialization is performed. In this initial setting, first, a color image RGB0 (see FIG. 35) in which the color misregistration is corrected by the colorization process is read. Next, information on the amount of phase difference corresponding to each partial area of the image to be processed is prepared in advance. Here, since the information on the phase difference amount has already been acquired in any of the colorization processes in the 2D embodiments 1 to 3, this information is used (however, the pixel for which the phase difference amount has not been acquired). If there is (for example, when the phase difference amount is acquired based on the image before demosaicing or when there is a pixel for which the phase difference amount cannot be acquired), interpolation processing or the like is performed. Thus, the phase difference amount for all pixels is generated). Further, a left-eye image RGB-L and a right-eye image RGB-R, and a left-eye count image and a right-eye count image are generated (here, each of these four images has an initial pixel value of 0 for all pixels). ).

(Step S62)
Next, a partial region is set as shown in FIG. 35 in the color image RGB0 in which the color misregistration is corrected. The size of the partial region may be the same size as the partial region used for measuring the phase difference amount, but may be an arbitrary size. For example, as described with reference to FIG. 29, the size of the partial region may be increased in accordance with the amount of phase difference (for example, in proportion).

(Step S63)
Subsequently, a phase difference amount corresponding to the partial region set in step S62 is acquired from the information of the phase difference amount prepared in advance in step S61.

(Step S64)
Then, according to the phase difference amount (the direction of the phase difference and the magnitude of the phase difference) generated between the R image and the B image, the partial area of the color image RGB0 is converted into the phase difference in the left-eye image RGB-L. Copy addition is performed on the partial area shifted by a half of the amount (see FIG. 35).

(Step S65)
Further, +1 is added to the pixel value of each pixel in the same area as the “partial area copied and added in step S64” of the left eye count image. This left-eye count image is used to perform pixel value normalization processing in the subsequent step S69.

(Step S66)
Similarly, according to the phase difference amount generated between the R image and the B image, the partial region of the color image RGB0 is set to step S64 by a half of the phase difference amount in the right-eye image RGB-R. Copy addition is performed on the partial area shifted in the reverse direction (see FIG. 35).

(Step S67)
Similarly, +1 is added to the pixel value of each pixel in the same area as the “partial area copied and added in step S66” of the right eye count image. This right eye count image is used to perform pixel value normalization processing in the subsequent step S69.

(Step S68)
Thereafter, it is determined whether or not the processing for all the partial areas in the color image RGB0 is completed. Then, the processes in steps S62 to S67 are repeated while shifting the position of the partial area until the process is completed. Here, an arbitrary value can be set as the step for shifting the partial area, but it is preferable that the value be smaller than the width of the partial area (that is, the partial areas should be sequentially set while overlapping). . As a specific example, for example, the partial area may be set while being shifted by 10 [pixel] with respect to the partial area having a size of 51 × 51 [pixel]. However, they do not have to overlap, and may be set so that tiles are laid out (for example, a partial area is set while shifting by 51 [pixels] with respect to a partial area having a size of 51 × 51 [pixels]. Etc.)

(Step S69)
Thus, if it is determined in step S68 that the processing for all the partial regions has been completed, the pixel value of the left eye image RGB-L is normalized by dividing the pixel value of the left eye count image for each same pixel position. In addition to obtaining a left eye image, a normalized right eye image is obtained by dividing the pixel value of the right eye image RGB-R by the pixel value of the right eye count image. In addition, when this process is completed and a pixel to which no pixel value is given remains, an interpolation process is performed to provide the pixel value of the pixel.

  Thus, when the process of step S69 is completed, the process shown in FIG. 34 is terminated.

  In the above description, when the partial area of the color image RGB0 is copied and added to the left-eye image RGB-L and the right-eye image RGB-R, the partial area is shifted by a half of the phase difference amount. This is because it is preferable to be a half of the phase difference amount because it is faithful to the parallax amount of the actually observed stereoscopic image. However, it is not limited to this. For example, if a value obtained by multiplying the acquired phase difference amount by a predetermined constant greater than 1 is used as the corrected phase difference amount, a stereoscopic image with a more pronounced stereoscopic effect can be generated. In addition, if a value obtained by multiplying the acquired phase difference amount by a predetermined constant smaller than 1 is used as the corrected phase difference amount, a stereoscopic image in which the stereoscopic effect is further suppressed can be generated. The predetermined constant by which the phase difference amount is multiplied is determined according to preference by the user viewing the displayed image while operating the operation unit 32 of the imaging apparatus while displaying the stereoscopic image on the display unit 27. It may be possible to make adjustments (desired adjustments by the user via the GUI). Alternatively, depending on the subject to be imaged (for example, depending on whether the subject is a person, the distance to the subject, the composition, etc.), the system controller 30 automatically You may make it set to.

  According to the 3D Example 1 as described above, a stereoscopic image having a color preferable for viewing can be obtained from an image photographed by light that has passed through different pupil regions of the imaging optical system depending on the band.

Further, in the 3D embodiment 1, since the color stereoscopic image is generated using the result of the colorization process, the image after the colorization process obtained in the process is stored. In this case, there is an advantage that both the stereoscopic image and the image after colorization with reduced color shift can be acquired and observed as desired.
[3D Example 2]

  FIG. 37 to FIG. 40 show 3D Example 2 according to the embodiment of the present invention. FIG. 37 shows the left eye image from the image of the subject on the far side from the in-focus position in the stereoscopic image generation processing. FIG. 38 is a diagram illustrating a shift state when generating a right-eye image from an image of a subject that is farther than the in-focus position in the stereoscopic image generation process. FIG. 39 is a diagram illustrating a state after performing a filter process for approximating the blurred shape of each color in the left-eye image after the shift, and FIG. 40 is a diagram after performing a filter process for approximating the blurred shape of each color in the shifted right-eye image FIG.

  In this 3D Example 2, the same parts as those in the above-mentioned 3D Example 1 are denoted by the same reference numerals, description thereof will be omitted, and only different points will be mainly described.

  In the 3D embodiment 1 described above, a stereoscopic image is generated using the result of the colorization process by the color image generation unit 37. However, the 3D embodiment 2 requires the result of the colorization process. Instead, a stereoscopic image is generated.

  In order to simplify the description below, the case where the subject is on the far side from the in-focus position will be described as an example, but the following processing is performed even when the subject is on the near side from the in-focus position. It is possible to apply in the same manner by appropriately changing.

  First, when the band limiting filter 12 as shown in FIG. 3 is used, the blur of each color image is as shown in FIG. 10 when the subject is on the far side from the in-focus position.

  As can be seen from FIG. 3, the RG filter 12r is located on the left eye side, and the GB filter 12b is located on the right eye side. Accordingly, the center-of-gravity position Cr of the R-component subject image IMGr shown in FIG. 10 is the position to be the center-of-gravity position of each color component in the left-eye image, and the center-of-gravity position Cb of the B-component subject image IMGb shown in FIG. This is the position that should be the center of gravity of each color component.

  Therefore, first, as described with reference to FIG. 26, the phase difference amount is detected.

  When generating the left-eye image, as shown in FIG. 37, the B-component subject image IMGb is moved (shifted) by the phase difference amount so that the centroid position matches the centroid position Cr of the R-component subject image IMGr. Then, the G component subject image IMGg is moved by ½ of the phase difference amount so that the center of gravity position coincides (or approaches) the center of gravity position Cr of the R component subject image IMGr.

  Similarly, when generating the right-eye image, as shown in FIG. 38, the R-component subject image IMGr is moved (shifted) by the phase difference amount to change the center-of-gravity position to the center-of-gravity position Cb of the B-component subject image IMGb. The G component subject image IMGg is moved by ½ of the phase difference amount to match (or approach) the center of gravity position to the center of gravity position Cb of the B component subject image IMGb.

  Thus, the color image after the shift shown in FIGS. 37 and 38 is a shift color image.

  Next, a filter kernel such as a circular Gaussian filter as shown in FIG. 20 or an elliptic Gaussian filter as shown in FIGS. 21 and 22 is applied to the R image and B image in the left-eye image after the shift. As shown in FIG. 39, the blur shape is approximated to the G image by applying the blur filter having the.

  Further, for the R image and the B image in the shifted right eye image, the circular Gaussian filter as shown in FIG. 20 similar to the left eye image, or the elliptical Gaussian filter as shown in FIG. 21 and FIG. By applying a blur filter having the filter kernels, the blur shape is approximated to a G image as shown in FIG.

  When using an elliptical Gaussian filter, for example, as shown in Table 3, the filtering process is performed while changing the filter shape according to the phase difference, as described above.

  Further, here, an example in which the blur filter is used to approximate the blur shape of the R image and the B image to the blur shape of the G image has been described, but FIGS. 31 to 34 are illustrated with respect to the 2D embodiment 3 of the color processing. In substantially the same manner as described with reference to the above description, the shape of the blur may be approximated or the size of the blur may be controlled by combining the restoration process and the filtering process.

  Further, also in the 3D embodiment 2, the amount of movement to be shifted is increased or decreased based on the corrected phase difference amount (a value obtained by multiplying the phase difference amount by a predetermined constant). May be made more prominent or the stereoscopic effect may be further suppressed.

According to the 3D Example 2 as described above, the same effect as the 3D Example 1 described above can be obtained, and since it is not necessary to perform the colorization process, the processing load is increased when only a stereoscopic image is to be acquired. It becomes possible to reduce.
[Embodiment 1]

  41 and 42 show the first embodiment of the present invention, FIG. 41 is a diagram showing an example of a phase difference image displayed on the display unit of the imaging apparatus, and FIG. 42 shows display processing in the imaging apparatus. It is a flowchart.

  The imaging apparatus according to the present embodiment is configured as illustrated in FIG. 1, and receives and photoelectrically converts light in a plurality of wavelength bands, and performs the first image in the first band and the second band. A color imaging device 22 that generates a second image and a third image in the third band, an imaging optical system 9 that forms a subject image on the imaging device 22, and the imaging device 22 via the imaging optical system 9. Is disposed on the optical path of the imaging light flux reaching to, and blocks the first band light in the imaging light flux that attempts to pass through the first area that is a part of the pupil area of the imaging optical system 9. And a second band in the imaging light flux that attempts to pass through a second region that is another part of the pupil region of the imaging optical system 9. Band-limiting filter 1 that performs the second band limitation that blocks the light of the first band and allows the light of the first band and the third band to pass. And a distance calculation unit 39 for calculating a phase difference amount between the first image and the second image, and a blur of the second image and the third image based on the phase difference amount calculated by the distance calculation unit 39. A one-eye color image in which the center of gravity position of the first image is moved in the direction of the center of gravity position of the blur of the first image is generated, and the center of gravity position of the blur of the first image and the third image is generated. A stereo image generator 40 that generates a color stereoscopic image by generating another one-eye color image moved in the direction of the center of gravity of the blur, and a phase difference between the first image and the second image And a display unit 27 for simultaneously displaying one color image for one eye and another color image for one eye.

  Here, in the present embodiment, for example, as described above, one of the first band and the second band is a blue (B) band and the other band is a red (R) band. And the third band is a green (G) band.

  As shown in FIG. 41, the phase difference image displayed on the screen 27a by the display unit 27 is an image in which the color left-eye image and the color right-eye image generated by the stereo image generation unit 40 are superimposed.

  In this case, as described above, the subject OBJc at the in-focus position does not become a color double image, and the subjects OBJf and OBJn at the far side or near-distance side from the in-focus position are in focus. A color double image showing a split according to the amount of deviation from the position is obtained. Accordingly, it is possible to easily confirm visually which subject is in focus and which subject is out of focus.

  Processing for displaying such an image will be described with reference to FIG. The process shown in FIG. 42 includes a number of steps already described in detail, and will be described briefly.

  When this process is started, first, it is determined whether or not the display to be performed is a live view display (step S71).

  If it is determined that the live view display is selected, it is further determined whether or not the focus mode is set to the manual focus mode (step S72).

  If it is determined that the manual focus mode is set, an image is input from the image sensor 22 (step S73), the intercolor correction unit 36 performs intercolor correction, and the image processing unit 25 performs demosaicing. In step S74, the distance calculator 39 detects the phase difference based on the demosaiced R image and B image (step S75).

  When the stereo image generation unit 40 generates a stereoscopic image according to the 3D embodiment 1, for example, first, the color image generation unit 37 performs colorization processing (step S76). However, when the stereo image generation unit 40 generates a stereoscopic image by the technique of the 3D embodiment 2 described above, it is possible to eliminate the colorization process in step S76.

  Subsequently, based on the result of the colorization process, the stereo image generation unit 40 creates color left-eye image data (step S77) and creates color right-eye image data (step S78).

  Furthermore, the generated left-eye image and right-eye image are combined (step S79), and the combined image is converted into display data by the image processing unit 25 (step S80). The display is as shown in FIG. 41 (step S81).

  If it is determined in step S72 that the manual focus mode is not set, it is determined that the mode is the autofocus mode, and the color image generation unit 37 performs the colorization process to perform two-dimensional without color misregistration. (2D) The live image is displayed on the screen 27a of the display unit 27 (step S82).

  If the process of step S81 or step S82 is performed, it is determined whether or not to end the live view display (step S83). If not, the process returns to step S71 to perform the above-described process.

  On the other hand, if it is determined in step S71 that it is not live view display, reproduction display for reproducing the captured image is performed (step S84), and reproduction display in step S84 is performed until the reproduction display ends (step S85). Repeat this step.

  If it is determined in step S83 that the live view display is to be ended, or if it is determined in step S85 that the reproduction display is to be ended, this process is ended.

According to the first embodiment, the subject at the in-focus position is not a double image, and the subject not at the in-focus position is a double image. Therefore, the in-focus portion in the image is not in focus. The part can be easily confirmed visually. In addition, since the subject that is not in the in-focus position becomes a double image that indicates the split according to the amount of deviation from the in-focus position, it is easy to sensuously grasp how much the image is out of the in-focus position. . In addition, since the phase difference image is a color image, the subject can be recognized with a natural feeling. Since a general-purpose image sensor can be used as the image sensor 22, the manufacturing cost can be reduced as compared with the case where a dedicated image sensor in which a light shielding film is formed on a pixel is used.
[Embodiment 2]

  43 and 44 show the second embodiment of the present invention, FIG. 43 is a diagram showing an example of a phase difference image displayed on the display unit of the imaging apparatus, and FIG. 44 shows display processing in the imaging apparatus. It is a flowchart.

  In the second embodiment, parts that are the same as those in the first embodiment are given the same reference numerals and description thereof is omitted, and only differences are mainly described.

  The configuration of the imaging apparatus of this embodiment is the same as that of the first embodiment described above, as shown in FIG.

  However, in the present embodiment, the stereo image generation unit 40 is a color in which the stereoscopic effect is enhanced based on the corrected phase difference amount obtained by multiplying the phase difference amount calculated by the distance calculation unit 39 by a predetermined coefficient larger than 1. The stereoscopic image is generated.

  That is, assuming that the amount of deviation between the left eye image and the right eye image of the subject OBJf on the far side shown in FIG. 41 of the first embodiment described above is z, in this embodiment, as shown in FIG. The phase difference image is displayed so that k × z obtained by multiplying z by a predetermined coefficient k greater than 1 is the amount of deviation between the left eye image and the right eye image of the subject OBJf.

  The processing for displaying such an image is as shown in FIG. This process is almost the same as the process shown in FIG. 42, but phase difference enhancement is performed when creating left-eye image data (step S77A), and phase difference enhancement is performed when creating right-eye image data (step S77A). S78A) is different.

According to the second embodiment, the effects similar to those of the first embodiment described above are obtained, the phase difference is emphasized, and the amount of deviation from the in-focus position of the subject that is not in the in-focus position increases. It is possible to more easily visually check the in-focus portion and the out-of-focus portion in the image.
[Embodiment 3]

  45 to 47 show the third embodiment of the present invention. FIG. 45 is a block diagram showing the detailed configuration of the image processing unit and the system controller in the imaging apparatus. FIG. 46 is a display on the display unit of the imaging apparatus. FIG. 47 is a flowchart showing display processing in the imaging apparatus.

  In the third embodiment, parts that are the same as those in the first and second embodiments are given the same reference numerals, description thereof is omitted, and only differences are mainly described.

  As shown in FIG. 45, the image processing unit 25 of the present embodiment includes a split image generation unit 45. The stereo image generation unit 40 is not shown in the image processing unit 25, but may be provided or omitted.

  The split image generation unit 45 is picked up by the image sensor 22, subjected to inter-color correction by the inter-color correction unit 36, and demosaiced by the image processing unit 25. And a second image of the second band color are combined to generate a split image. Specifically, the split image generation unit 45 combines the R image after the demosaicing and the B image to generate a split image.

  Therefore, the phase difference image displayed on the screen 27a of the display unit 27 is a split image, and the R image and the B image are superimposed on the subject OBJc at the in-focus position. The subjects OBJf and OBJn on the short distance side become a double image indicating a color shift in which the R image and the B image are split according to the shift amount from the in-focus position.

  Processing for displaying such an image will be described with reference to FIG.

  Assume that this processing is started and the processing up to step S74 is performed.

  Then, next, it is determined whether only the R image and the B image are to be combined, or in addition to these, the G image is also combined (step S91). Here, the phase difference image obtained by synthesizing only the R image and the B image has an advantage that the unfocused portion can be clearly confirmed, and the phase difference image obtained by synthesizing the R image, the B image, and the G image is obtained. There is an advantage that the overall color is the same as that of a normal color image and there is little uncomfortable feeling. Therefore, the user can select as desired whether or not to add the G image to the image composition by inputting from the operation unit 32. Therefore, it is determined here how the setting by the user is.

  When it is determined that the G image is to be combined, the split image generation unit 45 combines the R image, the B image, and the G image after demosaicing (step S92), and does not combine the G image. If it is determined, only the R image and B image after demosaicing are combined (step S93).

  Thereafter, the processing after step S80 is performed as described above.

  According to the third embodiment, substantially the same effect as in the first and second embodiments described above is obtained, and which subject is in focus and which subject is out of focus due to the color shift generated in the out-of-focus portion. It can be easily confirmed by visual inspection. Furthermore, the amount of deviation from the in-focus position can be grasped based on the degree of color deviation. In addition, the right / left position of the color misregistration (whether the R image or the B image is on the left side and the other is on the right side) grasps whether the target subject is on the near side or the far side. be able to.

In addition, since a process for creating a color stereoscopic image is not required, the image processing becomes simpler and the processing load can be reduced.
[Embodiment 4]

  48 to 50 show the fourth embodiment of the present invention, FIG. 48 is a block diagram showing the detailed configuration of the image processing unit and the system controller in the imaging apparatus, and FIG. 49 is displayed on the display unit of the imaging apparatus. FIG. 50 is a flowchart illustrating display processing in the imaging apparatus.

  In the fourth embodiment, the same parts as those in the first to third embodiments are denoted by the same reference numerals, description thereof is omitted, and only different points will be mainly described.

  As shown in FIG. 48, the image processing unit 25 of the present embodiment includes a monochrome split image generation unit 46. The stereo image generation unit 40 is not shown in the image processing unit 25, but may be provided or omitted.

  The monochrome split image generation unit 46 is a first image and a second image that are captured by the image sensor 22, corrected between colors by the intercolor correction unit 36, and demosaiced by the image processing unit 25. Are combined as a monochrome image to generate a monochrome split image. Specifically, the monochrome split image generation unit 46 regards each pixel value as a luminance value, combines the R and B images after demosaicing normalized by inter-color correction, and performs monochrome splitting. Generate an image.

  Accordingly, the phase difference image displayed on the screen 27a of the display unit 27 is a monochrome split image. In this monochrome split image, the subject OBJc at the in-focus position is not a monochrome double image (completely superimposed image mrb shown in FIG. 49), but the subject OBJf at the far side or near side from the in-focus position. , OBJn is a monochrome double image showing a split according to the amount of deviation from the in-focus position (as shown in FIG. 49, a monochrome double image of a monochrome image mr based on the R image and a monochrome image mb based on the B image). Image).

  Processing for displaying such an image will be described with reference to FIG.

  Assume that this processing is started and the processing up to step S74 is performed.

  Then, the pixel value of each pixel is regarded as a luminance value, the inter-color correction unit 36 normalizes the luminance value of the R image (step S101), and normalizes the luminance value of the B image (step S101). Step S102).

  Then, the monochrome split image generation unit 46 adds the normalized pixel value of the R image and the pixel value of the B image for each pixel position (or addition average), and performs monochrome image synthesis (step S103).

  Thereafter, the processing after step S80 is performed as described above.

  Note that if the pixel value of the G image is also used for the monochrome image composition, the luminance image is generated in the out-of-focus portion, and therefore the pixel of the G image is used in this embodiment so that the deviation is not unclear. No value is used. However, it may be used as necessary (for example, to obtain a natural monochrome image without a sense of incongruity).

According to the fourth embodiment, the same effects as those of the first to third embodiments described above can be obtained, and which subject is in focus and which subject is focused by the brightness double image generated in the out-of-focus portion. Whether or not it is out of focus can be easily confirmed visually. Furthermore, the amount of deviation from the in-focus position can be grasped based on the degree of deviation of the double image. And since the process which produces a color stereoscopic vision image becomes unnecessary similarly to Embodiment 3, image processing becomes simpler and it becomes possible to reduce a processing load. In addition, since it is a monochrome process, the process is lighter than in the third embodiment, and a display in a short time is possible.
[Embodiment 5]

  51 to 54 show Embodiment 5 of the present invention. FIG. 51 is a block diagram showing the detailed configuration of the image processing unit and the system controller in the imaging apparatus. FIG. 52 is a display on the display unit of the imaging apparatus. FIG. 53 is a flowchart showing display processing in the imaging apparatus, and FIG. 54 is a diagram showing an example of a color predetermined according to the magnitude of the phase difference amount.

  In the fifth embodiment, the same parts as those in the first to fourth embodiments are denoted by the same reference numerals, description thereof is omitted, and only different points will be mainly described.

  As shown in FIG. 51, the image processing unit 25 of this embodiment includes a pseudo color image generation unit 47. The stereo image generation unit 40 is not shown in the image processing unit 25, but may be provided or omitted.

  First, the distance calculation unit 39 calculates the phase difference amount between the first image and the second image, specifically, the phase difference amount between the R image and the B image with respect to the entire image. ing.

  Then, the pseudo color image generation unit 47 applies a predetermined color according to the magnitude of the phase difference amount as shown in FIG. 54 to the entire image for which the phase difference amount has been calculated, thereby producing a pseudo color image. Is generated.

  The specific color assignment shown in FIG. 54 shows an example in which each color takes an 8-bit gradation, that is, a value from 0 to 255.

  First, a phase difference of 0, that is, a focused subject is assigned a color of B: G: R = 0: 255: 0 and is displayed in green.

  For a subject that is closer to the in-focus position than the in-focus position, the B component value is first increased as the distance from the in-focus position decreases, and when the B component reaches 255, the G component starts decreasing. In the phase difference -MAXN corresponding to the close distance, the G component becomes 0, the color of B: G: R = 255: 0: 0 is assigned, and is displayed in blue.

  For a subject that is farther away than the in-focus position, the R component value is first increased as the distance from the in-focus position is further away, and when the R component reaches 255, the G component starts decreasing. Thus, the G component becomes 0 in the phase difference + MAXF corresponding to the infinity distance, and the color of B: G: R = 0: 0: 255 is assigned and displayed in red.

  By performing such color assignment, if it is green, it is at or near the in-focus position, and the higher the purity of the blue component, the closer to the in-focus position, and the higher the purity of the red component. It can be easily confirmed visually that it is farther from the in-focus position as it is included.

  The pseudo color image generated by the pseudo color image generation unit 47 is displayed on the screen 27 a of the display unit 27. That is, for example, as shown in FIG. 52, the subject OBJc at the in-focus position is represented as a green region Ag, and the subject OBJf at the far side from the in-focus position is a red component corresponding to the amount of deviation from the in-focus position. The subject OBJn that is closer to the in-focus position than the in-focus position is expressed as an area Ab that includes a blue component corresponding to the amount of deviation from the in-focus position.

  Processing for displaying such an image will be described with reference to FIG.

  Assume that this processing is started and the processing up to step S75 is performed.

  At this point, the distribution of the phase difference amount at each pixel position in the entire image is obtained.

  Therefore, the pseudo color image generation unit 47 performs a pseudo color image generation process for assigning colors as shown in FIG. 54 according to the phase difference amount of the pixel position (step S111). Thereby, for example, a pseudo color image as shown in FIG. 52 is generated.

  Thereafter, the processing after step S80 is performed as described above.

  In the above description, the G color is assigned to the in-focus position, the B color is assigned to the short distance side, and the R color is assigned to the long distance side. However, this is only an example, and other color combinations may be applied. The user may set it as desired.

  According to the fifth embodiment, the same effects as those of the first to fourth embodiments described above can be obtained, and by looking at the color, which subject is in focus and which subject is out of focus, It can be easily confirmed visually. Such pseudo color display has an advantage that it is easy to determine the in-focus portion instantaneously. Moreover, it is possible to easily determine whether it is the long distance side or the short distance side depending on whether it includes the R component or the B component. Further, the degree of separation from the in-focus position can be easily recognized depending on the purity of the R component or the B component. In addition, as in the third and fourth embodiments, the process of creating a color stereoscopic image is not necessary, so that the image processing becomes simpler and the processing load can be reduced.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, you may delete some components from all the components shown by embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined. Thus, it goes without saying that various modifications and applications are possible without departing from the spirit of the invention.

DESCRIPTION OF SYMBOLS 1 ... Lens unit 2 ... Body unit 9 ... Imaging optical system 10 ... Lens (a focus lens is included)
DESCRIPTION OF SYMBOLS 11 ... Aperture 12 ... Band-limiting filter 12b ... GB filter 12r ... RG filter 12g, 12g1, 12g2 ... G filter 12w ... W filter 14 ... Lens control part 15 ... Lens side communication connector 18 ... Color selective transmission element 18L ... First color Selective transmission element 18R ... Second color selective transmission element 21 ... Shutter 22 ... Imaging element (color imaging element)
DESCRIPTION OF SYMBOLS 23 ... Imaging circuit 24 ... Imaging drive part 25 ... Image processing part 26 ... Image memory 27 ... Display part 27a ... Screen 28 ... Interface (IF)
DESCRIPTION OF SYMBOLS 29 ... Recording medium 30, 30A ... System controller 31 ... Sensor part 32, 32A ... Operation part 33 ... Strobe control circuit 34 ... Strobe 35 ... Body side communication connector 36 ... Inter-color correction part 37 ... Color image generation part 38 ... Contrast AF Control unit 38a ... AF assist control unit 39 ... Distance calculation unit 40 ... Stereo image generation unit 41 ... Image processing device 42 ... Recording unit 45 ... Split image generation unit 46 ... Monochrome split image generation unit 47 ... Pseudo color image generation unit

Claims (7)

Receiving light of a plurality of wavelength bands, photoelectrically converting them, and generating a first image in the first band, a second image in the second band, and a third image in the third band An image sensor;
An imaging optical system that forms a subject image on the imaging device;
The first part of the photographing light beam that is disposed on the optical path of the photographing light beam that passes through the image pickup optical system and reaches the image pickup element and that attempts to pass through a first region that is a part of the pupil region of the image pickup optical system. A first band limitation that blocks light in one band and allows light in the second band and the third band to pass; and a second area that is another part of the pupil area of the imaging optical system. A band-limiting filter that performs a second band limitation that blocks light in the second band in the imaging light flux to be passed and allows the light in the first band and the third band to pass;
A display unit for displaying a phase difference image indicating a phase difference between the first image and the second image;
An imaging apparatus comprising: A distance calculation unit for calculating a phase difference amount between the first image and the second image;
One eye in which the center of gravity position of the second image and the third image is moved in the direction of the center position of the blur of the first image based on the phase difference amount calculated by the distance calculation unit. And a color image for another eye in which the position of the center of gravity of the blur of the first image and the third image is moved in the direction of the position of the center of gravity of the blur of the second image. A stereo image generator that generates a color stereoscopic image;
Further comprising
The imaging apparatus according to claim 1, wherein the display unit simultaneously displays the one-eye color image and the other one-eye color image as the phase difference image.   The stereo image generating unit generates a color stereoscopic image with enhanced stereoscopic effect based on a corrected phase difference amount obtained by multiplying the phase difference amount calculated by the distance calculating unit by a predetermined coefficient larger than 1. The imaging apparatus according to claim 2, wherein: A split image generating unit configured to generate a split image by combining the first image of the first band color and the second image of the second band color;
The imaging apparatus according to claim 1, wherein the display unit displays the split image as the phase difference image. A monochrome split image generation unit configured to combine the first image and the second image as a monochrome image to generate a monochrome split image;
The imaging apparatus according to claim 1, wherein the display unit displays the monochrome split image as the phase difference image. A distance calculation unit for calculating a phase difference amount between the first image and the second image;
A pseudo color image generation unit that generates a pseudo color image by applying a predetermined color according to the magnitude of the phase difference amount to the entire image in which the phase difference amount is calculated;
Further comprising
The imaging apparatus according to claim 1, wherein the display unit displays the pseudo color image as the phase difference image.   Of the first band and the second band, one band is a blue (B) band, the other band is a red (R) band, and the third band is green (G). The imaging apparatus according to claim 1, wherein the imaging device has a bandwidth of 1.

Images