JP5066851B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus, an image processing apparatus, and a pixel processing method that perform processing on a captured image.

  A pair of photographing lenses having a predetermined parallax are photographed at the same time, and one of the pair of images is obtained based on the distance distribution of the object scene obtained based on the pair of images. An image processing apparatus that performs image processing for adding blur is known (for example, see Patent Document 1).

Prior art documents related to the invention of this application include the following.
JP 09-181966 A

However, the above-described conventional image processing apparatus has the following problems.
Since the blur amount is proportional to the defocus amount (difference between the planned image plane and the actual image plane), it is necessary to calculate the defocus amount in the process of calculating the blur amount, but based only on the subject distance The defocus amount cannot be calculated directly. It is necessary to specify the distance (focus distance) of the main subject to be focused and calculate the defocus amount based on the difference between the focus distance and the subject distance. Therefore, a device for inputting the in-focus distance is required, and an error occurs in the defocus amount and the blur amount in accordance with the in-focus distance error.
Further, in the process of calculating the defocus amount based on the difference between the focus distance and the subject distance, optical parameters such as the focal length of the photographing lens are required. An input device for inputting an optical parameter is required, and the defocus amount and the blur amount also cause an error according to the error of the optical parameter.

An image pickup apparatus according to a first aspect of the present invention is an image pickup device that picks up a subject image by a subject light beam passing through an image pickup optical system, and a plurality of pixels each composed of a microlens and a photoelectric conversion unit are two-dimensionally arranged. Each of the microlenses divides the luminous flux of the subject into a pair of luminous fluxes that respectively pass through a pair of pupil regions of the pupil of the imaging optical system, and the plurality of photoelectric conversion units are connected to one of the pair of luminous fluxes. An imaging device including a plurality of first photoelectric conversion units that receive a plurality of light beams and a plurality of second photoelectric conversion units that receive the other light beam of the pair of light beams, and the plurality of first photoelectric conversions Image synthesizing means for generating a synthesized image by adding the first image data based on the output signal of the unit and the second image data based on the output signals of the plurality of second photoelectric conversion units; and the first image data And the second Based on the image data, a focus detection unit that calculates a defocus amount in each of a plurality of small areas in the imaging screen, and the plurality of small regions according to the magnitude of the defocus amount calculated by the focus detection unit Classification means for classifying an area into one of a plurality of defocus attributes, and small areas that cannot be calculated by the focus detection means and that cannot be calculated, are located around the small areas that cannot be calculated. Defocus attribute setting means for setting a defocus attribute of the small area that cannot be calculated based on the defocus amount or image data in the area, the defocus attribute classified by the classification means, and the defocus attribute setting means The composite image corresponding to each of the small areas divided into a plurality of groups based on the defocused attribute In part, characterized in that it comprises, image processing means for performing image processing corresponding to the defocus attribute of the group.
An image pickup apparatus according to a second aspect of the present invention is an image pickup device that picks up a subject image by a subject light flux passing through an image pickup optical system, and a plurality of pixels each composed of a microlens and a photoelectric conversion unit are two-dimensionally arranged. Each of the microlenses divides the luminous flux of the subject into a pair of luminous fluxes that respectively pass through a pair of pupil regions of the pupil of the imaging optical system, and the plurality of photoelectric conversion units are connected to one of the pair of luminous fluxes. An imaging device including a plurality of first photoelectric conversion units that receive a plurality of light beams and a plurality of second photoelectric conversion units that receive the other light beam of the pair of light beams, and the plurality of first photoelectric conversions Image synthesizing means for generating a synthesized image by adding the first image data based on the output signal of the unit and the second image data based on the output signals of the plurality of second photoelectric conversion units; and the first image data And the second Based on the image data, a focus detection unit that calculates a defocus amount in each of a plurality of small areas in the imaging screen, and the plurality of small regions according to the magnitude of the defocus amount calculated by the focus detection unit Classification means for classifying an area into one of a plurality of defocus attributes, and small areas that cannot be calculated by the focus detection means and that cannot be calculated, are located around the small areas that cannot be calculated. Defocus attribute setting means for setting a defocus attribute of the small area that cannot be calculated based on the defocus amount or image data in the area, the defocus attribute classified by the classification means, and the defocus attribute setting means The composite image corresponding to each of the small areas divided into a plurality of groups based on the defocused attribute An image processing unit that performs image processing according to a defocus attribute of the group, a display unit that displays the composite image processed by the image processing unit as a through image, and a release operation that starts a shooting operation A first image data based on an output signal output from the plurality of first photoelectric conversion units and an output signal output from the plurality of second photoelectric conversion units according to an operation of the member, the release operation member Recording means for recording the second image data .

  According to the present invention, it is possible to add blur to an obtained image with high accuracy with a simple configuration.

  An embodiment in which an imaging apparatus including an image processing apparatus according to the present invention is applied to a digital still camera will be described. FIG. 1 shows the configuration of an embodiment. A digital still camera 201 according to an embodiment includes an interchangeable lens 202 and a camera body 203, and the interchangeable lens 202 is attached to the camera body 203 by a mount unit 204.

  The interchangeable lens 202 includes a lens drive control device 206, a diaphragm 207, a zooming lens 208, a lens 209, a focusing lens 210, and the like. The lens drive control device 206 includes peripheral components such as a microcomputer and a memory. The lens drive control device 206 controls the driving of the focusing lens 210 and the diaphragm 207, detects the states of the diaphragm 207, the zooming lens 208 and the focusing lens 210, and body drive control described later. Transmission of lens information to the device 214 and reception of camera information are performed.

  The camera body 203 includes an imaging element 211, a body drive control device 214, a liquid crystal display element drive circuit 215, a liquid crystal display element 216, an eyepiece lens 217, a memory card 219, and the like. Pixels, which will be described later, are arranged in a two-dimensional manner on the imaging element 211, and are arranged on the planned imaging plane of the interchangeable lens 202 to capture a subject image formed by the interchangeable lens 202. The body drive control device 214 includes peripheral components such as a microcomputer and a memory, reads out an image signal from the image sensor 211, corrects the image signal, detects a focus adjustment state of the interchangeable lens 202, and a lens from the lens drive control device 206. Receives information, transmits camera information (defocus amount), and controls the operation of the entire digital still camera. The body drive control device 214 and the lens drive control device 206 communicate via the electrical contact portion 213 of the mount portion 204.

  The liquid crystal display element driving circuit 215 drives a liquid crystal display element 216 of a liquid crystal viewfinder (EVF: electrical viewfinder). The photographer can observe an image displayed on the liquid crystal display element 216 via the eyepiece lens 217. The memory card 219 is removable from the camera body 203 and is a portable storage medium that stores and stores image signals.

  The subject image formed on the image sensor 211 after passing through the interchangeable lens 202 is photoelectrically converted by the image sensor 211, and its output is sent to the body drive controller 214. The body drive control device 214 generates defocus distribution information on the screen based on the output of the image sensor 211, and adds image processing to the image sensor output according to the defocus distribution information. Further, the body drive control device 214 calculates a defocus amount at a predetermined focus detection position based on the pixel output, and sends this defocus amount to the lens drive control device 206. Further, the body drive control device 214 stores an image signal generated based on the output of the image sensor 211 in the memory card 219 and sends the image signal to the liquid crystal display element drive circuit 215 to display an image on the liquid crystal display element 216. Let The photographer can observe the image displayed on the liquid crystal display element 216 through the eyepiece lens 217.

  The camera body 203 is provided with unshown operation members (shutter button, focus detection position setting member, image processing adjustment parameter input member, etc.), and the body drive control device 214 receives operation state signals from these operation members. , And controls the operation (imaging operation, focus detection position setting operation, image processing operation) according to the detection result.

  The lens drive control device 206 changes the lens information according to the focusing state, zooming state, aperture setting state, aperture opening F value, and the like. Specifically, the lens drive control device 206 monitors the positions of the lenses 208 and 210 and the diaphragm position of the diaphragm 207, calculates lens information according to the monitor information, or monitors from a lookup table prepared in advance. Select lens information according to the information. The lens drive control device 206 calculates a lens drive amount based on the received defocus amount, and drives the focusing lens 210 to a focal point by a drive source such as a motor (not shown) based on the lens drive amount.

  FIG. 2 is an enlarged front view showing the pixel arrangement configuration of the image sensor 211. In the imaging element 211, pixels 311 and pixels 312 are regularly arranged in a two-dimensional manner.

  FIG. 3 shows the configuration of the pixel. As illustrated in FIG. 3A, the pixel 312 includes the microlens 10 and the photoelectric conversion unit 13. The photoelectric conversion unit 13 has a rectangular shape in which the right side is substantially in contact with the perpendicular bisector of the microlens 10. As shown in FIG. 3B, the pixel 311 includes a microlens 10 and a photoelectric conversion unit 12. The photoelectric conversion unit 12 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The photoelectric conversion units 12 and 13 are arranged in the horizontal direction when the microlenses 10 are displayed in an overlapped manner, and have a symmetrical shape with respect to the vertical bisector of the microlens 10. In the image sensor 211, pixel rows in which pixels 311 and 312 are alternately arranged in the horizontal direction (alignment direction of the photoelectric conversion units 12 and 13) are arranged by shifting one pixel every other row in the vertical direction.

  FIG. 4 is a cross-sectional view of a pixel. (a) shows a cross section of the pixel 311. In the pixel 311, the microlens 10 is disposed in front of the photoelectric conversion unit 12, and the photoelectric conversion unit 12 is projected forward by the microlens 10. The photoelectric conversion unit 12 is formed on the semiconductor circuit substrate 29, and the microlens 10 is integrally and fixedly formed thereon by a semiconductor image sensor manufacturing process. The photoelectric conversion unit 12 is disposed on one side of the optical axis of the microlens 10. (b) shows a cross section of the pixel 312. In the pixel 312, the microlens 10 is disposed in front of the photoelectric conversion unit 13, and the photoelectric conversion unit 13 is projected forward by the microlens 10. The photoelectric conversion unit 13 is formed on the semiconductor circuit substrate 29, and the microlens 10 is integrally and fixedly formed thereon by the manufacturing process of the semiconductor image sensor. The photoelectric conversion unit 13 is disposed on one side of the optical axis of the microlens 10 and on the opposite side of the photoelectric conversion unit 12.

  Next, the principle of focus detection using a pupil division type phase difference detection method using a microlens will be described with reference to FIG. In FIG. 5, reference numeral 90 denotes an exit pupil set at a distance d4 in front of the microlens arranged on the planned imaging plane of the interchangeable lens. Note that the distance d4 is a distance determined according to the curvature, refractive index, distance between the microlens and the photoelectric conversion unit, and the like, and is hereinafter referred to as a distance measurement pupil distance. 91 is an optical axis of the interchangeable lens, 10a to 10c are microlenses, 12a, 12b, 13a and 13b are photoelectric conversion units, 311a, 311b, 312a and 312b are pixels, and 72, 73, 82 and 83 are light beams.

  Reference numeral 92 denotes regions of the photoelectric conversion units 12a and 12b projected by the microlenses 10a and 10c. In this specification, the region 92 is called a distance measuring pupil. In FIG. 5, an elliptical region is used for ease of understanding, but the shape of the photoelectric conversion units 12 a and 12 b is actually an enlarged projection shape. Similarly, 93 is a region of the photoelectric conversion units 13a and 13b projected by the microlenses 10b and 10d, that is, a distance measuring pupil. In FIG. 5, an elliptical region is used for easy understanding, but the shape of the photoelectric conversion units 13 a and 13 b is actually an enlarged projection shape.

  FIG. 5 schematically illustrates four adjacent pixels 311a, 311b, 312a, and 312b. Similarly, in other pixels, the photoelectric conversion unit arrives at each microlens from the corresponding distance measurement pupil. Receives light flux.

  The microlenses 10a to 10c are disposed in the vicinity of the planned imaging plane of the interchangeable lens, and the shapes of the photoelectric conversion units 12a, 12b, 13a, and 13b disposed behind the microlenses 10a to 10c are the microlenses 10a to 10c. Is projected onto an exit pupil 90 separated by a distance measurement pupil distance d4, and the projection shape forms distance measurement pupils 92 and 93. That is, the projection direction of the photoelectric conversion unit in each pixel is determined so that the projection shape (ranging pupils 92 and 93) of the photoelectric conversion unit of each pixel matches on the exit pupil 90 at the projection distance d4.

  The photoelectric conversion unit 12a passes through the distance measuring pupil 92 and outputs a signal corresponding to the intensity of the image formed on the microlens 10a by the light beam 72 directed to the microlens 10a. The photoelectric conversion unit 12b outputs a signal corresponding to the intensity of the image formed on the microlens 10c by the light beam 82 passing through the distance measuring pupil 92 and directed to the microlens 10c. Further, the photoelectric conversion unit 13a outputs a signal corresponding to the intensity of the image formed on the microlens 10b by the light flux 73 passing through the distance measuring pupil 93 and directed to the microlens 10b. In addition, the photoelectric conversion unit 13b outputs a signal corresponding to the intensity of the image formed on the microlens 10d by the light beam 83 passing through the distance measuring pupil 92 and directed to the microlens 10d.

  A large number of these two types of pixels are arranged in a straight line, and the output of the photoelectric conversion unit of each pixel is grouped into an output group corresponding to the distance measuring pupil 92 and the distance measuring pupil 93, so that the distance measuring pupil 92 and the distance measuring Information on the intensity distribution of a pair of images formed on the pixel array by the focus detection light fluxes that pass through the pupils 93 is obtained. By applying an image shift detection calculation process (correlation calculation process, phase difference detection process), which will be described later, to this information, an image shift amount of a pair of images is detected by a so-called pupil division type phase difference detection method. Then, by multiplying this image shift amount by a predetermined conversion coefficient, the current imaging plane with respect to the planned imaging plane (the imaging plane at the focus detection position corresponding to the position of the microlens array on the planned imaging plane). A deviation (defocus amount) is calculated.

  FIG. 6 is a diagram showing a projection relationship on the exit pupil plane. The distance measuring pupils 92 and 93 obtained by projecting the photoelectric conversion unit from each pixel onto the exit pupil plane 90 by the microlens are set to a size including the exit pupil 89 of the optical system. Therefore, the light beam actually received by the photoelectric conversion unit of the pixel is a light beam that passes through a region in which the distance measuring pupils 92 and 93 are narrowed by the exit pupil 89 of the optical system. In this way, by setting the distance measuring pupils 92 and 93 in a region larger than the exit pupil 89 of the optical system, the photoelectric conversion unit receives a light amount corresponding to the control amount of the aperture when the aperture of the optical system is controlled. This makes it possible to use a pixel output as image data.

  The image formation state in the pupil division type phase difference detection method will be described with reference to FIG. As shown in FIG. 7A, the light beam which is divided into distance measuring pupils 92 and 93 on the exit pupil plane 90 of the optical system and forms an image is divided into a light beam 72 passing through the distance measuring pupil 92 and a distance measuring pupil 93. It is divided into a luminous flux 73 that passes through. With such a configuration, for example, when a line pattern (white line on a black background) on the optical axis 91 and perpendicular to the paper surface of FIG. 7 is imaged by the optical system, the distance measuring pupil 92 is formed on the focusing plane P0. The light beam 72 passing through and the light beam 73 passing through the distance measuring pupil 93 form a high-contrast line image pattern at the same position on the optical axis 91 as shown in FIG.

  On the other hand, on the plane P1 ahead of the focusing plane P0, the light beam 72 passing through the distance measuring pupil 92 and the light beam 73 passing through the distance measuring pupil 93 are lines blurred at different positions as shown in FIG. 7B. An image pattern is formed. On the surface P2 behind the focusing plane P0, the light beam 72 passing through the distance measuring pupil 92 and the light beam 73 passing through the distance measuring pupil 93 are as shown in FIG. 7 (b). A blurred line image pattern is formed at a different position in the opposite direction. Therefore, two images formed by the light beam 72 passing through the distance measuring pupil 92 and the light beam 73 passing through the distance measuring pupil 93 are detected separately, and the relative positional relationship (image shift amount) between the two images is detected. By calculating the focus adjustment state of the optical system on the surface where the two images are detected can be detected.

  FIG. 8 is a flowchart illustrating the operation of the digital still camera (imaging device) according to the embodiment. The body drive control device 214 repeatedly executes this operation when the camera is powered on. When the power is turned on in step 100, the process proceeds to step 110, a pair of image data (data of two types of pixels) is read from the image sensor 211, and a pair of image data is added to generate basic image data. In step 120, image shift detection calculation processing (correlation calculation processing) described later is performed for each minute block obtained by dividing the screen into minute regions, the image shift amount is calculated, and the defocus amount is calculated to generate defocus distribution information. To do.

  In step 130, image processing for blur control is performed on the basic image data based on the defocus distribution information. Details of this process will be described later. In subsequent step 140, the basic image data subjected to the image processing is displayed on the electronic viewfinder. In step 150, it is checked whether or not the defocus amount at the focus detection position is close to focusing, that is, whether or not the absolute value of the calculated defocus amount is within a predetermined value. The focus detection position is designated by the photographer using a position setting operation member (not shown). If it is determined that the lens is not in focus, the process proceeds to step 160 where the defocus amount is transmitted to the lens drive control unit 206 to drive the focusing lens 210 of the interchangeable lens 202 to the in-focus position, and the process returns to step 110 to perform the above-described operation. repeat. Even when focus detection is impossible, the process branches to step 160, a scan drive command is transmitted to the lens drive control device 206, and the focusing lens 210 of the interchangeable lens 202 is scanned from infinity to the nearest position, and the process returns to step 110. The above operation is repeated.

  If it is determined that it is close to the in-focus state, the process proceeds to step 170 to determine whether or not a shutter release has been performed by operating an operation member (not shown). If it is determined that the shutter release has not been performed, the process returns to step 110 and described above. Repeat the operation. On the other hand, if it is determined that the shutter release has been performed, the process proceeds to step 180 where the image pickup device 211 performs an image pickup operation, reads a pair of image data (data of two types of pixels) from the image pickup device 211, and a pair of image data. Are stored in the memory card 219 independently.

The details of the defocus amount calculation process executed in step 120 of FIG. 8 will be described with reference to FIG. When a pair of pixel data in a minute block is expressed as (E (1) to E (m)), (F (1) to F (m), where m is the number of data), the data series (E ( 1) to E (m)), the correlation amount C (k) in the shift amount k between the two data strings is calculated by Equation 1 while relatively shifting the data series (F (1) to F (m)). .
C (k) = Σ | E (n) −F (n + k) | (1)
In the equation (1), the Σ operation is calculated for n. In this Σ operation, the range of n and n + k is limited to a range of 1 to m. The shift amount k is an integer, and is a relative shift amount in units of the detection pitch of a pair of data.

As shown in FIG. 9A, the calculation result of the expression (1) shows that the correlation amount C (k) is the shift amount with high correlation between the pair of data series (k = kj = 2 in FIG. 9A). The minimum (the smaller the value, the higher the degree of correlation). Subsequently, the shift amount x that gives the minimum value C (x) with respect to the continuous correlation amount is obtained by using a three-point interpolation method according to the following equations (2) to (5).
x = kj + D / SLOP (2),
C (x) = C (kj) − | D | (3),
D = {C (kj-1) -C (kj + 1)} / 2 (4),
SLOP = MAX {C (kj + 1) -C (kj), C (kj-1) -C (kj)} (5)

The defocus amount DEF of the subject image plane with respect to the planned imaging plane can be obtained from the following equation (6) from the shift amount x obtained from the equation (2).
DEF = KX · PY · x (6)
In equation (6), PY is a detection pitch (arrangement pitch of the same type of pixels), and KX is a conversion coefficient determined by the size of the opening angle of the center of gravity of the light beam passing through the pair of distance measuring pupils. In addition, when the pair of data series exactly match (x = 0), the data string is actually shifted by half of the detected pitch, so the shift amount obtained by equation (2) is only half the pitch. It is corrected and applied to equation (6). Furthermore, since the size of the opening angle of the center of gravity of the light beam passing through the pair of distance measuring pupils changes according to the size of the aperture of the interchangeable lens 202 (open aperture value), it is determined according to the lens information. .

  Whether or not the calculated defocus amount DEF is reliable is determined as follows. As shown in FIG. 9B, when the degree of correlation between the pair of data series is low, the value of the minimum value C (x) of the interpolated correlation amount becomes large. Therefore, when C (x) is equal to or greater than a predetermined value, it is determined that the reliability is low. Alternatively, in order to normalize C (x) with the contrast of data, if the value obtained by dividing C (x) by SLOP that is proportional to the contrast is equal to or greater than a predetermined value, it is determined that the reliability is low. Alternatively, if SLOP that is proportional to the contrast is equal to or less than a predetermined value, it is determined that the subject has low contrast and the reliability of the calculated defocus amount DEF is low.

  As shown in FIG. 9C, when the correlation between the pair of data series is low and there is no drop in the correlation amount C (k) between the predetermined shift ranges kmin to kmax, the minimum value C (x) is set. In such a case, it is determined that the focus cannot be detected. When focus detection is possible, the defocus amount is calculated by multiplying the calculated image shift amount by a predetermined conversion coefficient.

  In the above description, the defocus amount obtained for a pair of data series is adopted, but the defocus amount obtained for a plurality of pairs of data series in the same minute block is subjected to statistical processing (for example, average or median). The final defocus amount may be obtained.

FIG. 10 is a block diagram showing an overview of image processing in steps 110 to 130 of FIG. The right ranging pupil image data R (x, y) and the left ranging pupil image data L (x, y) are added to calculate basic image data g (x, y). Here, (x, y) represents a two-dimensional discrete position of the pixel, and in the image sensor 211 shown in FIG. 2, x corresponds to the horizontal direction and y corresponds to the vertical direction. However, the right ranging pupil image data R (x, y) exists only when x is an odd number, and the left ranging pupil image data L (x, y) exists only when x is an even number. .
g (x, y) = R (x, y) + (L (x-1, y) + L (x + 1, y)) / 2, If x: odd number,
g (x, y) = L (x, y) + (R (x-1, y) + R (x + 1, y)) / 2, If x: even number
... (7)
Defocus distribution information data d (p, q) is calculated for each small block divided as described in FIG. Here, (p, q) represents a two-dimensional discrete position of the minute block.

The basic image data g (x, y) in the minute block is subjected to image processing according to the defocus amount of the minute block and the image processing control parameter designated from the outside, and the basic image data G (x , Y).
G (x, y) = F (g (x, y), d, z) (8)
The function F is a function for applying image processing to image data in accordance with the defocus amount d and the parameter z.

  As described above, since the image processing is performed on the basic image data obtained by adding the right ranging pupil image data and the left ranging pupil image data, the barycentric position of the image may be moved according to the focus adjustment. In addition, even when image-processed basic image data is displayed on the electronic viewfinder, it can be observed naturally without a sense of incongruity. When image processing is applied to only one of the image data, the center of gravity of the image moves according to the focus adjustment, so that an uncomfortable feeling occurs when displayed with an electronic viewfinder, and unnaturalness is felt. (See FIG. 7).

  In addition, when image processing is applied to only one of the image data, the original image data corresponds to an image formed by a light flux that passes through the distance measuring pupil on one side of the pair of distance measuring pupils. The bokeh reflects the shape of the distance pupil (it becomes semicircular because it is limited by the aperture), and even when the image data after adding blur is displayed on the electronic viewfinder, a sense of incongruity occurs and the unnaturalness I feel it.

  FIG. 11 is a flowchart showing details of the image processing in steps 120 to 130 of FIG. In step 200, the screen is divided into minute blocks 101 as shown in FIG. 12, and the defocus amount is calculated for each minute block. As a result, two-dimensional distribution information of the defocus amount d (defocus distribution information d (p, q)) as shown in FIG. 13 is obtained over the entire screen. In step 210, a histogram of the defocus amount d is calculated based on the defocus distribution information d (p, q) (see FIG. 14).

  In step 220, the defocus amount is classified into a plurality of ranges based on the defocus amount with a low frequency based on the defocus amount d histogram. Here, basic image data as shown in FIG. 15 will be described as an example. In FIG. 15, the near view is a person who is a main subject, the middle view is a building, and the distant view is a sky and clouds. When a histogram of the defocus amount d is taken with respect to this basic image data, as shown in FIG. 14, it is roughly classified into a portion 113 corresponding to the near view, a portion 114 corresponding to the middle view, and a portion 115 corresponding to the distant view. Can do. A threshold defocus amount d1 is set between the portions 113 and 114 (valley), a threshold defocus amount d2 is set between the portions 114 and 115, and the defocus range is set to L1 (+ ∞> d ≧ d1), Dividing into three ranges of L2 (d1> d ≧ d2) and L3 (d2> d> −∞). Defocus amounts representative of the defocus ranges L1, L2, and L3 are DL1, DL2, and DL3.

  In step 230, depending on which defocus range the defocus amount d of the micro block belongs to, the micro block has four defocus attributes (H0: focus cannot be detected, H1: defocus range L1, H2: defocus range L2, H3). : Defocus range L3) (see FIG. 16). In step 240, defocus characteristics H 1, H 2, and H 3 are determined based on image characteristics and similarities with surrounding micro blocks for a micro block (attribute H 0) incapable of focus detection due to low contrast or the like.

  For example, when a skin color region (130 in FIG. 17) is cut out from the basic image data and the defocus attributes of the minute blocks included in this region are concentrated on one attribute H1, focus detection that belongs to this region is impossible. Let H1 be the attribute of a small block. Alternatively, when the defocus attributes of the minute blocks belonging to the outer contour portion of this region are concentrated on one attribute H1, the attribute of the minute block that cannot be detected in the focus belonging to this region is set to H1. For example, based on image data obtained by performing contour extraction image processing on basic image data (see FIG. 18), when the focus detection is impossible for a minute block including the contour 131, the contour is continuous. The defocus attribute of the minute block capable of detecting the focus in the surrounding direction is adopted as the attribute of the minute block incapable of detecting the focus.

  The method for determining the defocus attribute of a minute block incapable of focus detection is not limited to the above method, and is determined based on the similarity and continuity of image characteristics of a minute block incapable of focus detection and surrounding minute blocks in which focus detection is possible. can do. As image characteristics, luminance information, hue information, saturation information, contrast information, and the like can be used.

  In step 250, the minute blocks are grouped based on the defocus attribute of the minute block. Here, as shown in FIG. 19, it is classified into group J1, group J2, and group J3. In subsequent step 260, the minute blocks belonging to the same group are collected and extracted for each group. FIG. 20 shows an example of extracting the group J1, FIG. 21 shows the group J2, and FIG. 22 shows the group J3. Next, the basic image data portion corresponding to the extracted area is cut out. 23A shows the basic image data portion corresponding to the group J1, FIG. 24A shows the basic image data portion corresponding to the group J2, and FIG. 25A shows the basic image data portion corresponding to the group J3. Each data part is shown.

  In step 270, image processing corresponding to the defocus attribute and the designated image processing control parameter is performed on the cut out partial image data, and an image layer is generated for each group. FIG. 23B shows an image layer corresponding to the group J1, FIG. 24B shows an image layer corresponding to the group J2, and FIG. 25B shows an image layer corresponding to the group J3. In step 280, the image layers are superimposed in the order corresponding to the distance, and final image-processed basic image data is generated. Here, the distant view is overlaid and the close-up view is overlaid in order. FIG. 26 shows an image at a stage where a middle scene is overlaid on a distant view, and FIG. 27 shows an image at a stage where a near scene is superimposed on FIG.

Next, the image processing in step 270 will be described in detail. Image processing is performed on the partial image W (x, y) based on the defocus attribute of the partial image and the designated image processing control parameter. Here, blur addition processing is performed as an example of image processing. The partial image W (x, y) is subjected to a convolution operation (filtering operation) with a point spread function (point spread function PSF (x, y)) by blur to generate a processed partial image V (x, y). To do.
V (x, y) = W (x, y) PSF (x, y) (9)
In equation (9), ◇ represents a two-dimensional convolution operation. Note that ∫∫PSF (x, y) dxdy = 1.

For example, the point spread function PSF (x, y) is as follows.
PSF (x, y) = 1 / (π * r2) if x2 + y2 ≦ r2,
PSF (x, y) = 0 if x2 + y2> r2 (10)
In the equation (10), r is the radius of the point image blur. However, when r = 0, PSF (x, y) = δ (x, y).
r = Zf · | dL | (11)
In the equation (11), Zf is an image processing control parameter, which is a parameter corresponding to the aperture F value that is manually input by operating an operation member (not shown) or automatically input according to exposure control. is there. DL is a defocus amount corresponding to the defocus attribute of the partial image. For example, for the defocus attribute H1 (defocus range L1), the defocus amount dL1, for the defocus attribute H2 (defocus range L2), the defocus amount dL2, and for the defocus attribute H3 (defocus range L3), the defocus amount. dL3 (see FIG. 14).

  According to equation (11), the value of the radius r changes according to the defocus amount dL and the input value Zf, so the amount of image processing applied to the partial image also changes. For example, if the defocus amount dL1 = 0 in FIG. 14, the basic image data (a) before image processing and the basic image data (b) after image processing do not change as shown in FIG. 23, but the defocus amount dL The larger the is, the larger the blur amount of the basic image data after image processing is. Further, as the value of the parameter Zf increases, the amount of blur of the basic image data after image processing increases.

  The operation member for inputting the image processing control parameter Zf can be used by expanding the operation member that is normally used for setting the aperture F value at the time of exposure. For example, according to the operation of the aperture ring for the aperture setting provided on the interchangeable lens, the viewfinder display and imaging are performed by controlling the aperture to the limit that the aperture aperture can accommodate, and the aperture aperture limit In the case of an aperture setting exceeding 1, an image obtained by image processing can be used for viewfinder display and imaging. Of course, a parameter input operation member dedicated to image processing may be provided.

  The point spread function PSF (x, y) is not limited to the step function as shown in the equation (11), and the shape of the function may change smoothly. Bokeh can be controlled by adjusting the shape and size of the function.

  In the image processing described above, individual image processing is applied to the partial images, and the final image processed image is generated by synthesizing the partial images after the image processing. Depending on the histogram of defocus amount, the partial image is cut out so that the subject whose relative distance is close enters the same partial image, so even when combining the partial images after image processing, A natural composite image without a sense of incongruity can be obtained. Further, when combining the partial images after the image processing, image processing such as averaging processing or contour enhancement processing may be performed so that the vicinity of the joints (near the boundary) of the partial images becomes more natural.

Further, the expression relating the defocus amount and the blur amount is not limited to the expression (11), and the following expression in which a focus adjustment parameter Off for the focus position offset is added may be used.
r = Zf · | dL−Off | (12)
By adjusting the focus adjustment parameter Off for the focus position offset manually or automatically, it is also possible to generate image-processed basic image data in which the focus position is adjusted and changed with respect to the basic image data.

  The generation of the basic image data is not limited to the simple addition average as shown in the expression (7), and weighted addition average and other synthesis calculation processes are performed as long as the center of gravity position of the image does not change according to the defocus amount. Can be used.

  It is not necessary to determine the defocus amount based on the relationship with the surrounding minute blocks for all the minute blocks where the focus cannot be detected. For example, an image with low contrast has little change even when image processing is applied. Therefore, a fine block whose contrast is equal to or lower than a predetermined threshold value and cannot be detected is used as it is without being subjected to image processing. May be.

  The image processing is not limited to the blur addition processing as expressed by the equation (9), and for example, an image that recovers the blurred image data by performing a sharpening process (high-pass filter process) using a parameter corresponding to the defocus amount dL. Processing may be performed. If a sharpening process according to the defocus amount becomes possible after such shooting, a pan-focus image can be easily obtained. Further, even when it is necessary to shoot with a large aperture opening due to insufficient light quantity, image data having a deep depth of field can be obtained from the captured image data.

  As image processing, not only blurring and sharpening but also blurring (a blurred shape reflecting the aperture opening shape) can be employed. For example, image data shot with a normal aperture shape (circular) is subjected to image processing that causes ring-shaped blurring when shooting with a reflective lens, or image data shot with a polygonal aperture. Can be processed into image data photographed with a circular aperture.

  It is also possible to apply image processing only to an image in a specific defocus range. For example, when the front blur is dirty, it is possible to obtain an image with a good front blur by applying blur shaping image processing only to the front blur (defocus amount +) image.

  Furthermore, it is possible to perform aberration correction processing according to the defocus amount. For example, even when an unsightly blur such as astigmatism or coma depending on the defocus amount occurs, it can be corrected to a preferable blur by applying the image processing of the present invention.

  As described above, the image processing is applied to the partial image according to the defocus amount, so that the effect according to the defocus amount is added without depending on physical restrictions (size or shape of the aperture opening). Images can be obtained. In addition, for example, physical control (aperture aperture size, shape, etc.) other than rendering controls (blur adjustment, depth of field adjustment, etc.) according to the defocus amount, shooting controls (total light intensity adjustment, peripheral light intensity adjustment) Even if it has to be used for aberration adjustment, resolution / contrast adjustment, flare / stray light adjustment, etc., it is possible to add depiction control according to the defocus amount by post-processing.

<< Modification of Embodiment of Invention >>
FIG. 28 shows a modification of the pixel array of the image sensor 211. The pixel 311 includes the microlens 10 and the photoelectric conversion unit 12. The photoelectric conversion unit 12 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The pixel 312 includes the microlens 10 and the photoelectric conversion unit 13. The photoelectric conversion unit 13 has a rectangular shape in which the right side is substantially in contact with the perpendicular bisector of the microlens 10. The photoelectric conversion units 12 and 13 are aligned in the horizontal direction when the microlenses 10 are displayed in a superimposed manner, and have a symmetrical shape with respect to the vertical bisector of the microlens 10.

  The pixel 316 includes the microlens 10 and the photoelectric conversion unit 16. The photoelectric conversion unit 16 has a rectangular shape in which the upper side is substantially in contact with the horizontal bisector of the microlens 10. The pixel 317 includes the microlens 10 and the photoelectric conversion unit 17. The photoelectric conversion unit 17 has a rectangular shape in which the lower side is substantially in contact with the horizontal bisector of the microlens 10. The photoelectric conversion units 16 and 17 are arranged vertically in the vertical direction when the microlenses 10 are superimposed and displayed, and have a symmetrical shape with respect to the horizontal bisector of the microlens 10.

  FIG. 29 shows distance measuring pupils 96 and 97 corresponding to the photoelectric conversion units 16 and 17. The pixel 316 receives the light beam coming from the distance measuring pupil 96, and the pixel 317 receives the light beam coming from the distance measuring pupil 97. The distance measuring pupil 96 and the distance measuring pupil 97 are arranged line-symmetrically with respect to the axis x.

  In the image sensor 211 shown in FIG. 28, the second pixel in which the pixels 316 and 317 are alternately arranged in the horizontal direction in the row next to the first pixel row in which the pixels 311 and 312 are alternately arranged in the horizontal direction. The pixel row is arranged, and the next row is arranged with a third row obtained by shifting the first pixel row in the horizontal direction by one pixel, and the next row is arranged with the second pixel row in the horizontal direction by one pixel. A fourth row shifted to is arranged. The above four rows of arrays are sequentially arranged in the vertical direction.

  As described above, in a square pixel array in which pixels are densely arranged in the horizontal direction and the vertical direction, image shift detection can be performed in the horizontal direction and the vertical direction. Therefore, in calculating the defocus amount of a minute block, the probability that focus detection becomes impossible due to the directionality of the subject pattern is reduced.

  FIG. 30 shows a pixel configuration of a modified example. As illustrated in FIG. 30A, the pixel 602 includes a microlens 10 and a photoelectric conversion unit 513. The photoelectric conversion unit 513 has a semicircular shape in which the right side is substantially in contact with the vertical bisector of the microlens 10. In addition, as illustrated in FIG. 30B, the pixel 601 includes a microlens 10 and a photoelectric conversion unit 512. The photoelectric conversion unit 512 has a semicircular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. By setting such a photoelectric conversion portion shape, the limited surface of the semiconductor substrate can be efficiently used for the maximum aperture (circular shape) of the exchange lens.

  FIG. 31 shows a pixel configuration of a modified example. The pixel 603 includes the microlens 10 and a pair of photoelectric conversion units 612 and 613. The photoelectric conversion unit 612 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The photoelectric conversion unit 613 has a rectangular shape in which the right side is substantially in contact with the vertical bisector of the microlens 10.

  32 is a cross-sectional view of the pixel 603 shown in FIG. In the pixel 603, the microlens 10 is disposed in front of the photoelectric conversion units 612 and 613, and the photoelectric conversion units 612 and 613 are projected forward by the microlens 10. The photoelectric conversion units 612 and 613 are formed on the semiconductor circuit substrate 29, and the microlens 10 is integrally and fixedly formed thereon by the manufacturing process of the semiconductor image sensor. The photoelectric conversion units 612 and 613 are arranged so as to be distributed across the optical axis of the microlens 10.

FIG. 33 shows a pixel arrangement configuration of the image sensor 211 of a modified example in which the pixels shown in FIG. 31 are developed two-dimensionally. In the image sensor 211, the pixels 603 are regularly arranged in a two-dimensional manner. The configuration shown in FIGS. 31 to 33 makes it possible to detect an image shift in the direction in which the pair of photoelectric conversion units 612 and 613 are aligned (the horizontal direction in FIG. 33). Since the image detection pitch is halved compared to the image sensor of FIG. 2, the defocus detection accuracy is improved. In the pixel arrangement shown in FIG. 33, the right ranging pupil image data R (x, y) and the left ranging pupil image data L (x, y) exist in the same pixel, so the basic image data g (X, y) is calculated by the following equation (13) instead of the equation (7).
g (x, y) = R (x, y) + L (x, y) (13)
Since the basic image data is generated by synthesizing the right ranging pupil image data and the left ranging pupil image data in the same pixel, the image quality is improved.

FIG. 34 shows a pixel arrangement configuration of the image sensor 211 of a modification. The spectral characteristic sensitivities of the photoelectric conversion units of the image sensor need not all be the same. In the image sensor 211 shown in FIG. 34, the pixels 421, 422, and 423 are regularly arranged in a two-dimensional manner. 34, the photoelectric conversion units 51 and 71 marked G have the green (G) spectral sensitivity characteristics shown in FIG. 35, and the photoelectric conversion units 52 and 72 marked R are red (R) shown in FIG. The photoelectric conversion units 53 and 73 marked with B have the spectral sensitivity characteristic of green (B) shown in FIG. The two pixels 421 and the pixels 422 and 423 form a Bayer array.

  The pixel 421 includes the microlens 10 and photoelectric conversion units 51 and 71. The photoelectric conversion unit 51 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The photoelectric conversion unit 71 has a rectangular shape in which the right side is substantially in contact with the vertical bisector of the microlens 10. The pixel 422 includes the microlens 10 and the photoelectric conversion units 52 and 72. The photoelectric conversion unit 52 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The photoelectric conversion unit 72 has a rectangular shape in which the right side is substantially in contact with the vertical bisector of the microlens 10. The pixel 423 includes the microlens 10 and the photoelectric conversion units 53 and 73. The photoelectric conversion unit 53 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The photoelectric conversion unit 73 has a rectangular shape in which the right side is substantially in contact with the vertical bisector of the microlens 10.

  FIG. 36 is a cross-sectional view of the pixel 421. In the pixel 421, the common microlens 10 is disposed in front of the photoelectric conversion units 51 and 71, and the photoelectric conversion units 51 and 71 are projected forward by the microlens 10. A green color filter 401 is disposed between the microlens and the photoelectric conversion unit. The photoelectric conversion units 51 and 71 are formed on the semiconductor circuit substrate 29, and the color filter 401 and the microlens 10 are integrally and fixedly formed thereon by the manufacturing process of the semiconductor image sensor. The pixels 422 and 423 have the same configuration except for the color of the color filter.

  In the pixel array configuration of the image sensor 211 shown in FIG. 34, image shift detection in the direction in which the pair of photoelectric conversion units (51, 71), (52, 72), (53, 73) is arranged (horizontal direction in FIG. 34). Is possible in three colors, green, red and blue. Accordingly, in calculating the defocus amount of the minute block, the probability that the focus detection becomes impossible due to the color of the subject pattern is reduced.

  FIG. 37 shows a pixel arrangement configuration of the imaging element 211 of a modification. In the image sensor 211 of this modification, the pixels 421, 423, 424, and 425 are regularly arranged in a two-dimensional manner. In FIG. 37, photoelectric conversion units 51, 71, 54 and 74 marked G have the green (G) spectral sensitivity characteristics shown in FIG. 35, and photoelectric conversion units 55 and 75 marked R are shown in FIG. The photoelectric conversion units 53 and 73 having red (R) spectral sensitivity characteristics and B are green (B) spectral sensitivity characteristics shown in FIG. The pixels 421, 423, 424, 425 form a Bayer array.

  The pixel 421 includes the microlens 10 and photoelectric conversion units 51 and 71. The photoelectric conversion unit 51 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The photoelectric conversion unit 71 has a rectangular shape in which the right side is substantially in contact with the vertical bisector of the microlens 10. The pixel 423 includes the microlens 10 and the photoelectric conversion units 53 and 73. The photoelectric conversion unit 53 has a rectangular shape in which the left side is substantially in contact with the vertical bisector of the microlens 10. The photoelectric conversion unit 73 has a rectangular shape in which the right side is substantially in contact with the vertical bisector of the microlens 10.

  The pixel 424 includes the microlens 10 and photoelectric conversion units 54 and 74. The photoelectric conversion unit 54 has a rectangular shape whose upper side is substantially in contact with the horizontal bisector of the microlens 10. The photoelectric conversion unit 74 has a rectangular shape in which the lower side is substantially in contact with the horizontal bisector of the microlens 10. The pixel 425 includes the microlens 10 and photoelectric conversion units 55 and 75. The photoelectric conversion unit 55 has a rectangular shape whose upper side is substantially in contact with the horizontal bisector of the microlens 10. The photoelectric conversion unit 75 has a rectangular shape in which the lower side is substantially in contact with the horizontal bisector of the microlens 10.

  In the pixel array configuration of the image sensor 211 shown in FIG. 37, the image shift detection in the arrangement direction (horizontal direction in FIG. 37) of the pair of photoelectric conversion units (51, 71), (53, 73) is green and blue. In addition to being possible in color, image shift detection in the direction in which the pair of photoelectric conversion units (54, 74) and (55, 75) are arranged (vertical direction in FIG. 37) is possible in two colors of green and red. Therefore, in calculating the defocus amount of the minute block, the probability that the focus cannot be detected decreases depending on the color and direction of the subject pattern.

  In the pixel array shown in FIG. 37, defocus amounts for a plurality of colors and directions are obtained in one minute block. As a method for finally determining one defocus amount, there are the following methods. (1) An average of a plurality of defocus amounts. (2) Prioritize the defocus amount of a specific color. For example, priority is given to a green defocus amount with high specific visibility. (3) By selecting a defocus amount of a color having a high average value of data, it is possible to detect a focus with high SN and high accuracy. (4) A defocus amount with higher reliability is selected based on the reliability determination described above. (5) A priority according to the focus detection direction is provided.

  FIG. 38 is a diagram for explaining focus detection by the re-image pupil division type phase difference detection method. The pupil division method is not limited to the method using the microlens shown in FIG. 5, and may be a re-imaging pupil division type phase difference detection method. In FIG. 38, 91 is an optical axis of the interchangeable lens, 110 is a condenser lens, 111 is an aperture mask, 112 and 113 are aperture openings, 114 and 115 are re-imaging lenses, 116 is an image sensor (CCD) for focus detection, Reference numerals 132 and 133 denote focus detection light beams. Reference numeral 101 denotes an exit pupil set at a distance d5 in front of the planned imaging plane of the interchangeable lens. Here, the distance d5 is a distance determined according to the focal length of the condenser lens 110 and the distance between the condenser lens 110 and the aperture openings 112 and 113, and is hereinafter referred to as a distance measuring pupil distance.

  Reference numeral 702 denotes an area of the aperture opening 112 projected by the condenser lens 110 (ranging pupil), and reference numeral 703 denotes an area of the aperture opening 113 projected by the condenser lens 110 (ranging pupil). The condenser lens 110, the diaphragm mask 111, the diaphragm apertures 112 and 113, the re-imaging lenses 114 and 115, and the image sensor 116 constitute a re-imaging unit that simultaneously performs imaging and focus detection.

  In FIG. 38, the re-imaging unit on the optical axis 91 includes a condenser lens 110 disposed in the vicinity of the planned imaging surface of the interchangeable lens, an image sensor 116 disposed behind the condenser lens 110, and the condenser lens 110 and the image sensor 116. A pair of re-imaging lenses 114 and 115 that are disposed between and re-image a primary image formed in the vicinity of a predetermined imaging surface on the image sensor 116, and in the vicinity of the pair of re-imaging lenses (in the figure, the front surface). ), A diaphragm mask 111 having a pair of diaphragm openings 112, 113. The image sensor 116 is an image sensor in which a plurality of photoelectric conversion units are two-dimensionally arranged.

  Information corresponding to the intensity distribution of the pair of images re-imaged on the image sensor 116 is output from the image sensor 116, and image shift detection calculation processing (correlation processing, phase difference detection processing) is performed on the pair of images. Thus, the image shift amount of the pair of images is detected by a so-called pupil division type phase difference detection method (re-imaging method). The condenser lens 110 projects the aperture openings 112 and 113 of the aperture mask 111 as regions 702 and 703 on the exit pupil 701. Regions 702 and 703 are called distance measurement pupils. That is, a pair of images re-imaged on the image sensor 116 is formed by the light beams 132 and 133 that pass through the pair of distance measuring pupils 702 and 703 on the exit pupil 701. The re-imaging pupil division type phase difference detection method shown in FIG. 38 has an advantage that a normal image sensor can be used as the image sensor 116.

  39, pupil division type focus detection using a polarization filter will be described. The pupil division method is not limited to the microlens method, and can be realized using a polarizing filter as shown in FIG. In the figure, reference numeral 690 denotes a polarizing filter holding frame, and portions other than the polarizing filter are shielded from light. This polarizing filter holding frame 690 is disposed near the stop of the interchangeable lens. Reference numeral 692 denotes a polarizing filter, which forms a distance measuring pupil according to the position and shape of the filter. Reference numeral 693 denotes a polarizing filter, which forms a distance measuring pupil according to the position and shape of the filter. The polarization directions of the polarization filters 692 and 693 are orthogonal to each other. Reference numeral 91 denotes an optical axis of the interchangeable lens. Reference numeral 621 denotes a polarizing filter whose polarization direction coincides with that of the polarizing filter 692. Further, reference numeral 622 is a polarizing filter, and the polarizing direction is the same as that of the polarizing filter 693. 611 and 612 are photoelectric conversion units, 631 and 632 are pixels, and 672, 673, 682, and 683 are light beams.

  In FIG. 39, four adjacent pixels are schematically illustrated. In the pixel 631, the photoelectric conversion unit 611 receives the light beam passing through the distance measuring pupil formed by the polarization filter 692 by the action of the polarization filter 621, and a signal corresponding to the intensity of the image formed by the light beam 672 or 682. Is output. In the pixel 632, the photoelectric conversion unit 612 receives the light beam passing through the distance measuring pupil formed by the polarization filter 693 by the action of the polarization filter 622, and corresponds to the intensity of the image formed by the light beam 673 or 683. Output the signal.

  The pixels 631 and 632 using the polarization filter as described above are two-dimensionally arranged, and the output of the photoelectric conversion unit of each pixel is collected into an output group corresponding to the distance measurement pupil, thereby passing through each distance measurement pupil. It is possible to obtain information on the intensity distribution of a pair of images formed by the focus detection light beam to be formed on the pixel row. By applying an image shift detection calculation process (correlation calculation process, phase difference detection process), which will be described later, to this information, the image shift amount of a pair of images is detected by a so-called pupil division type phase difference detection method.

  The example of the image sensor shown in FIGS. 34 and 37 shows an example in which filters of three primary colors of red, blue, and green are used. However, an image sensor having only two colors and a filter for detecting four or more colors are used. The present invention can also be applied to an image sensor provided. Although the color separation filter is a primary color filter (RGB), a complementary color filter (green: G, yellow: Ye, magenta: Mg, cyan: Cy) may be employed. Furthermore, color separation can also be achieved by changing the spectral sensitivity characteristics of the photodiodes constituting the photoelectric conversion unit for each photoelectric conversion unit in addition to the color filter.

  In the pixel array of the image sensor shown in FIGS. 28 and 37, an example in which image misalignment detection is performed in two directions, the horizontal direction and the vertical direction, is shown. However, the alignment direction of the pair of distance measurement pupils is a direction other than the horizontal direction and the vertical direction. By arranging these pixels in the direction in which the pair of distance measuring pupils are arranged, it is possible to detect image misalignment in directions other than the horizontal direction and the vertical direction. In this way, the probability that focus detection is impossible in a minute block is reduced, and more accurate defocus distribution information can be obtained, so that image quality by image processing is improved.

  Note that the above-described imaging element can be formed as a CCD image sensor or a CMOS image sensor.

  In the flowchart shown in FIG. 8, an example in which the image data is stored in the memory card is shown, but the image data may be displayed on an electronic viewfinder or a back monitor screen (not shown) provided on the back of the body.

  FIG. 40 shows a configuration of an imaging apparatus according to a modification. In the image pickup apparatus shown in FIG. 1, an example in which the image pickup device 211 is used for both focus detection and image pickup is shown. However, as shown in FIG. 40, an image pickup device 212 dedicated to image pickup is provided, and the image pickup device of the above-described embodiment 211 may be used for focus detection and electronic viewfinder display. In FIG. 40, the camera body 203 is provided with a half mirror 221 for separating a photographic light beam, an imaging device 212 dedicated to imaging is provided on the transmission side, and an imaging device for focus detection and electronic viewfinder display on the reflection side. 211 is arranged. Before shooting, focus detection and electronic viewfinder display are performed according to the output of the image sensor 211. At the time of release, image data corresponding to the output of the imaging element 212 dedicated to imaging is generated. The half mirror 221 may be a total reflection mirror, and may be retracted from the photographing optical path during photographing. In such a configuration, the image processing of the embodiment described above is used only for display of the electronic viewfinder. It is possible to observe an image when the aperture is narrowed while securing the light quantity without actually reducing the aperture during observation.

<< Applicable scope of one embodiment >>
The imaging apparatus according to the embodiment is not limited to a digital still camera or a film still camera including an interchangeable lens and a camera body, but can be applied to a lens-integrated digital still camera, a video camera, or a film camera. Further, the present invention can also be applied to a small camera built in a mobile phone or the like, a surveillance camera, a teleo imaging device, or the like.

  Further, a device that captures an image and a device that performs image processing on the captured image may be formed separately. In image processing, a processing unit (CPU) capable of processing a large amount of data at high speed and an accompanying increase in power consumption and space are expected. It is possible to make the apparatus simple and compact, and it is possible to perform advanced image processing at high speed on the image data once stored. An apparatus for applying image processing is provided with an operation member for inputting image processing control parameters. Furthermore, it is possible to view images while adjusting the depiction performance at the time of reproduction with respect to the images captured and stored.

  There are the following two methods for transferring information between the image capturing device and the image processing device. (1) A pair of image data is transferred from the image capturing device side to the image processing device side, and basic image data and defocus distribution information are calculated on the image processing device side. In such a configuration, various methods can be applied to calculate the defocus distribution. (2) The basic image data and defocus distribution information are calculated on the image capturing device side, and the basic image data and defocus distribution information are passed to the image processing device side. With such a configuration, the volume of data to be transferred can be reduced.

As described above, according to the embodiment, the following effects can be obtained.
(1) Blur can be added to an image formed using only an image formed by a single optical system.
(2) Since the defocus amount distribution in the image is directly calculated based on the pair of images, it is not necessary to specify a focus distance or input optical parameters of the photographing lens, and the accuracy of image processing is improved.
(3) Since the image picked up by the taking lens is picked up by the image pickup device, image processing can be performed in real time, and the image processing parameters can be adjusted while viewing the image processing result at the shooting site. It becomes possible.
(4) Even when image processing is performed on an image captured by a single photographing lens, a natural image can be generated without a sense of incongruity.
(5) When calculating a defocus amount distribution based on a pair of images, even if an area where the defocus amount cannot be calculated occurs on the screen, the image characteristics (luminance, hue, saturation) of the area and the surrounding area are generated. A good defocus amount distribution can be determined based on the similarity of degree, contrast, contour, and the like.
(6) When calculating the defocus amount distribution based on the image shift amount of the pair of images, the defocus amount cannot be calculated because the defocus amount is calculated based on the image shift amounts in a plurality of directions. The area that becomes is reduced.
(7) Since the pair of images are stored in the portable recording medium independently, the image processing can be performed again so that the pair of images can be read later to obtain a desired result.
(8) Since the basic image data and the defocus distribution information are independently stored in the portable recording medium, the basic image data and the defocus distribution information are read out later, and image processing is performed so that a desired result is obtained. You can start over.
(9) Even when photographing is performed using a photographing optical system in which the aperture is not large, the blur effect when photographing with a large aperture can be obtained by post-processing. For example, in a compact camera, it is difficult to mount a photographic lens having a large aperture in space. When the aperture is increased, various characteristics of the optical system (field curvature, spherical aberration, chromatic aberration, astigmatism, etc.) Aberration, distortion, peripheral light, contrast, resolution, etc.) will deteriorate, so even if you have to shoot with the aperture opening restricted during shooting, you can post-process the blur effect when shooting with a large aperture opening. Can be obtained. Furthermore, when shooting a high-luminance subject, the blurring effect of shooting with a large aperture opening can be achieved even when adjusting the exposure amount by reducing the aperture opening to exceed the high-speed limit of the shutter speed. It can be obtained by post-treatment.

The figure which shows the structure of one embodiment Front enlarged view showing pixel arrangement configuration of image sensor Diagram showing pixel configuration Cross section of pixel Diagram explaining the principle of pupil detection type phase difference detection focus detection using microlenses Diagram showing the projection relationship on the exit pupil plane The figure explaining the image formation state in a pupil division type phase difference detection system The flowchart which shows operation | movement of the digital still camera (imaging device) of one embodiment The figure explaining the detail of defocus amount calculation processing Block diagram showing an overview of image processing Flow chart showing details of image processing Figure with the screen divided into small blocks Diagram showing defocus amount distribution Diagram showing defocus amount histogram Diagram showing examples of basic image data A figure that categorizes small blocks on the screen into four defocus attributes. Diagram of skin color area cut out from basic image data The figure which shows the image data which performed the image processing of outline extraction with respect to basic image data The figure which shows the example which groups the minute block of the screen Illustration of group J1 extracted from a small block on the screen Illustration of group J2 extracted from a small block on the screen Illustration of group J3 extracted from a small block on the screen The figure which shows the basic image data part corresponding to group J1 The figure which shows the basic image data part corresponding to group J2 The figure which shows the basic image data part corresponding to group J3 The figure which shows the image of the stage where the middle view is superimposed on the distant view The figure which shows the image of the stage where the middle scene was superimposed on the distant view, and the image of the stage where the foreground was further superimposed on it The figure which shows the modification of the pixel arrangement | sequence of an image pick-up element The figure which shows the ranging pupil corresponding to a photoelectric conversion part The figure which shows the pixel structure of a modification. The figure which shows the pixel structure of a modification. FIG. 31 is a cross-sectional view of the pixel shown in FIG. The figure which shows the pixel arrangement | sequence structure of the image pick-up element of the modification which expanded the pixel shown in FIG. 31 two-dimensionally The figure which shows the pixel arrangement structure of the image pick-up element of a modification The figure which shows the spectral sensitivity characteristic of a photoelectric conversion part Cross section of pixel The figure which shows the pixel arrangement structure of the image pick-up element of a modification The figure for demonstrating the focus detection of a re-imaging pupil division type phase difference detection system The figure explaining the focus detection of the pupil division method using a polarization filter The figure which shows the structure of the imaging device of a modification

Explanation of symbols

211 Image sensor 214 Body drive control device 219 Memory card

Claims (8)

An imaging element that captures a subject image by a subject light flux passing through an imaging optical system, wherein a plurality of pixels each composed of a microlens and a photoelectric conversion unit are two-dimensionally arranged, and each of the microlenses is The subject light beam is divided into a pair of light beams that respectively pass through a pair of pupil regions of the pupil of the imaging optical system, and the plurality of photoelectric conversion units receive a plurality of first light beams that receive one of the pair of light beams. An imaging device including a photoelectric conversion unit and a plurality of second photoelectric conversion units that receive the other light beam of the pair of light beams;
Image composition means for generating a composite image by adding first image data based on output signals of the plurality of first photoelectric conversion units and second image data based on output signals of the plurality of second photoelectric conversion units. When,
Focus detection means for calculating a defocus amount in each of a plurality of small areas in the imaging screen based on the first image data and the second image data;
Classification means for classifying the plurality of small areas into one of a plurality of defocus attributes according to the magnitude of the defocus amount calculated by the focus detection means;
The small area that cannot be calculated based on the defocus amount or the image data in the small area that is located around the small area that cannot be calculated for the small area that cannot be calculated by the focus detection unit. Defocus attribute setting means for setting the defocus attribute of
The combined image portion corresponding to each of the small regions divided into a plurality of groups based on the defocus attribute classified by the classification unit and the defocus attribute set by the defocus attribute setting unit is included in the defocus attribute of the group. An image processing apparatus comprising: an image processing unit that performs image processing according to a focus attribute. An imaging element that captures a subject image by a subject light flux passing through an imaging optical system, wherein a plurality of pixels each composed of a microlens and a photoelectric conversion unit are two-dimensionally arranged, and each of the microlenses is The subject light beam is divided into a pair of light beams that respectively pass through a pair of pupil regions of the pupil of the imaging optical system, and the plurality of photoelectric conversion units receive a plurality of first light beams that receive one of the pair of light beams. An imaging device including a photoelectric conversion unit and a plurality of second photoelectric conversion units that receive the other light beam of the pair of light beams;
Image composition means for generating a composite image by adding first image data based on output signals of the plurality of first photoelectric conversion units and second image data based on output signals of the plurality of second photoelectric conversion units. When,
Focus detection means for calculating a defocus amount in each of a plurality of small areas in the imaging screen based on the first image data and the second image data;
Classification means for classifying the plurality of small areas into one of a plurality of defocus attributes according to the magnitude of the defocus amount calculated by the focus detection means;
The small area that cannot be calculated based on the defocus amount or the image data in the small area that is located around the small area that cannot be calculated for the small area that cannot be calculated by the focus detection unit. Defocus attribute setting means for setting the defocus attribute of
The combined image portion corresponding to each of the small regions divided into a plurality of groups based on the defocus attribute classified by the classification unit and the defocus attribute set by the defocus attribute setting unit is included in the defocus attribute of the group. Image processing means for performing image processing according to the focus attribute;
Display means for displaying the composite image processed by the image processing means as a through image;
A release operation member for starting a photographing operation;
First image data based on output signals output from the plurality of first photoelectric conversion units and second output based on output signals output from the plurality of second photoelectric conversion units in response to an operation of the release operation member. An image pickup apparatus comprising: a recording unit that records image data. The imaging device according to claim 1 or 2,
The first and second photoelectric conversion units are arranged corresponding to the microlenses,
The image synthesizing unit adds the output signals of the first and second photoelectric conversion units corresponding to the microlenses . The imaging device according to claim 1 or 2 ,
The pair of microlenses arranged adjacent to each other among the plurality of microlenses has the first photoelectric conversion unit arranged corresponding to one of the microlenses, and the second microlens corresponding to the other microlens. Photoelectric conversion part is arranged,
The image pickup device adds the output signals of the first and second photoelectric conversion units respectively corresponding to the adjacently arranged microlenses . In the imaging device according to claim 3 or 4 ,
The plurality of microlenses includes: a first microlens that divides a pupil of the imaging optical system into a pair of light beams that respectively pass through a pair of pupil regions arranged in a predetermined direction; and the pupil of the imaging optical system, An imaging apparatus comprising: a second microlens that divides a pair of light beams that respectively pass through a pair of pupil regions arranged in a direction different from a predetermined direction . The imaging device according to claim 1 or 2 ,
The defocus attribute setting means determines the defocus attribute of the non-calculatable small area when the defocus attribute is the same in a plurality of small areas located around the non-calculatable small area. An image pickup apparatus having the same defocus attribute as that of the image pickup apparatus. The imaging device according to claim 1 or 2,
The defocus attribute setting means is not capable of calculating when the luminance information, hue information, saturation information, and contrast information of the image in the small area that cannot be calculated and the surrounding small areas are similar. An image pickup apparatus, wherein a defocus attribute of a small area is set to be the same as a defocus attribute of a similar surrounding small area. The imaging device according to claim 1 or 2,
And a grouping unit that divides the plurality of small areas into a plurality of groups based on the defocus attribute classified by the classification unit and the defocus attribute set by the defocus attribute setting unit. Imaging device.

Images