JP6254843B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus and a control method thereof.

  There has been proposed an imaging apparatus that can divide an exit pupil of a photographing lens (imaging optical system) into a plurality of regions and simultaneously photograph a plurality of parallax images corresponding to the divided pupil regions.

  Patent Document 1 discloses an imaging apparatus using a two-dimensional imaging element in which one microlens and a plurality of divided photoelectric conversion units (subpixels) are formed for one pixel. Each photoelectric conversion unit is configured to receive outgoing light from different partial areas of the exit pupil of the photographing lens, and performs pupil division. A plurality of parallax images corresponding to the partial areas can be generated by acquiring signals from the photoelectric conversion units that receive the same partial area from the plurality of pixels.

  The plurality of obtained parallax images are equivalent to light field (LF) data which is spatial distribution and angle distribution information of light intensity. Non-Patent Document 1 discloses a refocus technique for changing the in-focus position of a captured image after shooting by synthesizing images on a virtual imaging plane different from the imaging plane using the acquired LF data. .

US Pat. No. 4,410,804

Ren.Ng, et al., “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02, 2005.4.20

  When the exit pupil distance of the photographic lens and the set pupil distance of the image sensor are different in an interchangeable lens camera, the effective aperture value differs between the parallax images at the peripheral image height (region where the image height is large), and between the parallax images There is a problem that the amount of blur is different.

  The present invention has been made in view of the above problems, and a plurality of parallaxes obtained by using output signals of a plurality of sub-pixels that receive light emitted from different partial regions of the exit pupil of the imaging optical system. The object is to reduce the difference in blur amount between images.

  The above-described object is an imaging device in which a plurality of pixels are arranged, and each of the pixels is provided with a plurality of sub-pixels that receive light beams that pass through different pupil partial regions of the imaging optical system. A plurality of parallax images are generated from an input image by generating parallax images based on an acquisition unit that acquires an input image obtained from an image sensor and a signal of a subpixel that receives a light beam passing through the same pupil partial region. A generating means, a defocus amount calculating means for obtaining a defocus amount distribution from a plurality of parallax images, a defocus amount distribution, an effective aperture value of an exit pupil of an imaging optical system for each of a plurality of sub-pixels, and A blur amount calculation unit that obtains a blur amount distribution for each of a plurality of parallax images from an image height, and a correction process that applies a correction process that suppresses a difference in the amount of blur between each of the plurality of parallax images and another parallax image. It is achieved by an image processing apparatus characterized by comprising a means.

  According to the present invention, it is possible to reduce a difference in blur amount between a plurality of parallax images obtained by using output signals of a plurality of sub-pixels that receive light emitted from different partial areas in an exit pupil of an imaging optical system. it can.

1 is a diagram illustrating a functional configuration example of a digital still camera as an example of an image processing apparatus according to an embodiment of the present invention. The figure which shows the outline of the pixel arrangement | sequence of the image pick-up element in embodiment of this invention. The figure which shows the structural example of the pixel which the image pick-up element in embodiment of this invention has. The figure for demonstrating the outline of the pixel and pupil division in embodiment of this invention The figure for demonstrating the outline of the image pick-up element in the embodiment of this invention, pupil division, and the outline of the image shift amount and defocus amount between parallax images Diagram for explaining the outline of pupil shift The figure which shows the example of the difference in the amount of blurs between several parallax images by the difference in an effective aperture value Flowchart for explaining blur correction processing in an embodiment of the present invention

● (first embodiment)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.
[overall structure]
FIG. 1 is a diagram illustrating a functional configuration example of a digital still camera 100 (hereinafter simply referred to as a camera 100) as an example of an image processing apparatus according to an embodiment of the present invention. Note that the present invention can be applied not only to an imaging apparatus but also to any electronic device (computer, tablet terminal, game machine, mobile phone, robot, etc.) provided with a camera. In addition, since an imaging function is not essential, the present invention can be applied to any electronic device (such as a general-purpose computer or an embedded device) that does not include a camera.

  The first lens group 101 is disposed at the tip of the photographing optical system (imaging optical system) and is held so as to be movable back and forth along the optical axis. The shutter 102 functions not only as a shutter for controlling the exposure time at the time of still image shooting, but also as an aperture for adjusting the amount of light at the time of shooting by adjusting the aperture diameter. The second lens group 103 disposed on the back surface (on the imaging element side) of the shutter 102 can be moved back and forth along the optical axis integrally with the shutter 102, and realizes a zoom function together with the first lens group 101.

  The third lens group 105 is a focus lens and can move back and forth along the optical axis. The optical low-pass filter 106 is disposed in front of the image sensor 107 and reduces false colors and moire generated in the captured image. The image sensor 107 is composed of a two-dimensional CMOS image sensor and its peripheral circuit, and is disposed on the image forming surface of the image forming optical system.

Under the control of the zoom drive circuit 129, the zoom actuator 111 rotates a cam cylinder (not shown) to drive at least one of the first lens group 101 and the third lens group 105 along the optical axis to zoom (change). Double) to realize the function. The shutter actuator 112 controls the exposure light amount at the time of capturing a still image while controlling the aperture diameter of the shutter 102 to adjust the amount of imaged light according to the control of the shutter drive circuit 128.
The focus actuator 114 drives the third lens group 105 along the optical axis according to the control of the focus drive circuit 126.

  The flash 115 is preferably a flash illumination device using a xenon tube, but may be an illumination device including LEDs that emit light continuously. The AF auxiliary light output unit 116 projects an image of a mask having a predetermined aperture pattern onto a subject field via a posting lens, and improves the focus detection capability for a low-luminance subject or a low-contrast subject.

  The CPU 121 controls the operation of the entire camera 100, and includes a calculation unit, a ROM, a RAM, an A / D converter, a D / A converter, a communication interface circuit, and the like (not shown). The CPU 121 executes a program stored in the ROM and controls various circuits included in the camera 100 to realize functions of the camera 100 such as AF, AE, image processing, and recording.

  The flash control circuit 122 controls the lighting of the flash 115 in synchronization with the imaging operation. The auxiliary light drive circuit 123 controls the lighting of the AF auxiliary light output unit 116 during the focus detection operation. The image sensor driving circuit 124 controls the operation of the image sensor 107 and A / D converts the image signal read from the image sensor 107 and outputs the image signal to the CPU 121. The image processing circuit 125 applies image processing such as γ conversion, color interpolation, and JPEG encoding to the image signal.

  The focus drive circuit 126 moves the third lens group 105 along the optical axis by driving the focus actuator 114 based on the focus detection result, and performs focus adjustment. The shutter drive circuit 128 controls the opening diameter and opening / closing timing of the shutter 102 by driving the shutter actuator 112. The zoom drive circuit 129 drives the zoom actuator 111 according to a zoom operation input from the photographer by pressing a zoom operation switch included in the operation switch 132, for example.

  The display 131 is an LCD or the like, and displays information related to the imaging mode of the camera 100, a preview image before imaging and a confirmation image after imaging, information on a focusing state at the time of focus detection, and the like. The operation switch 132 includes a power switch, a release (imaging trigger) switch, a zoom operation switch, an imaging mode selection switch, and the like. The recording medium 133 is a detachable semiconductor memory card, for example, and records captured images.

[Image sensor]
FIG. 2 is a diagram schematically illustrating an arrangement example of the imaging pixels and the focus detection pixels in the imaging element 107, and representatively illustrates an area in which the imaging pixels are arranged in 4 horizontal pixels × 4 vertical pixels. In the present embodiment, the photoelectric conversion area of each imaging pixel is divided into two in the vertical and horizontal directions, and each photoelectric conversion area is referred to as a sub-pixel. Therefore, in FIG. 2, it can also be said that the sub-pixels are regions in which 8 horizontal pixels × 8 vertical pixels are arranged.

In this embodiment, the pixel period ΔX is 10 μm, the effective pixel number N _LF is 3600 horizontal rows × 24,000 vertical rows = approximately 8.6 million pixels (the sub pixel cycle Δx is 5 μm, and the effective sub pixel number N is 7200 horizontal rows × 4800 vertical rows). The description will be made assuming that the image pickup device is approximately 34.6 million pixels.

  In the present embodiment, the 2 × 2 pixel group 200 in the upper left of FIG. 2 corresponds to a repeating unit of a primary color Bayer array color filter provided in the image sensor 107. Accordingly, the pixel 200R having R (red) spectral sensitivity is arranged at the upper left, the pixel 200G having G (green) spectral sensitivity is arranged at the upper right and lower left, and the pixel 200B having B (blue) spectral sensitivity is arranged at the lower right. Has been. Further, as representatively shown in the upper right pixel in FIG. 2, each pixel has sub-pixels 201 to 204 which are photoelectric conversion units equally divided into 2 × 2.

  FIG. 3A shows a plan view of one pixel shown in FIG. 2 (here, 200G) as viewed from the light receiving surface side (+ z side) of the image sensor 107, and a- in FIG. FIG. 3B shows a cross-sectional view of the a cross section viewed from the −y side.

  As shown in FIG. 3, in the pixel 200 </ b> G of the present embodiment, a microlens 305 for condensing incident light is formed on the light receiving side of each pixel, and is divided into N (two divisions) in the x direction and N in the y direction. Divided (divided into two) photoelectric conversion units 301 to 304 are formed. The photoelectric conversion units 301 to 304 correspond to the subpixels 201 to 204, respectively.

  The photoelectric conversion units 301 to 304 may be a pin structure photodiode in which an intrinsic layer is sandwiched between a p-type layer and an n-type layer, or an intrinsic layer may be omitted as necessary to form a pn junction photodiode. Also good.

  In each pixel, a color filter 306 is formed between the microlens 305 and the photoelectric conversion units 301 to 304. Further, as necessary, the spectral transmittance of the color filter may be changed for each sub-pixel, or the color filter may be omitted.

  Light incident on the pixel 200 </ b> G illustrated in FIG. 3 is collected by the microlens 305, dispersed by the color filter 306, and then received by the photoelectric conversion units 301 to 304.

  In the photoelectric conversion unit, a pair of electrons and holes are generated according to the amount of received light, separated by a depletion layer, negatively charged electrons are accumulated in an n-type layer, and the holes are connected to a constant voltage source (not shown). Then, it is discharged to the outside of the image sensor 107 through the p-type layer 300.

  The electrons accumulated in the n-type layers of the photoelectric conversion units 301 to 304 are transferred to the capacitance unit (FD) via the transfer gate and converted into a voltage signal.

  How pupil division is performed by the pixel structure of this embodiment shown in FIG. 3 will be described with reference to FIGS. 4 and 5A. FIG. 4 shows the relationship between the aa cross section of the pixel 200G shown in FIG. 3B and the exit pupil of the imaging optical system. In FIG. 4, in order to correspond to the coordinate axis of the exit pupil plane, the x-axis and y-axis of the cross-sectional view are inverted with respect to FIG. FIG. 5A schematically shows an optical path from the subject to the photoelectric conversion units 303 and 304.

The image sensor 107 is disposed in the vicinity of the imaging surface of the photographing lens (imaging optical system), and the light flux from the subject passes through the exit pupil 400 of the imaging optical system and enters each pixel. A surface on which the image sensor is arranged is defined as an imaging surface 800 (FIG. 5A). The pupil partial regions 501 to 504 are roughly divided by the light receiving surfaces of the photoelectric conversion units 301 to 304 (subpixels 201 to 204) divided by N _θ × N _θ (N _θ = 2 in this embodiment) and the microlenses. Has a conjugate relationship. Accordingly, the pupil partial areas 501 to 504 represent pupil partial areas that can be received by the photoelectric conversion units 301 to 304 (subpixels 201 to 204), respectively.

The exit pupil 400 of the imaging optical system is _{N p (= N θ × N} θ) divided into different partial pupil region, called the _{N p} and the pupil division number. When the aperture value of the imaging optical system is F, the effective aperture value of the pupil partial region is approximately N _θ F. In FIG. 4, a pupil region 500 is a pupil region that can receive light in the entire pixel 200 </ b> G including the photoelectric conversion units 301 to 304 (subpixels 201 to 204).

  The generation principle of the parallax image in the present embodiment will be described with reference to FIG. The photoelectric conversion units 301 to 304 (subpixels 201 to 204) of each pixel receive light beams that pass through different pupil partial areas of the pupil partial areas 501 to 504, respectively. FIG. 5A shows an optical path where the photoelectric conversion unit 303 receives a light beam emitted from the pupil partial region 503 and the photoelectric conversion unit 304 receives a light beam emitted from the pupil partial region 504.

In this way, the photoelectric conversion units 301 to 304 receive the light beams emitted from different pupil partial regions. Receiving light beams emitted from different pupil partial areas corresponds to different viewpoints. Therefore, if a signal of a specific photoelectric conversion unit (sub-pixel) is acquired from each pixel, the pupil from which the received light beams are emitted. A parallax image corresponding to the partial area can be obtained. For example, if a signal of the sub-pixel 203 (photoelectric conversion unit 303) is acquired from each pixel, a parallax image having a resolution of the effective number of pixels corresponding to the pupil partial region 503 of the imaging optical system can be obtained. The same applies to other sub-pixels. Therefore, in the present embodiment, an imaging device in which a plurality of pixels provided with a plurality of sub-pixels that receive light beams that pass through different pupil partial regions of the imaging optical system are arranged for each different pupil partial region (that is, pupil division). equal number of) the parallax image number N _p is obtained.
Further, by adding the signals of the sub-pixels 201 to 204 for each pixel, it is possible to generate a captured image having a resolution of the effective number of pixels.

Hereinafter, the relationship between the image shift amount and the defocus amount of the parallax image in the present invention will be described.
FIG. 5B shows a schematic relationship diagram between the image shift amount and the defocus amount between parallax images. Arranged imaging element 107 to the imaging surface 800, similarly to FIG. 5 (a), the exit pupil of the imaging optical system is _{N p} divided into partial pupil region 501 - 504 (4 split).

  The defocus amount d is defined by the absolute value | d | of the distance from the imaging position of the subject to the imaging surface, and a code representing the positional relationship between the imaging position of the subject and the imaging surface. A negative sign (d <0) indicates a front pin state where the imaging position of the subject is on the subject side from the imaging surface, and a positive sign indicates a rear pin state where the imaging position of the subject is on the opposite side of the subject from the imaging surface (d> 0). ). The in-focus state where the imaging position of the subject is on the imaging surface is d = 0. FIG. 5B shows an example in which the subject at position 801 is in focus (d = 0) and the subject at position 802 is in the front pin state (d <0). The front pin state (d <0) and the rear pin state (d> 0) are combined to form a defocus state (| d |> 0).

  In the front pin state (d <0), the light beam that has passed through the pupil partial region 503 (504) out of the light beam from the subject at the position 802 is once condensed and then centered on the gravity center position G3 (G4) of the light beam. And spreads to the width Γ3 (Γ4), resulting in a blurred image on the imaging surface 800. The blurred image is received by the sub-pixel 203 (204) that constitutes each pixel arranged in the image sensor, and a parallax image corresponding to the pupil partial region 503 (504) is generated. Therefore, in the parallax image generated from the signal of the sub-pixel 203 (204), the subject at the center of gravity G3 (G4) is recorded as a subject image in which the subject at the position 802 is blurred by the width Γ3 (Γ4). The blur width Γ3 (Γ4) of the subject image generally increases in proportion to the amount of defocus amount d | d |. Similarly, the magnitude | p | of the image shift amount p (= G3-G4) of the subject image between the parallax images is approximately proportional as the magnitude | d | of the defocus amount d increases. It will increase. The rear pin state (d> 0) is the same as the front pin state except that the image shift direction of the subject image between the parallax images is opposite to the front pin state. In the focused state (d = 0), the positions of the centers of gravity of the subject images between the parallax images match (p = 0), and no image shift occurs.

  Therefore, the image shift amount between the plurality of parallax images obtained in the present embodiment increases as the defocus amount of the parallax image increases. In the present embodiment, the amount of image shift between parallax images is calculated by correlation calculation because of the relationship that the amount of image shift between parallax images increases as the defocus amount of the parallax image increases. Thus, focus detection by the imaging surface phase difference method is performed. A defocus map is calculated by performing focus detection using an imaging surface phase difference method for each image height.

  The pupil shift that occurs at the peripheral image height (position where the image height is large) of the image sensor according to the magnitude relationship between the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the image sensor will be described. FIG. 6 shows pupil partial areas 501a (501b, 501c) to 504a (504b, 504c) received by the subpixels 201 to 204 of the pixels located in the periphery of the image sensor, and the exit pupil 400 of the imaging optical system. The positional relationship of is shown.

  FIG. 6A shows a case where the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the image sensor substantially coincide (D1≈Ds). In this case, the exit pupil 400 of the imaging optical system is pupil-divided into pupil partial areas 501a to 504a almost uniformly in both the central pixel having a small image height and the peripheral pixels having a large image height. .

  On the other hand, FIG. 6B shows a case where the exit pupil distance Dl of the imaging optical system is shorter than the set pupil distance Ds of the image sensor (D1 <Ds). In this case, in the peripheral pixels, the exit pupil 400 of the imaging optical system is non-uniformly divided into pupil partial regions 501b to 504b. In the example of FIG. 6B, the effective aperture value of the pupil partial region 503b is smaller (brighter) than the effective aperture values of the pupil partial regions 501b and 504b, and the effective aperture value of the pupil partial region 502b is higher. Large (dark) value. Conversely, in the pixel at the target position across the optical axis center, the effective aperture value of the pupil partial region 502b is smaller (brighter) than the effective aperture values of the pupil partial region 501b and the pupil partial region 504b. The effective aperture value of the pupil partial area 503b is a larger (darker) value.

  FIG. 6C shows a case where the exit pupil distance Dl of the imaging optical system is longer than the set pupil distance Ds of the image sensor (D1> Ds). Also in this case, in the peripheral pixels, the exit pupil 400 of the imaging optical system is non-uniformly divided into pupil partial regions 501c to 504c. In the example of FIG. 6C, contrary to the example of FIG. 6B, the effective aperture value of the pupil partial region 502c is smaller (brighter) than the effective aperture values of the pupil partial regions 501c and 504c. Thus, the effective aperture value of the pupil partial region 503c is a larger (darker) value. In contrast, in the pixel at the opposite position across the center of the optical axis, the effective aperture value of the pupil partial region 503c is smaller (brighter) than the effective aperture value of the pupil partial regions 501c and 504c. The effective aperture value in the region 502c is a larger (darker) value.

  As described above, when the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the image sensor are different, the effective aperture values in the peripheral pixels are different between the parallax images, and thus the amount of blur in the peripheral portion is the parallax image. The phenomenon of difference between them occurs.

FIG. 7 is a diagram illustrating an example of a plurality of parallax images 601 to 604 at different viewpoints corresponding to the pupil partial regions 501 to 504.
In FIG. 7, regions 701, 702, 703, and 704 in the defocused state (| d |> 0) in the upper left peripheral portion of each parallax image have the largest amount of blur in the region 703, and regions 701 and 704 are intermediate. It can be seen that the amount of blur in the region 702 is the smallest. On the contrary, areas 705 to 708 in the defocused state (| d |> 0) in the peripheral portion on the right side of each parallax image have the largest amount of blur in the area 706, and areas 705 and 708 have an intermediate blur amount. It can be seen that the amount of blur of 707 is the smallest.

  In this embodiment, the blur amount is adjusted for each parallax image according to the effective aperture value for each image height of each parallax image and the defocus map calculated from the image shift amount between the parallax images, and the same parallax image is obtained. The difference in blur amount in the area is corrected.

[Defocus amount correction processing]
Hereinafter, the blur amount correction processing in the present embodiment will be described with reference to the flowchart shown in FIG. In this embodiment, the blur amount correction process is realized by the CPU 121 executing software, but part or all of the process may be realized by a circuit such as an ASIC.

In S <b> 100, the CPU 121 reads out signals from the sub-pixels 201 to 204 of the image sensor 107 through the image sensor drive circuit 124, and acquires an input image to which at least color interpolation (demosaiking) is applied by the image processing circuit 125. Note that an equivalent image may be acquired from the recording medium 133 or another device. Note that when an image is acquired from the image sensor 107, it is assumed that the exposure condition is set in advance by the automatic exposure setting that is performed by a general digital camera.
In the following description, it is assumed that each pixel has information on each component of RGB (or YUV) for ease of explanation and understanding. However, the input image may be a RAW image before demosaicing. In this case, after applying the following processing for each pixel of the same color, the mosaic image may be returned and demosaicing processing may be performed.

The number of pixels in the x direction of the image sensor 107 is N _x , the number of pixels in the y direction is N _y , the position in the x direction is i (= 1 to N _x ), and the position in the y direction is j (= 1 to N _y ). To do. The starting point of the position may be, for example, the pixel at the upper left corner. Therefore, the number of effective pixels of the image sensor 107 is N _LF = N _x × N _y . Further, the pixel position of the image sensor 107 (i, j) and P (i, j), the integer a = a 1 to N _theta, pixel P (i, j) of the a-th sub-pixel _P a (i , J). Further, a signal received by the sub-pixel P _a (i, j) is defined as L _a (i, j).

Therefore, the input image has all the sets L = {L _a (i, j) | i = 1 to N _x , j = 1 to N _y , a = 1 to N _θ } of the signal L _a (i, j). It is expressed.

  In S200, the CPU 121 generates four parallax images corresponding to the pupil partial regions 501 to 504 of the imaging optical system by collecting the signals of the same sub-pixels of the respective pixels. As described above, in these parallax images, as the defocus amount of the parallax image increases, the amount of image shift between the plurality of parallax images increases.

A set L _a = {L _a (i, j) | i = 1 to N _x , j = 1 to N _y } limited to the signal of the a-th subpixel is each parallax image. A plurality of parallax images {L _a | a = 1 to N _θ } are generated.

Since in this embodiment is a pupil division number N θ ₌ 4, the sub-pixels 201 to 204 and 1 to 4 th sub-pixel, respectively. That is, a = 1 corresponds to the subpixel 201, a = 2 corresponds to the subpixel 202, a = 3 corresponds to the subpixel 203, and a = 4 corresponds to the subpixel 204. Therefore, the parallax image corresponding to the pupil partial region 501 is L ₁ , the parallax image corresponding to the pupil partial region 502 is L ₂ , the parallax image corresponding to the pupil partial region 503 is L ₃ , and the parallax image corresponding to the pupil partial region 504 it is _{L 4.}

  In S300, the CPU 121 calculates a defocus map from the relationship in which the magnitude of the image shift amount between the parallax images increases as the magnitude of the defocus amount of the parallax images increases. The CPU 121 as the defocus amount calculation means calculates the image shift amount between the two parallax images by correlation calculation, and the image shift amount corresponds to the focus state, zoom state, aperture value, sensor image height coordinate, etc. of the lens. The defocus amount is calculated by multiplying the conversion factor. The defocus map is a set of defocus amounts obtained for each coordinate of the parallax image, for example, and represents a distribution of the defocus amounts. There is no particular limitation on the method of calculating the defocus amount for each coordinate, but it can be obtained by the following method, for example.

A parallax image L ₁₃ obtained by adding the parallax image L ₁ and the parallax image L ₃ , and a parallax image L ₂₄ obtained by adding the parallax image L ₂ and the parallax image L ₄ are used. Predetermined number of horizontal pixel lines of a predetermined number of pixels centered on the coordinates of interest, taken from two parallax images of the parallax images L ₁₃ and parallax images L _24, the defocus amount of the horizontal direction by the correlation operation between the two calculate. Also, a parallax image L ₁₂ obtained by adding the parallax image L ₁ and the parallax image L ₂ , and a parallax image L ₃₄ obtained by adding the parallax image L ₃ and the parallax image L ₄ are used. Similarly, a predetermined number of vertical pixel lines of a predetermined number of pixels centered on the coordinates of interest, taken from two parallax images of the parallax images L ₁₂ and parallax images L _34, the correlation operation of both vertical de Calculate the focus amount. The average defocus amount in the horizontal direction and the vertical direction is set as the defocus amount of the target coordinates. In addition to the method using a horizontal pixel line or a vertical pixel line, other methods such as obtaining a defocus amount using a pixel block centered on a target coordinate may be used. The defocus amount may be calculated for other combinations of two parallax images.

In S400, the CPU 121 performs a correction process for adjusting the amount of blur for each parallax image. Details of the correction processing will be described with reference to the flowchart of FIG.
First, the CPU 121 generates information indicating the relationship between the image height and the effective aperture value for each parallax image. CPU121, for each parallax image L _a, the information of the exit pupil of the imaging optical system (the incident angle dependence of the light-receiving rate) pupil intensity distribution of a sub-pixel, and the image height, the pupil portion a sub-pixel is received Calculate the relationship with the effective aperture value of the area.

  The lens pupil distance can be acquired from the lens unit, and the sensor pupil distance is stored in the camera. Therefore, the CPU 121 can know the magnitude relationship between the lens pupil distance and the sensor pupil distance, and thereby know in which direction the center of the exit pupil and the center of the pixel are shifted in the peripheral portion where the image height is large. Further, the CPU 121 can calculate the current size of the exit pupil 400 from the current F value and the lens pupil distance. If the size of the exit pupil is known, the pupil shift amount corresponding to the image height can be obtained.

The effective aperture value of the sub-pixel is smaller as the overlap between the light receiving portion of each sub-pixel indicated by the dotted line in FIG. 6 and the exit pupil 400 is larger, and becomes larger as the overlap is smaller. In addition, the light receiving efficiency of the sub-pixel has an incident angle dependency and is called a pupil intensity distribution. In general, the distribution is such that the efficiency decreases toward the periphery with the center of the pixel as the maximum efficiency. Therefore, the CPU 121 obtains an effective F value corresponding to the overlap between the pupil intensity distribution for each sub-pixel and the current exit pupil. Since the incident angle of light is different between a pixel at a small image height and a pixel at a large image height, the pupil intensity distribution differs depending on the image height. Accordingly, the pupil intensity distribution corresponding to each coordinate is obtained in advance, or the pupil intensity distribution at coordinates where the image height is greater than 0 can be estimated from the pupil intensity distribution at image height 0 using the sensor pupil distance.
In this way, the CPU 121 obtains an effective aperture value for each pixel coordinate, and creates an effective aperture value map representing the relationship between the image height and the effective aperture value or the distribution of the effective aperture value for each parallax image (S401). .

  Next, the CPU 121 as the blur amount calculation unit calculates a blur amount map representing the distribution of the blur amount for each coordinate for each parallax image from the effective aperture value map and the defocus map obtained for each parallax image (S402). ). The blur amount can be obtained by defocus amount / effective aperture value.

  Next, the CPU 121 serving as an average blur amount calculation unit generates an average blur amount map representing the distribution of the average blur amount for each coordinate from the blur amount map created for the four parallax images (S403). Further, the CPU 121 as the blur amount difference calculating unit generates a blur amount difference map representing the distribution of the blur amount difference for each coordinate for each parallax image from the blur amount map and the average blur amount map (S404). When generating the blur amount difference map, the CPU 121 gives a negative sign to the difference for a blur amount smaller than the average, and gives a positive sign to the difference for a blur amount larger than the average.

Then, the CPU 121 applies blur correction processing corresponding to the difference from the average blur amount for each parallax image (S405). In this embodiment, if the value of the blur amount difference map is negative (if the blur amount is smaller than the average blur amount), the CPU 121 applies processing for increasing the blur amount. For example, the blur amount can be increased so as to be the value of the average blur amount map. On the other hand, if the value of the blur amount difference map is positive and larger than the predetermined amount (if the blur amount is larger than the average blur amount by a predetermined amount), the CPU 121 performs processing for reducing the blur amount, for example, edge enhancement processing. For example, the edge enhancement processing can be performed so that the average blur amount map value is obtained. If the value of the blur amount difference map is positive and less than or equal to the predetermined amount, the blur amount is not particularly corrected.
Note that the blur correction process may be performed for each coordinate (pixel), or may be performed collectively on a region where the difference amount has a similar value.

  One of the parallax images subjected to the blur correction process in this way can be displayed on the display 131. If the display 131 is a stereoscopic display, two parallax images can be displayed on the display 131 to provide a stereoscopic display.

  As described above, in this embodiment, the defocus map is calculated from a plurality of parallax images, and the amount of blur according to the position in each parallax image from the effective aperture value of the sub-pixel according to the image height and the defocus map. Ask for. Then, since the process of correcting the blur amount is applied to each parallax image so as to reduce the difference in blur amount between the parallax images, the pupil generated in the peripheral portion particularly when the lens pupil distance and the sensor pupil distance are different. It is possible to suppress the influence of variation in the effective aperture value between the sub-pixels due to the shift.

  If necessary, dark correction, shading correction, and the like may be performed on either or both of the input image and the parallax image using the image processing circuit 125.

  Further, although the present embodiment has been described in the case where the exit pupil of the imaging optical system is divided into 2 × 2 into different pupil partial regions, the same applies to other pupil division numbers. The number of pupil divisions in the x direction and the number of pupil divisions in the y direction may be different. Further, the pupil division direction may be different from the x direction and the y direction.

● (Second Embodiment)
Next, a second embodiment of the present invention will be described. The functional configuration of the image processing apparatus according to the present embodiment may be the same as that of the first embodiment, and the blur amount correction processing for the parallax image is also calculated until the blur amount map is calculated for each parallax image in S400. This is the same as that described in the first embodiment with reference to FIG.

  In the first embodiment, correction processing is performed to increase the blur amount for a blur amount smaller than the average blur amount, and to reduce the blur amount for a blur amount larger than the average blur amount by a predetermined amount or more.

On the other hand, in the present embodiment, a maximum blur amount map representing the distribution of the maximum blur amount in the four parallax images for each coordinate is generated from the blur amount map created for the four parallax images. Further, by subtracting the maximum blur amount map from the blur amount map, a blur amount difference map representing the distribution of the difference in blur amount from the maximum blur amount for each coordinate is generated for each parallax image. The blur amount difference map of the present embodiment is a non-positive (0 or negative) value because the maximum blur amount map is subtracted from the blur amount map.
Thereafter, a blur correction process corresponding to the difference from the maximum blur amount is applied to each parallax image. In this embodiment, if the value of the blur amount difference map is negative (if the blur amount is smaller than the average blur amount), the CPU 121 applies processing for increasing the blur amount. For example, the blur amount can be increased so as to be the value of the maximum blur amount map.

  By performing the correction for increasing the blur amount for a portion with a small blur amount, it is possible to reduce the difference in the blur amount between the parallax images while reducing the processing.

● (Third embodiment)
Next, a third embodiment of the present invention will be described. The functional configuration of the image processing apparatus according to the present embodiment may be the same as that of the first embodiment. Further, the blur amount correction processing for the parallax image is also performed in the first embodiment using FIG. 8 except that point image data is generated in S400 and the point image data is used in addition to the blur amount difference map for blur correction. The same as described.

  In the present embodiment, point image data is calculated for each parallax image and for each image height in addition to the effective aperture value map, the blur amount map, the average blur amount map, and the blur amount difference map. A point image can be identified from the F value, the lens pupil distance, and the pupil intensity distribution. Since the point image data represents how the blur is spread, the use of the point image data makes it possible to accurately correct the blur amount.

  In S400, the CPU 121 calculates point image data for each parallax image in addition to the effective aperture value map. For the point image data, point image data for each coordinate is calculated from the pupil intensity distribution (incidence angle dependency of the light reception rate) of the subpixels forming the parallax image, the F value, and the lens pupil distance for each parallax image. Further, the CPU 121 generates a blur amount map, an average blur amount map, and a blur amount difference map in the same manner as in the first embodiment.

  If the value of the blur amount difference map is negative for each coordinate of the parallax image, the CPU 121 performs a convolution process (a process for increasing blur) based on the value of the blur amount difference map and the point image data. On the other hand, when the value of the blur amount difference map is positive and larger than the predetermined value, deconvolution processing (processing for reducing blur) based on the value of the blur amount difference map and the point image data is performed.

  According to the present embodiment, although the processing load increases, it is possible to correct the blur amount with high accuracy, and it is possible to more accurately suppress the difference in blur amount between parallax images. Here, as in the first embodiment, the case where the correction is performed when the blur amount is smaller than the average and when the blur amount is larger than the average by a predetermined amount has been described. However, the same correction as in the second embodiment can be performed. .

(Other embodiments)
In the above-described embodiment, various maps are generated by obtaining values for each coordinate of the parallax image. However, various maps may be generated by obtaining values for each pixel block including a plurality of pixels. .

  In the above-described embodiment, the method of correcting the blur amount so as to reduce the difference from the average blur amount has been described. However, if the difference in the blur amount between parallax images is suppressed, the method other than the average blur amount is described. The blur amount may be corrected based on the value.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (11)

An imaging device in which a plurality of pixels are arranged, and each of the pixels is obtained from an imaging device provided with a plurality of sub-pixels that receive light beams that pass through different pupil partial regions of the imaging optical system. Acquisition means for acquiring an input image
Generating means for generating a plurality of parallax images from the input image by generating a parallax image based on a signal of a sub-pixel that receives a light beam passing through the same pupil partial region;
Defocus amount calculating means for obtaining a distribution of defocus amount from the plurality of parallax images;
A blur amount calculation for obtaining a blur amount distribution for each of the plurality of parallax images from the distribution of the defocus amount, the effective aperture value of the exit pupil of the imaging optical system for each of the plurality of sub-pixels, and the image height. Means,
A correction unit that applies a correction process that suppresses a difference in blur amount from another parallax image for each of the plurality of parallax images;
An image processing apparatus comprising: Further, an average blur amount calculating means for obtaining a distribution of average blur amounts of the plurality of parallax images from the blur amount distribution,
2. The correction unit suppresses a difference in blur amount from another parallax image by correcting the blur amount in the parallax image so as to reduce a difference from the average blur amount. The image processing apparatus described. The correction means includes
For areas where the blur amount is smaller than the average blur amount, increase the blur amount according to the difference from the average blur amount,
The blur amount than the average blur amount reduces the volume Ke amount for large area exceeds a predetermined amount,
3. The blur amount difference from another parallax image is suppressed by not correcting the blur amount for a region where the blur amount is larger than the average blur amount by a predetermined amount or less. Image processing apparatus. A blur amount difference calculating means for obtaining a difference distribution with respect to the maximum blur amount between the plurality of parallax images from the blur amount distribution;
The image according to claim 1, wherein the correction unit corrects a blur amount of the parallax image in accordance with a difference in the difference distribution, thereby suppressing a difference in blur amount with another parallax image. Processing equipment.   The blur amount calculation means obtains the effective aperture value according to the size of the overlap between the pupil intensity distribution of the sub-pixel and the exit pupil of the imaging optical system, thereby obtaining the effective aperture value and the image height. The image processing apparatus according to claim 1, wherein the relationship is obtained.   The correction unit increases the blur amount for an area where the blur amount is smaller than that of the other parallax image, and decreases the blur amount for an area where the blur amount is larger than the other parallax image. The image processing apparatus according to claim 1, wherein a difference in blur amount with respect to the parallax image is suppressed.   The correction unit increases the blur amount for a region where the blur amount is smaller than that of the other parallax image, and decreases the blur amount for a region where the blur amount is greater than a predetermined amount than the other parallax image. The image processing apparatus according to claim 1, wherein a difference in blur amount from another parallax image is suppressed. Furthermore, it has means for generating point image data for each of the plurality of parallax images,
Wherein the correction means, when increasing the blur amount applies the convolution processing using the point image data, applying the deconvolution processing using the point image data when reducing the blur amount The image processing apparatus according to claim 1, wherein a difference in blur amount from another parallax image is suppressed. The imaging element;
The image processing apparatus according to any one of claims 1 to 8,
Means for acquiring information of the imaging optical system;
An imaging device comprising: A control method for execution by a control means of an image processing apparatus,
An imaging device in which a plurality of pixels are arranged, and each of the pixels is obtained from an imaging device provided with a plurality of sub-pixels that receive light beams that pass through different pupil partial regions of the imaging optical system. Obtaining an input image to be obtained;
Generating a plurality of parallax images from the input image by generating a parallax image based on a signal of a sub-pixel that receives a light beam passing through the same pupil partial region; and
Obtaining a defocus amount distribution from the plurality of parallax images;
Obtaining a blur amount distribution for each of the plurality of parallax images from the defocus amount distribution, an effective aperture value of an exit pupil of the imaging optical system for each of the plurality of sub-pixels, and an image height;
Applying a correction process that suppresses a difference in blur amount from another parallax image for each of the plurality of parallax images;
A control method for an image processing apparatus, comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 8.