JP6800648B2 - Image processing device and its control method, program and imaging device - Google Patents

Description

The present invention relates to an image processing device, a control method thereof, a program, and an image pickup device.

As one of the focus detection methods of an image pickup device, an imaging surface phase difference method is known in which focus detection is performed by a phase difference method using a signal obtained by an image sensor. Patent Document 1 discloses an image pickup device that uses a two-dimensional image pickup element in which one microlens and a plurality of photoelectric conversion units are formed for one pixel. Since the photoelectric conversion unit divided into a plurality of portions is configured to receive the light flux that has passed through different regions of the exit pupil of the photographing lens via one microlens, so-called pupil division can be realized. Since the amount of image shift can be calculated from the parallax between a plurality of viewpoint signals obtained by pupil division, it is possible to perform focus detection by the phase difference method by converting the obtained amount of image shift into the amount of defocus. it can. Further, Patent Document 2 discloses that an imaging signal is generated by adding a plurality of viewpoint signals received by a plurality of photoelectric conversion units.

U.S. Pat. No. 4,410,804 Japanese Unexamined Patent Publication No. 2001-083407

By the way, in shooting using a shooting device, a shooting expression that makes the main subject stand out effectively may be used by focusing on the main subject and greatly blurring the foreground and the background. As a technique capable of changing the focus state after shooting, a refocus technique for generating a composite image in which the focus state is changed by using a plurality of viewpoint signals obtained by the above-mentioned pupil division is known. However, the refocusing technique is a technique for changing the focus state of the entire composite image by changing the virtual focusing position, and the degree of focusing or blurring of other areas with respect to a predetermined area in the image. Is not more emphasized. Therefore, a technique capable of adjusting the effect of making the in-focus state of another region such as a subject stand out by comparing with a predetermined region such as a blurred foreground or background by using a plurality of viewpoint images obtained by the above-mentioned pupil division. Is desired.

An object of the present invention has been made in view of the above problems. That is, an image processing device, a control method thereof, a program, and an imaging device capable of generating an image in which the focusing state of another region is changed by comparison with a predetermined region in the image using a plurality of viewpoint images after shooting. The purpose is to provide.

In order to solve this problem, for example, the image processing apparatus of the present invention has the following configuration. That is, an acquisition means for acquiring an image signal obtained by an image pickup element in which a plurality of unit pixels provided with a plurality of sub-pixels for receiving light beams passing through different pupil region regions of an imaging optical system are arranged, and each unit. Indicates the image shift between the first viewpoint image and the second viewpoint image in which signals obtained from the first predetermined position of the pixel and the sub-pixels at the second predetermined position are selected from the image signals, respectively. Based on the detection means that detects the amount of image shift for each region and the amount of image shift for each detected region, the first viewpoint image and the second viewpoint image are combined at different ratios for each region. possess a processing means for generating an image obtained by changing the depth, and processing means, each of the weights for combining the first viewpoint image and the second viewpoint image, the image shift for each detected region The setting means that is set for each area based on the amount and the first viewpoint image and the second viewpoint image are added based on the respective weights for each set area to generate an image with a changed depth. The setting means has, and the setting means has a difference between the amount of image shift for each detected region and the amount of image shift in a predetermined region of the imaged subject, as compared with the first region. It is characterized in that it is set so that the difference in weighting becomes large in the small second region .

According to the present invention, it is possible to generate an image in which the focusing state of another region is changed by comparison with a predetermined region in the image by using a plurality of viewpoint images after shooting.

A block diagram showing a functional configuration example of a digital camera as an example of the image processing device according to the first embodiment. The figure which shows schematic the pixel array in 1st Embodiment Schematic plan view (A) and schematic cross-sectional view (B) of pixels in the first embodiment The figure which shows the relationship between the pixel and pupil division in the 1st Embodiment schematicly. The figure which shows the example of the light intensity distribution inside a pixel in 1st Embodiment The figure which shows the example of the pupil intensity distribution in 1st Embodiment The figure which shows roughly the relationship between the image sensor and pupil division in 1st Embodiment The figure which shows roughly the relationship between the defocus amount of the 1st viewpoint image and the 2nd viewpoint image and the image shift amount in 1st Embodiment. A flowchart showing a series of operations related to the depth correction process in the first embodiment. The figure explaining the shading by the pupil deviation of the 1st viewpoint image and the 2nd viewpoint image in 1st Embodiment. The figure which illustrates the captured image in 1st Embodiment The figure which illustrates the contrast distribution of the captured image and the viewpoint image in 1st Embodiment The figure which shows the relationship between the parallax and the perspective competition between the viewpoint images in the 1st Embodiment schematicly. The figure which illustrates (A) contrast difference amount distribution, (B) contrast distribution, (C) image deviation amount distribution in 1st Embodiment. The figure which illustrates the image deviation difference amount distribution from the predetermined shift amount in the 1st Embodiment The figure explaining the effect of the process which sharpens the parallax in the 1st Embodiment. The figure which roughly explains the depth correction possible range by the synthesis process in 1st Embodiment The figure which shows schematic the pixel array in 2nd Embodiment Schematic plan view (A) and schematic sectional view (B) of pixels in the second embodiment.

Hereinafter, exemplary embodiments will be described in detail with reference to the drawings. In the following embodiments, the case where the present invention is applied to a digital camera as an example of an image processing device will be described as an example, but the present invention can be applied to any device capable of processing LF data. Any device may include, for example, a personal computer, a tablet, a game machine, a wearable terminal such as a spectacle type or a clock type, an in-vehicle system, a system for a surveillance camera, and a medical device.

(First Embodiment)
FIG. 1 is a block diagram showing a functional configuration example of the digital camera 100 as an example of the image processing device according to the present embodiment. The first lens group 101 arranged at the tip of the photographing lens (imaging optical system) is held by the lens barrel so as to be able to move forward and backward in the optical axis direction. The aperture combined shutter 102 adjusts the amount of light at the time of shooting by adjusting the aperture diameter thereof, and also has a function as a shutter for adjusting the exposure seconds at the time of shooting a still image. The second lens group 103 advances and retreats in the optical axis direction integrally with the diaphragm and shutter 102, and has a magnification-changing action (zoom function) in conjunction with the advancing and retreating operation of the first lens group 101. The third lens group 105 is a focus lens that adjusts the focus by advancing and retreating in the optical axis direction. The optical low-pass filter 106 is an optical element for reducing false colors and moire of captured images. The image sensor 107 includes, for example, a two-dimensional CMOS (complementary metal oxide semiconductor) photosensor and peripheral circuits, and is arranged on the image plane of the imaging optical system. Details of the image sensor 107 will be described later.

The zoom actuator 111 rotates the cam cylinder (not shown) to move the first lens group 101 and the second lens group 103 in the optical axis direction to perform a magnification change operation. The aperture shutter actuator 112 controls the aperture diameter of the shutter 102 that also serves as an aperture to adjust the amount of shooting light, and also controls the exposure time during still image shooting. The focus actuator 114 moves the third lens group 105 in the optical axis direction to perform a focus adjustment operation.

The electronic flash 115 for illuminating a subject includes a flash illuminating device using a xenon tube or a illuminating device including an LED (light emitting diode) that continuously emits light, and is used at the time of photographing. The AF (autofocus) auxiliary light source 116 projects an image of a mask having a predetermined aperture pattern onto the field of view via a projection lens. As a result, the focus detection ability for a low-luminance subject or a low-contrast subject is improved.

The control unit 121 includes, for example, a CPU (Central Processing Unit) and controls the operation of the entire digital camera 100. The control unit 121 includes a calculation unit, a ROM (read only memory), a RAM (random access memory), an A (analog) / D (digital) converter, a D / A converter, a communication interface circuit, and the like. The control unit 121 drives each unit of the digital camera 100 by expanding and executing a predetermined program stored in the ROM in the RAM, and executes a series of operations such as AF control, imaging processing, image processing, and recording processing. To do.

The electronic flash control circuit 122 controls the lighting of the electronic flash 115 in synchronization with the photographing operation in accordance with the control command of the control unit 121. The auxiliary light source drive circuit 123 lights and controls the AF auxiliary light source 116 in synchronization with the focus detection operation in accordance with the control command of the control unit 121. The image pickup element drive circuit 124 controls the image pickup operation of the image pickup element 107, and also A / D converts the acquired image signal and transmits it to the control unit 121. The image processing circuit 125 performs processing such as gamma conversion, color interpolation, and JPEG (Joint Photographic Experts Group) compression on the image signal acquired by the image sensor 107 in accordance with the control command of the control unit 121.

The focus drive circuit 126 drives the focus actuator 114 based on the focus detection result in accordance with the control command of the control unit 121, and moves the third lens group 105 in the optical axis direction to adjust the focus. The aperture shutter drive circuit 128 drives the aperture shutter actuator 112 in accordance with the control command of the control unit 121 to control the aperture diameter of the aperture shutter 102. The zoom drive circuit 129 drives the zoom actuator 111 according to the zoom operation instruction of the photographer in accordance with the control command of the control unit 121.

The display unit 131 includes a display device such as an LCD (liquid crystal display), and includes information on a shooting mode of the camera, a preview image before shooting and a confirmation image after shooting, and a composite image generated by a depth change process described later. Etc. are displayed. The operation unit 132 includes, for example, an operation switch such as a power switch, a release (shooting trigger) switch, a zoom operation switch, and a shooting mode selection switch, and a touch panel, and outputs an operation instruction signal to the control unit 121. The recording medium 133 is a recording medium that can be attached to and detached from the digital camera 100 such as a flash memory, and records captured image data, LF data described later, a generated composite image, and the like.

(Configuration of image sensor)
FIG. 2 shows a schematic diagram of the arrangement of the pixels and the sub-pixels of the image sensor 107 according to the present embodiment. The horizontal direction in FIG. 2 is the x direction (horizontal direction), the vertical direction is the y direction (vertical direction), and the x direction and the direction orthogonal to the y direction (direction perpendicular to the paper surface) are the z direction (optical axis direction). In the example shown in FIG. 2, the pixel (unit pixel) array of the two-dimensional CMOS sensor (image sensor) is shown in the range of 4 columns × 4 rows, and the sub-pixel array is shown in the range of 8 columns × 4 rows.

In the pixel group 200 of 2 columns × 2 rows, for example, the pixel 200R having the spectral sensitivity of R (red) of the first color at the upper left position is the pixel 200G having the spectral sensitivity of G (green) of the second color at the upper left. Is arranged in the upper right and lower left, and the pixel 200B having the spectral sensitivity of B (blue) of the third color is arranged in the lower right. Further, each pixel (unit pixel) is divided into two in the x direction (Nx division) and one division in the y direction (Ny division), and the first sub-pixel 201 has a number of divisions of 2 (division number _NLF = Nx × Ny). It is composed of a plurality of sub-pixels of the second sub-pixel 202 (first sub-pixel to _NLF sub-pixel).

In the example shown in FIG. 2, by arranging 4 columns × 4 rows of pixels (8 columns × 4 rows of sub-pixels) on the imaging surface, the captured image and a plurality of viewpoints having 2 divisions ( _NLF divisions) It is possible to acquire an image signal (LF data) for generating an image. In the image sensor of the present embodiment, the pixel period P is 4 μm (micrometer), and the number of pixels N is 5575 columns (horizontal) x 3725 rows (vertical) = about 20.75 million pixels. Further, the column period _{P S} of the sub-pixel is 2 [mu] m, the sub-pixel number _{N S} horizontal 11150 rows × vertical 3725 lines = from about 41,500,000 pixels.

FIG. 3A shows a plan view of one pixel 200G of the image pickup device 107 shown in FIG. 2 when viewed from the light receiving surface side (+ z side) of the image pickup device 107. The z-axis is set in the direction perpendicular to the paper surface of FIG. 3A, and the front side is defined as the positive direction of the z-axis. Further, the y-axis is set in the vertical direction orthogonal to the z-axis so that the upper side is the positive direction of the y-axis, the x-axis is set in the left-right direction orthogonal to the z-axis and the y-axis, and the right side is the positive direction of the x-axis. Is defined as. FIG. 3 (A) shows a cross-sectional view taken from the −y side along the aa cutting line in FIG. 3 (A).

As shown in FIGS. 3A and 3B, a microlens 305 is formed in the pixel 200G on the light receiving surface side (+ z direction) of each pixel, and the incident light is collected by the microlens 305. Will be done. Further, a plurality of photoelectric conversion units of the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 having two divisions divided into two in the x direction and one division in the y direction are formed, and the first photoelectric conversion unit 301 and the first photoelectric conversion unit 301 are formed. The two photoelectric conversion units 302 correspond to the first sub-pixel 201 and the second sub-pixel 202, respectively. More generally, (x direction Nx divided, the division number N _LF which is Ny divided in the y-direction) a plurality of photoelectric conversion units of the first N _LF photoelectric conversion unit is formed from the first photoelectric conversion unit, the first photoelectric The conversion unit to the N _LF photoelectric conversion unit correspond to the first sub-pixel to the N _LF sub-pixel, respectively.

The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are two independent pn junction photodiodes, and are composed of a p-type well layer 300 and an n-type layer 301 and an n-type layer 302 divided into two. To. If necessary, an intrinsic layer may be sandwiched and formed as a pin structure photodiode. A color filter 306 is formed in each pixel between the microlens 305 and the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302. If necessary, the spectral transmittance of the color filter 306 may be changed for each pixel or each photoelectric conversion unit, or the color filter may be omitted.

The light incident on the pixel 200G is collected by the microlens 305, further separated by the color filter 306, and then received by the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302, respectively. In the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302, electrons and holes are pair-produced according to the amount of light received, and after being separated by the depletion layer, electrons are accumulated. On the other hand, the hole is discharged to the outside of the image sensor 107 through a p-type well layer connected to a constant voltage source (not shown). The electrons accumulated in the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via the transfer gate and converted into a voltage signal.

FIG. 4 schematically shows the correspondence between the pixel structure and the pupil division. FIG. 4 shows a cross-sectional view of the cut surface of the pixel structure shown in FIG. 3A on the aa line when viewed from the + y direction, and the exit pupil surface of the imaging optical system in the −z direction. The figure seen from is shown. In FIG. 4, in order to correspond with the coordinate axes of the exit pupil surface, the x-axis and the y-axis are shown in reverse in the cross-sectional view from the state shown in FIG.

The image sensor 107 is arranged near the image plane of the photographing lens (imaging optical system), and the light flux from the subject passes through the exit pupil 400 of the imaging optical system and is incident on each pixel. The surface on which the image sensor 107 is arranged is defined as the image pickup surface.

The 2 × 1 divided first pupil region 501 and the second pupil region 502 (Nx × Ny divided first pupil region to _NLF pupil region) are respectively the first photoelectric conversion unit 301. The light receiving surface of the second photoelectric conversion unit 302 (from the first photoelectric conversion unit to the _NLF photoelectric conversion unit) and the microlens form an optically conjugated relationship. Further, the first pupil portion region 501 and the second pupil portion region 502 (Nx × Ny-divided first pupil portion region to _NLF pupil portion region) are the first sub-pixel 201 and the second sub-pixel 202 (second sub-pixel 201). It is a pupil region that can receive light from the 1st sub-pixel to the _NLF sub-pixel). The center of gravity of the first pupil region 501 of the first sub-pixel 201 is eccentric to the + X side on the pupil surface, and the second pupil region 502 of the second sub-pixel 202 has the center of gravity on the −X side on the pupil surface. Is eccentric.

Further, in the pupil region 500, the first photoelectric conversion unit 301 divided into 2 × 1 and the second photoelectric conversion unit 302 (from the first photoelectric conversion unit divided into Nx × Ny to the _NLF photoelectric conversion unit) are all combined. The light-receiving surface and the microlens generally form an optically conjugated relationship. Further, the pupil region 500 is a pupil region that can receive light in the entire pixel 200G in which the first sub-pixel 201 and the second sub-pixel 202 (first sub-pixel to _NLF sub-pixel) are all combined.

An example shows a light intensity distribution when light is incident on a microlens formed in each pixel. FIG. 5A shows a light intensity distribution in a cross section parallel to the optical axis of the microlens. Further, FIG. 5B shows the light intensity distribution in the cross section perpendicular to the optical axis of the microlens at the focal position of the microlens. In FIG. 5, H indicates the convex surface of the microlens 305, and f indicates the focal length of the microlens. Further, nFΔ indicates the movable range of the focal position due to refocusing described later, and φ indicates the maximum angle of the incident luminous flux. The incident light is focused on the focal position by the microlens 305. However, due to the influence of diffraction due to the wave nature of light, the diameter of the focused spot cannot be made smaller than the diffraction limit Δ, and has a finite size. The size of the light receiving surface of the photoelectric conversion unit is about 1 to 2 μm, whereas the focusing spot of the microlens is about 1 μm. Therefore, the first pupil region 501 and the second pupil region 502 shown in FIG. 4, which are conjugate to the light receiving surface of the photoelectric conversion unit via the microlens, are clearly divided into pupils due to diffraction blur. Instead, the light receiving rate distribution (pupil intensity distribution) depends on the incident angle of light.

FIG. 6 shows an example of a light receiving rate distribution (pupil intensity distribution) depending on the incident angle of light. The horizontal axis represents the pupil coordinates, and the vertical axis represents the light receiving rate. The graph line L1 shown by a solid line in FIG. 6 represents the pupil intensity distribution along the X axis of the first pupil portion region 501 of FIG. The light receiving rate shown by the graph line L1 rises sharply from the left end, reaches a peak, then gradually decreases, and then the rate of change becomes gradual and reaches the right end. Further, the graph line L2 shown by the broken line in FIG. 6 represents the pupil intensity distribution along the X axis of the second pupil partial region 502. Contrary to the graph line L1, the light receiving rate shown by the graph line L2 rises sharply from the right end, reaches a peak, and then gradually decreases, and then the rate of change becomes gentle and reaches the left end. As shown in the figure, it can be seen that the pupils are gently divided.

Next, the correspondence between the image sensor 107 and the pupil division will be described with reference to FIG. 7. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 (from the first photoelectric conversion unit to the _NLF photoelectric conversion unit) are the first sub-pixel 201 and the second sub-pixel 202 (first sub-pixel to N-th), respectively. _LF sub-pixel). In each pixel of the image sensor 107, the 2 × 1 divided first sub-pixel 201 and the second sub-pixel 202 (Nx × Ny-divided first sub-pixel to _NLF sub-pixel) are respectively imaging optics. It receives light flux that has passed through different pupil region regions of the first pupil region 501 and the second pupil region 502 (from the first pupil region to the _NLF pupil region) of the system. From the signal received by each sub-pixel, an image signal including information on the spatial distribution and the angular distribution of the light intensity is acquired. This image signal is also referred to as LF (Light Field) data.

First, from this image signal (LF data), the first sub-pixel 201 and the second sub-pixel 202 (Nx × Ny-divided first sub-pixel to _NLF sub-pixel) are divided into 2 × 1 for each pixel. A signal of a specific sub-pixel is selected from the above. As a result, a viewpoint image corresponding to a specific pupil region in the first pupil region 501 and the second pupil region 502 (from the first pupil region to the _NLF pupil region) of the imaging optical system is generated. can do. For example, by selecting the signal of the first sub-pixel 201 for each pixel, it is possible to generate a first viewpoint image having a resolution of the number of pixels N corresponding to the first pupil partial region 501 of the imaging optical system. The same applies to other sub-pixels.

Further, regarding the signal of the LF data, the signals of the first sub-pixel 201 and the second sub-pixel 202 (Nx × Ny-divided first sub-pixel to the N- _LF sub-pixel) divided into 2 × 1 are used for each pixel. By synthesizing all of them, it is possible to generate an captured image having a resolution of N pixels. The above-mentioned viewpoint image and captured image will be described as an example in which the control unit 121 acquires LF data from the image sensor 107 and generates the image. However, the viewpoint image and the captured image may be generated inside the image sensor 107, and the image processing circuit 125 may generate the image. LF data may be acquired and generated.

As described above, the image pickup device 107 of the present embodiment has a structure in which a plurality of pixels provided with a plurality of sub-pixels for receiving light flux passing through different pupil region regions of the imaging optical system are arranged, and the LF data. Can be obtained. In the above description, the pupil region is divided into two in the horizontal direction, but the pupil division may be performed in the vertical direction depending on the method of dividing the sub-pixels.

(Relationship between defocus amount and image shift amount)
Further, with reference to FIG. 8, the defocus amount of the first viewpoint image and the second viewpoint image (from the first viewpoint image to the _NLF viewpoint image) that can be generated based on the LF data acquired by the image sensor 107. The relationship with the amount of image shift will be described.

FIG. 8 schematically shows the relationship between the defocus amount of the first viewpoint image and the second viewpoint image and the image shift amount between the first viewpoint image and the second viewpoint image. An image sensor (not shown) is arranged on the image pickup surface 600, and the exit pupils of the imaging optical system are 2 in the first pupil portion region 501 and the second pupil portion region 502 as in the cases of FIGS. 4 and 7. It is divided into × 1.

The defocus amount d has a magnitude | d | representing the distance from the imaging position of the subject image to the imaging surface 600. The orientation is defined as a negative sign (d <0) in the front pin state where the image formation position of the subject image is on the subject side of the imaging surface 600, and as a positive sign (d> 0) in the opposite rear pin state. To do. In the in-focus state where the image formation position of the subject image is on the imaging surface (that is, the in-focus position), d = 0. For example, the position of the subject 801 indicates a position corresponding to the in-focus state (d = 0), and the position of the subject 802 indicates a position corresponding to the front pin state (d <0). Hereinafter, the front pin state (d <0) and the rear pin state (d> 0) are collectively referred to as a defocus state (| d |> 0).

In the front pin state (d <0), among the luminous fluxes received from the subject 802, the luminous flux passing through the first pupil region 501 (or the second pupil region 502) is once focused and then the position of the center of gravity of the luminous flux. It extends to a width Γ1 (or Γ2) centered on G1 (or G2). In this case, a blurred image is obtained on the imaging surface 600. The blurred image is received by the first sub-pixel 201 (or the second sub-pixel 202) constituting each pixel portion arranged in the image sensor 107, and the first viewpoint image (or the second viewpoint image) is generated. Therefore, the first viewpoint image (or the second viewpoint image) is stored as an image signal of a subject image (blurred image) having a width Γ1 (or Γ2) at the center of gravity position G1 (or G2) on the imaging surface 600. Is remembered in. The width Γ1 (or Γ2) of the subject image increases substantially proportionally as the magnitude | d | of the defocus amount d increases. Similarly, if the amount of image shift of the subject image between the first viewpoint image and the second viewpoint image is described as "p", the magnitude | p | increases the magnitude | d | of the defocus amount d. It increases with it. For example, as shown in FIG. 8, the image shift amount p can be defined as the difference “G1-G2” in the position of the center of gravity of the luminous flux, and its magnitude | p | becomes as | d | increases. It increases roughly in proportion. In the rear focus state (d> 0), the image shift direction of the subject image between the first viewpoint image and the second viewpoint image is opposite to that in the front pin state, but | p | is similarly defocused. It tends to be proportional to the amount | d |.

Therefore, in the present embodiment, as the defocus amount of the first viewpoint image and the second viewpoint image or the captured image obtained by adding the first viewpoint image and the second viewpoint image increases or decreases, the first viewpoint image and the first viewpoint image and the second viewpoint image are added. The amount of image shift between the two viewpoint images increases.

(A series of operations related to depth correction processing)
Next, with reference to FIG. 9, a depth correction process for re-correcting the depth of field (also simply referred to as depth) of the captured image will be described. In particular, in the present embodiment, the depth is based on the image shift amount distribution obtained from the relationship between the defocus amount and the image shift amount of the first viewpoint image and the second viewpoint image (from the first viewpoint image to the _NLF viewpoint image). An example of performing correction processing will be described. In this depth correction process, an output image (composite image) in which the depth of the subject in the vicinity of focusing is corrected for each area is generated. The series of operations related to the depth correction process shown in FIG. 9 is started when, for example, the operation unit 132 detects a shooting instruction by the user or selection of LF data previously recorded on the recording medium 133. A series of operations related to the depth correction process is realized by the control unit 121 expanding and executing the program recorded in the ROM (not shown) in the RAM (not shown) and controlling each part such as the image processing circuit 125.

<Generation of viewpoint image and captured image>
In S1, the control unit 121 generates a viewpoint image and a captured image from the input image signal (that is, LF data). For example, the control unit 121 acquires LF data from the image sensor 107, generates a plurality of viewpoint images for each different pupil region of the imaging optical system, and synthesizes an imaging image in which different pupil regions of the imaging optical system are combined. Generate an image.

More specifically, first, the control unit 121 acquires LF data from the image sensor 107. Alternatively, the LF data previously photographed by the image sensor 107 of the present embodiment and stored in the recording medium 133 may be acquired. Next, the control unit 121 generates a first viewpoint image and a second viewpoint image (from the first viewpoint image to the _NLF viewpoint image) for each different pupil region of the imaging optical system. In the following description, the LF data is also simply referred to as LF. Further, in each pixel signal of the LF data, the sub-pixel signal in the column direction i _S (1 ≤ i _S ≤ Nx) th and the row direction j _S (1 ≤ j _S ≤ Ny) th sub-pixel signal is k = Nx (j _S). -1) As + i _S (1 ≦ k ≦ N _LF ), the kth sub-pixel signal is used. The k-th viewpoint image Ik (j, i) corresponding to the k-th pupil partial region of the imaging optical system is generated according to the equation (1) at the i-th column direction and the j-th row direction.

The present embodiment shows an example that bisected the x direction as _{Nx = 2, Ny = 1,} N LF = 2. Further, from the LF data, the control unit 121 further determines a specific sub-pixel of the first sub-pixel 201 and the second sub-pixel 202 (Nx × Ny-divided first sub-pixel to N- _LF sub-pixel) for each pixel. Select the pixel signal. That is, a first viewpoint image or a second viewpoint image (1st viewpoint image or a second viewpoint image) corresponding to a specific pupil part region among the first pupil part region 501 and the second pupil part region 502 (the first pupil part region to the _NLF pupil part region). The _NLF viewpoint image) is generated from the first viewpoint image. Each generated viewpoint image is a Bayer array RGB signal having a resolution of N pixels.

Here, with reference to FIG. 10, shading due to pupil displacement of the first viewpoint image and the second viewpoint image (from the first viewpoint image to the _NLF viewpoint image) will be described. 10 (A) to 10 (C) show the first pupil region 501 received by the first photoelectric conversion unit 301, the second pupil region 502 received by the second photoelectric conversion unit 302, and the ejection of the imaging optical system. It shows the relationship with the pupil 400.

FIG. 10A shows a case where the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the image sensor 107 are the same. In this case, the exit pupil 400 of the imaging optical system is divided into pupils substantially evenly by the first pupil region 501 and the second pupil region 502. On the other hand, FIG. 10B shows a case where the exit pupil distance Dl of the imaging optical system is shorter than the set pupil distance Ds of the image sensor. In this case, at the peripheral image height of the image pickup device 107, the exit pupil of the image pickup optical system and the entrance pupil of the image pickup device are displaced from each other, and the exit pupil 400 of the image pickup optical system is unevenly divided. In the example of FIG. 10B, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is smaller (brighter) than the effective aperture value of the second viewpoint image corresponding to the second pupil region 502. It becomes a value. On the opposite side of the image height (not shown), on the contrary, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is the effective aperture value of the second viewpoint image corresponding to the second pupil region 502. Greater (darker) value.

Similarly, even when the exit pupil distance Dl of the imaging optical system shown in FIG. 10C is longer than the set pupil distance Ds of the image sensor, the peripheral image height of the image sensor 107 is used to image the exit pupil of the image sensor. The pupil of the entrance pupil of the element is displaced. Therefore, even in the case shown in FIG. 10C, the exit pupil 400 of the imaging optical system is unevenly divided into pupils. In the example of FIG. 10C, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is larger than the effective aperture value of the second viewpoint image corresponding to the second pupil region 502 (dark). It becomes a value. On the opposite side of the image height (not shown), on the contrary, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is the effective aperture value of the second viewpoint image corresponding to the second pupil region 502. It will be a smaller (brighter) value.

As described above, as the pupil division becomes non-uniform, the intensities of the first viewpoint image and the second viewpoint image also become non-uniform, and the intensity of either the first viewpoint image or the second viewpoint image becomes large. In addition, shading occurs in which the strength of the other is reduced. And this shading occurs for each pixel, that is, for each RGB.

Therefore, the control unit 121 performs shading correction processing (optical correction processing) (for each RGB) with respect to the first viewpoint image and the second viewpoint image (from the first viewpoint image to the _NLF viewpoint image) as necessary. ) May be performed. Further, if necessary, scratch correction processing, saturation processing, demosiking processing, and the like may be performed.

Further, the control unit 121 generates an image captured by synthesizing different pupil region regions of the imaging optical system. That is, each viewpoint image corresponds to each pupil region, while the captured image corresponds to the pupil region. The control unit 121 generates the captured image I (j, i) at the i-th in the column direction and the j-th in the row direction according to the equation (2).

From the input image signal (LF data), the control unit 121 outputs signals of the first sub-pixel 201 and the second sub-pixel 202 (Nx × Ny-divided first sub-pixel to _NLF sub-pixel) for each pixel. Take out and synthesize. As a result, an captured image which is an RGB signal of a Bayer array having a resolution of N pixels is generated. If necessary, shading correction processing, scratch correction processing, saturation processing, demosiking processing, and the like may be performed on the generated captured image. FIG. 11 shows an example of a captured image generated by the above-mentioned processing and further subjected to the demosaiking processing. A person (doll) is placed in the center, and a fine checkered flat plate is placed on the left side at an angle from the front to the back.

As described above, in S1, a plurality of viewpoint images for each different pupil region are generated from the LF data acquired by using the image sensor 107, and an captured image in which signals of different pupil regions are combined is generated.

<Generation of contrast distribution>
In S2, the control unit 121 generates a contrast distribution in which the high-frequency band component of the spatial frequency is extracted for each region with respect to the captured image and the plurality of viewpoint images generated in S1. The generated contrast distribution is used for adjustment according to the difference between the viewpoint images.

First, the control unit 121 generates an imaging luminance signal Y using the captured images I (j, i) which are RGB signals of the Bayer array. For example, according to the equation (3A), the imaging luminance signal Y is generated so that the color centroids of each RGB match at each position (j, i). Similarly, the k-th viewpoint image Ik (k = 1 to _NLF ), which is an RGB signal of the Bayer array, is used to generate the k-th viewpoint luminance signal Yk according to the equation (3B).

Next, the control unit 121 applies a bandpass filter that extracts high-frequency components of the spatial frequency to the obtained imaging luminance signal Y (j, i) to generate an imaging high-frequency signal dY (j, i). For example, a two-dimensional bandpass filter {F _BPF (j _BPF , i _BPF ) | -n _BPF ≤ j _BPF ≤ n _BPF , -m _BPF ≤ i _BPF ≤ m _BPF } is applied as a buntpass filter, and the imaged high frequency signal dY (J, i) is generated according to the formula (4A). Similarly, by applying the band-pass filter a k viewpoint luminance signal Yk (j, i) to (k = _{1 to N LF),} to generate the k-th viewpoint RF signal dYk the (j, i) according to equation (4B) ..

In this embodiment, a two-dimensional bandpass filter is used, for example, a one-dimensional filter Fx (i _BPF ) in the x direction (pupil division direction) and a one-dimensional filter Fy (j _BPF ) in the y direction (direction orthogonal to the pupil division direction). It is composed by direct product with. That is, the two-dimensional bandpass filter can be expressed as F _BPF (j _BPF , i _BPF ) = Fy (j _BPF ) × Fx (i _BPF ). The one-dimensional filter Fx (i _BPF ) in the x direction, which is the pupil division direction, is for extracting high frequency components of the spatial frequency in the x direction, for example, 0.5 × [1, 2, 0, -2, -1] +1.5 × [1, 0, -2, 0, 1] is configured as a mixing filter. That is, a first-order differential filter [1, 2, 0, -2, -1] and a second-order differential filter [1, 0, -2, 0, 1] are combined. Generally, when the differential filter processing is performed, there is a 0 point in the portion where the positive code changes to the negative code in the filtered signal. Therefore, when combined with the absolute value calculation, a node may occur in the region containing the high frequency component of the spatial frequency (however, the position where the node occurs depends on the derivative order of the differential filter). Therefore, in the present embodiment, for example, the generation of nodes is suppressed by a mixed type filter in which a first-order differential type filter and a second-order differential type filter (generally, differential type filters of different orders) are combined. If necessary, a first-order differential filter such as [1, 2, 0, -2, -1], a second-order differential filter such as [1, 0, -2, 0, 1], and a higher-order filter are used. A differential filter and a more general one-dimensional bandpass filter may be used.

A high-frequency cut (low-pass) filter is used for the one-dimensional filter Fy (j _BPF ) in the y direction, which is orthogonal to the pupil division direction, in order to suppress high-frequency noise in the y-direction. For this high frequency cut (low pass) filter, for example, [1, 1, 1, 1, 1], [1, 4, 6, 4, 1] and the like can be used. Further, if necessary, a bandpass filter process for extracting high frequency components of spatial frequencies may be performed in either the x direction or the y direction. In the present embodiment, a two-dimensional bandpass filter composed of a direct product of two one-dimensional filters has been illustrated, but the present invention is not limited to this, and a general two-dimensional bandpass filter can be used.

Further, the control unit 121 standardizes the image pickup high frequency signal dY (j, i) and generates the standardized image pickup high frequency signal dZ (j, i). For example, with Y ₀ > 0, the imaging high frequency signal dY (j, i) is standardized by the imaging luminance signal Y (j, i) according to the equation (5A). Similarly, the k viewpoint RF signal DYK (j, i) the (k = _{1 to N LF),} the k-th viewpoint luminance signal Yk (j, i) by normalized, the normalized k-th viewpoint RF signal dZk (j , I) are generated according to the equation (5B). In this example, division by 0 is prevented by determining the maximum value with Y ₀ > 0 in the denominator. Further, if necessary, the control unit 121 refers to the imaging luminance signal Y (j, i) and the k-th viewpoint luminance signal Yk (j, i) before standardization in the equations (5A) and (5B). A low-pass filter may be applied to suppress high-frequency noise.

Next, the control unit 121 generates an imaging contrast distribution C (j, i) using the standardized imaging high frequency signal dZ (j, i). For example, the imaging contrast distribution C (j, i) is generated according to the formula (6A) as a low luminance threshold value Ymin, a maximum contrast threshold value Cmax, and an index γ. When the imaging luminance signal Y (j, i) is smaller than the low luminance threshold value Ymin, the value of the imaging contrast distribution C (j, i) is set to 0 (first line of the equation (6A)). On the other hand, when the standardized imaging high frequency signal dZ (j, i) is larger than the maximum contrast threshold Cmax, the value of the imaging contrast distribution C (j, i) is set to 1 (third line of the equation (6A)). In other cases, the imaging contrast distribution C (j, i) is set to a value obtained by normalizing the standardized imaging high frequency signal dZ (j, i) with the maximum contrast threshold Cmax and raising it to the γ power (formula (6A)). 2nd line). As described above, the imaging contrast distribution C (j, i) takes a value within the range of [0,1] (0 or more and 1 or less). When the value of C (j, i) is close to 0, the contrast is low, and when the value is close to 1, the contrast is high. Further, in the present embodiment, a power of γ is applied in order to adjust the tone curve in the range of 0 to 1 of the imaging contrast distribution C (j, i). For example, the index γ is set to 1.5 or more and 2.5 or less so that the tone curve changes gently on the low contrast side and sharply on the high contrast side. If necessary, the composite function F (C (j, i)) is used by using the function F from the domain [0,1] to the range [0,1]: [0,1] → [0,1]. May be used as the imaging contrast distribution. Similarly, the k-th viewpoint contrast distribution Ck (j, i) (k = 1 to _NLF ) is generated according to the equation (6B).

A distribution example of the imaging contrast distribution C (j, i) of the present embodiment is shown in FIG. 12 (A), and a distribution example of the first viewpoint contrast distribution C ₁ (j, i) is shown in FIG. 12 (B). A distribution example of the contrast distribution C ₂ (j, i) is shown in FIG. 12 (C), respectively. In the distribution example shown in FIGS. 12 (A) to 12 (C), the high / low contrast index is represented by the gray scale display in the range [0, 1] shown on the right side of the image. The white part where the value is near 1 has many high frequency components of the spatial frequency in the x direction and indicates a high contrast region, and the black part where the value is near 0 has few high frequency components of the spatial frequency in the x direction. It shows a region with low contrast.

<Perspective competition or occlusion in contrast distribution>
The relationship between the parallax between the plurality of viewpoint images (first viewpoint image and the second viewpoint image) in the present embodiment and the perspective competition and occlusion will be described with reference to FIGS. 13A to 13C. An image sensor 107 (not shown) is arranged on the image pickup surface 600 of FIGS. 13 (A) to 13 (C), and the exit pupil of the imaging optical system is the first pupil as in FIGS. 4, 7, and 8. It is divided into a partial region 501 and a second pupil partial region 502.

FIG. 13A shows an example in which perspective competition occurs in the captured image. That is, the focused image p1 of the subject q1 is photographed by superimposing the blurred image Γ1 + Γ2 of the subject q2 in the foreground, and perspective competition occurs in the captured image. An example of this is shown in FIGS. 13 (B) and 13 (C) separately for a light flux passing through the first pupil region 501 of the imaging optical system and a light flux passing through the second pupil region 502, respectively. ..

In FIG. 13B, the luminous flux from the subject q1 passes through the first pupil portion region 501 and is imaged on the image p1 in the focused state. On the other hand, the luminous flux from the subject q2 in the foreground passes through the first pupil portion region 501, spreads to the blurred image Γ1 in the defocused state, and is received by the sub-pixel 201 of each pixel of the image sensor 107. At this time, the control unit 121 generates a first viewpoint image from the received signal of the sub-pixel 201. That is, in the first viewpoint image, the image p1 of the subject q1 and the blurred image Γ1 of the subject q2 in the foreground do not overlap and are taken at different positions. Therefore, in the first viewpoint image in the example shown in FIG. 13B, perspective competition and occlusion do not occur between the plurality of subjects (subject q1 and subject q2).

On the other hand, in FIG. 13C, the luminous flux from the subject q1 passes through the second pupil portion region 502 and is imaged on the image p1 in the focused state. Further, the luminous flux from the subject q2 in the foreground passes through the second pupil portion region 502, spreads to the blurred image Γ2 in the defocused state, and is received by the sub-pixel 202 of each pixel of the image sensor. At this time, the control unit 121 generates a second viewpoint image from the received signal of the sub-pixel 202. That is, in the second viewpoint image, the image p1 of the subject q1 and the blurred image Γ2 of the subject q2 in the foreground overlap each other. Therefore, in the second viewpoint image in the example shown in FIG. 13C, perspective competition and occlusion occur between a plurality of subjects (subject q1 and subject q2).

As described above, in the vicinity of the region where perspective conflict or occlusion occurs in the captured image, the state in which perspective conflict or occlusion occurs may differ between the first viewpoint image and the second viewpoint image. That is, the examples shown in FIGS. 13A to 13C show that there is a high possibility that the difference between the first viewpoint image and the second viewpoint image becomes large in the vicinity of the region. Therefore, by detecting the region where the difference between the plurality of viewpoint images is large, it is possible to estimate the region where there is a high possibility that perspective competition or occlusion has occurred.

FIG. 14 (A) shows the difference distribution C ₁ (j, i) -C ₂ (j, i) between the first viewpoint contrast distribution C ₁ (j, i) and the second viewpoint contrast distribution C ₂ (j, i). Illustrate to. In the distribution example shown in FIG. 14 (A), the difference between the contrast of the first viewpoint image and the contrast of the second viewpoint image (first viewpoint contrast distribution) in the grayscale display in the range [-1, 1] on the right side. And the difference amount of the second viewpoint contrast distribution). The black portion where the value is in the vicinity of 0 indicates a region where the difference between the contrast of the first viewpoint image and the contrast of the second viewpoint image is small. On the other hand, the white portion where the value is in the vicinity of ± 1 indicates a region where the difference between the contrast of the first viewpoint image and the contrast of the second viewpoint image is large.

In FIG. 14A, in the region of the boundary between the body of the person (doll) and the checkered flat plate in the lower center, the region where the difference between the contrast of the first viewpoint image and the contrast of the second viewpoint image is large (white). Area) has been detected. That is, in the region where the difference between the contrasts is large, it is shown that perspective competition and occlusion occur due to the body of the person and the checkered flat plate. Further, in the region where the difference between the contrasts is large, in addition to the region where perspective competition and occlusion occur, there is a region in which the high frequency band component of the spatial frequency changes significantly between the first viewpoint image and the second viewpoint image. It has been detected. For example, a region where the amount of image shift is large while maintaining high contrast, such as a subject edge portion in a defocused state, is also detected. In the detected region, subject images having significantly different spatial frequency components are captured between the first viewpoint image and the second viewpoint image. As described above, in the captured image in which the first viewpoint image and the second viewpoint image are combined, these detection regions are regions in which a plurality of subject images having significantly different spatial frequency components are mixed.

When image processing such as sharpening or smoothing is strongly performed on a mixed region of a plurality of subject images having different spatial frequency components, the image quality quality may be deteriorated. Therefore, in the present embodiment, the spatial frequency component is determined by using the absolute value | C ₁ (j, i) -C ₂ (j, i) | of the difference amount distribution between the first viewpoint contrast distribution and the second viewpoint contrast distribution. Detects a mixed region of different multiple subject images. As a result, it can be applied while suppressing image processing such as sharpening and smoothing in the detected mixed region. As a result, it is possible to perform sharpening and smoothing image processing while maintaining good image quality quality.

Next, the control unit 121 generates a contrast difference amount distribution _CDIFF (j, i) in order to detect a mixed region of a plurality of subject images having different spatial frequency components. For example, the control unit 121 uses the first viewpoint contrast distribution C ₁ (j, i) and the second viewpoint contrast distribution C ₂ (j, i) to formulate the contrast difference amount distribution C _DIFF (j, i). Generate according to 7A). Further, the control unit 121 multiplies the imaging contrast distribution C (j, i) by the contrast difference amount distribution C _DIFF (j, i) according to the equation (7B) to generate the contrast distribution _MCON (j, i). ..

The contrast difference distribution C _DIFF (j, i) is distributed in the range of [0, 1], and the value becomes 0 in a region where the contrast difference between the viewpoint images is large and the spatial frequency components are often mixed. Get closer. On the other hand, the value approaches 1 in a region where the contrast difference between the viewpoint images is small and the mixture of subject images having different spatial frequency components is small. That is, since the contrast distribution _MCON (j, i) is a distribution obtained by multiplying the imaging contrast distribution C (j, i) by the contrast difference amount distribution _CDIFF (j, i), a plurality of subjects having different spatial frequency components. In the mixed region of the image, the value is suppressed to near 0.

FIG. 14B shows a distribution example of the generated contrast distribution _MCON (j, i). In this distribution example, the grayscale display in the range [0, 1] on the right side represents the index of high and low contrast. The white portion in the vicinity of the value 1 has many high-frequency components of the spatial frequency in the x direction, and indicates a high-contrast region. On the other hand, the black portion where the value is in the vicinity of 0 has few high frequency components of the spatial frequency in the x direction, and indicates a low contrast region. With respect to the imaging contrast distribution C (j, i) shown in FIG. 12A, the contrast value is suppressed in a region where the absolute value of the difference amount distribution of the viewpoint image contrast distribution is large. That is, the absolute value of the difference distribution between the first viewpoint contrast distribution C ₁ (j, i) and the second viewpoint contrast distribution C ₂ (j, i) | C ₁ (j, i) -C ₂ (j, i) The contrast value is suppressed in the region where | is large.

In the above description, a linear function that is monotonically decreasing with respect to the absolute value | C ₁ (j, i) -C ₂ (j, i) | of the difference amount distribution between the first viewpoint contrast distribution and the second viewpoint contrast distribution. Was used. However, more general functions may be used if desired.

As described above, in the present embodiment, the contrast distribution _MCON (j, i) is generated from the captured image and the plurality of viewpoint images according to the difference between the contrasts of each viewpoint image. The contrast distribution M _CON (j, i) is larger in the region where the difference is small than in the region where the difference between the contrasts of each viewpoint image is large, and the spatial frequency component of the captured image is small in the predetermined spatial frequency band. Many areas are larger than areas. Further, the contrast distribution is larger in the region where the brightness is high than in the region where the brightness of the captured image is low. When the repeated contrast distribution M _CON (j, i) is generated, the generated contrast distribution M _CON (j, i) is recorded on a recording medium 133 or the like, and the generation is omitted in the second and subsequent processes. May be good. By doing so, the processing can be speeded up.

<Generation of image shift distribution>
Next, in S3, the control unit 121 generates an image shift amount distribution using the contrast distribution _MCON (j, i), the first viewpoint image, and the second viewpoint image (a plurality of viewpoint images). For example, the control unit 121 correlates the first viewpoint image and the second viewpoint image (the degree of signal matching) at each position (j, i) where the value of the contrast distribution M _CON (j, i) is equal to or greater than a predetermined value. Generates an image shift distribution based on.

Control unit 121, first, the first viewpoint luminance signal Y ₁ generated in accordance with equation (3B), in the pupil division direction (column direction), the first focus detection signal by performing a one-dimensional band-pass filter processing dYA To generate. Further, the second viewpoint luminance signal Y ₂ which is generated according to equation (3B), in the pupil division direction (column direction), the one-dimensional band-pass filter process to generate a second focus detection signal DYB. Examples of the one-dimensional bandpass filter include first-order differential filters [1, 5, 8, 8, 8, 8, 5, 1, -1, -5, -8, -8, -8, -8, -5, -1] and the like can be used. If necessary, the pass band of the one-dimensional bandpass filter may be adjusted.

Next, the control unit 121 determines the image shift amount distribution M _DIS (j, i) at each position (j, i) where the value of the contrast distribution M _CON (j, i) is a predetermined value (for example, 0.2) or more. i) is calculated. That is, the amount of image shift is calculated in a region having high contrast and no perspective competition or occlusion. By doing so, it is possible to improve the accuracy of calculating the image shift amount and speed up the processing. For example, the control unit 121 expresses the degree of signal matching while relatively shifting the first focus detection signal dYA and the second focus detection signal dYB in the pupil division direction at each of the above-mentioned positions (j, i). Calculate the amount of correlation. Then, the image shift amount distribution M _DIS (j, i) when the focus detection signals most match based on the correlation amount is generated. On the other hand, each position (j, i) in which the value of the contrast distribution M _CON (j, i) is less than a predetermined value (for example, 0.2) is excluded from the calculation of the image shift amount.

The process of shifting the focus detection signal and calculating the correlation amount can be performed as follows, for example. First, with the position (j, i) as the center, the row direction j ₂ (-n ₂ ≤ j ₂ ≤ n ₂ ) th, and the column direction i ₂ (-m ₂ ≤ i ₂ ≤ m ₂ ) th in the pupil division direction. Let the first focus detection signal be dYA (j + j ₂ , i + i ₂ ) and the second focus detection signal be dYB (j + j ₂ , i + i ₂ ). Here, the correlation amount COR _EVEN (j, i, s) at each position (j, i) is calculated according to the equation (8A), where the shift amount is s (−n _s ≦ s ≦ n _s ). Similarly, the correlation amount COR _ODD (j, i, s) is calculated according to the formula (8B).

The correlation amount COR _ODD (j, i, s) shifts the shift amount of the first focus detection signal dYA and the second focus detection signal dYB by half phase-1 with respect to the correlation amount COR _EVEN (j, i, s). It is the amount of correlation. From the correlation amount COR _EVEN (j, i, s) and the correlation amount COR _ODD (j, i, s), the shift amount of the real value that minimizes the correlation amount is calculated by subpixel calculation, and the average value. Is calculated to generate the image shift distribution M _DIS (j, i).

The region where the value of the contrast distribution M _CON (j, i) is less than a predetermined value (for example, 0, 2) and is excluded from the calculation of the image shift amount is set to M _DIS (j, i) = 0. .. If necessary, a value other than 0 may be set.

FIG. 14C shows a distribution example of the image shift amount distribution _MDIS (j, i). In this example, the area where the contrast distribution _MCON (j, i) has a predetermined value of 0.2 or more and the amount of image shift is calculated is displayed in grayscale in the range of [-6,6]. it's shown. Note that, in FIG. 14C, the amount of image shift between the first viewpoint image and the second viewpoint image is shown in units of 1 pixel (1 pixel). A minus sign value (value shown in black) represents a region in the front pin state, and a value near 0 indicates a region near in-focus. The plus sign value (value shown in white) indicates the region in the rear pin state. In the distribution example of FIG. 14C, the value of the contrast distribution _MCON (j, i) is less than a predetermined value of 0.2, and is excluded from the calculation of the image shift amount, and _MDIS (j, i) =. The area set to 0 is displayed in black.

As described above, in this embodiment, the contrast distribution M _{CON (j,} i) to calculate the image shift amount distribution M _{DIS (j,} i) in accordance with the high contrast and, in the region where the perspective conflict or occlusion has not occurred , The amount of image shift can be calculated from multiple viewpoint images. The image shift amount distribution M _DIS (j, i) obtained earlier may be recorded on the recording medium 133, and the calculation of the image shift amount distribution M _DIS (j, i) in the second and subsequent times may be omitted. By doing so, the processing time can be shortened. Further, if necessary, the image shift amount distribution _MDIS (j, i) may be converted into a defocus distribution. For example, the control unit 121 applies a conversion coefficient according to the position (j, i), the aperture value of the imaging lens (imaging optical system), the exit pupil distance, and the like to the image shift amount distribution _MDIS (j, i). Then, it can be converted into a defocus amount distribution.

<Generation of image shift difference distribution>
In S4, the control unit 121 determines the image shift difference amount distribution M _DIFF (j, i) based on the image shift amount distribution M _DIS (j, i) and a predetermined image shift amount (also simply referred to as a predetermined image shift amount). To generate. First, the control unit 121 sets a predetermined image shift amount p for which depth correction processing is desired (in a predetermined area of the subject). For example, in the example of the image shift amount distribution _MDIS shown in FIG. 14C, the image shift amount in the region near the eyes of a person (doll) is about 2.5. For example, when the control unit 121 wants to perform depth correction processing on a region near the eyes of a person (doll), the control unit 121 sets a predetermined image shift amount p = 2.5. As will be described later, an image in which the focusing state (depth) of another region is changed is generated by comparison with the region near the eyes in the image. The predetermined image shift amount may be set, for example, by the user specifying a position on the captured image via the operation unit 132.

Next, the control unit 121 sets the image shift difference amount distribution M _DIFF (j, i) based on the image shift amount distribution M _DIS (j, i), the predetermined image shift amount p, and the contrast distribution M _CON (j, i). Is calculated according to the equation (9). However, σ _p > 0.

The image shift difference distribution M _DIFF (j, i) is based on the absolute value of the difference between the image shift distribution M _DIS (j, i) and the predetermined image shift p | M _DIS (j, i) -p |. It is a distribution obtained by multiplying a linear function that decreases monotonically and a contrast distribution _MCON (j, i). Image shift difference weight distribution _M DIFF (j, i) _{is, | M DIS (j, i} ) -p | < positive in _{σ p, | M DIS (j} , i) -p | = σ p at 0, | M It becomes negative when _DIS (j, i) -p |> σ _p . The region excluded from the calculation of the image shift amount because the value of the contrast distribution M _CON (j, i) is less than a predetermined value (for example, 0.2) is M _DIFF (j, i) = (1). − | P | / σ _p ) × M _CON (j, i). However, other values may be set for the excluded area as needed.

For example, FIG. 15 shows a distribution example of the above-mentioned image shift difference amount distribution _MDIFF (j, i). In the region where the image shift amount is calculated because the contrast distribution _MCON value is 0.2 or more, the image shift difference amount is shown in the grayscale display in the range [-1, 1] on the right side. ing. In the white part, which is a plus (+) code, the absolute value of the difference between the image shift amount distribution M _DIS (j, i) and the predetermined image shift amount p | M _DIS (j, i) -p | is small. Moreover, it shows a region with high contrast. In the black part with a minus (-) sign, the absolute value of the difference between the image shift distribution M _DIS (j, i) and the predetermined image shift p | M _DIS (j, i) -p | is large. Moreover, it shows a region with high contrast. In the display example of FIG. 15, since the value of the contrast distribution _MCON (j, i) is less than the predetermined value 0.2, it is excluded from the calculation of the image shift amount, and _MDIFF (j, i) = (1). The area set as − | p | / σ _p ) × M _CON (j, i) is displayed in black.

<Generation of modified viewpoint image>
In S5, the control unit 121 refers to the first viewpoint image and the second viewpoint image (from the first viewpoint image to the _NLF viewpoint image) according to the image shift difference amount distribution _MDIFF (j, i). The first sharpening and the first smoothing process are performed. The control unit 121 generates a first modified viewpoint image and a second modified viewpoint image (from the first modified viewpoint image to the _NLF modified viewpoint image) by the first sharpening and the first smoothing. Specifically, the control unit 121 sets the control unit 121 in a region where the image shift difference amount distribution is 0 or more (M _DIFF (j, i) ≥ 0) with respect to the first viewpoint image and the second viewpoint image (plural viewpoint images). , The difference between the viewpoint images is enlarged to sharpen the parallax (crosstalk correction, first sharpening). On the other hand, for the region where the image shift difference distribution is less than 0 (M _DIFF (j, i) <0), the difference between the viewpoint images is reduced to smooth the parallax (crosstalk, first smoothing). ).

First, for the first viewpoint image and the second viewpoint image (plurality of viewpoint images), a process of enlarging the difference between the viewpoint images to sharpen the parallax, or reducing the difference between the viewpoint images to reduce the parallax. A first intensity parameter k _ct (k _ct ≧ 0) that specifies the strength of the smoothing process is set. Next, the first intensity parameter distribution _Kct (j, i) is set according to the equation (10). In the present embodiment, the first intensity parameter distribution K _ct (j, i) is proportional to the image shift difference amount distribution M _DIFF (j, i) with k _ct as a proportional coefficient.

Further, the control unit 121 refers to the equation (11A) for the first viewpoint image I ₁ (j, i) and the second viewpoint image I ₂ (j, i) (from the first viewpoint image to the _NLF viewpoint image). ) and generates the formula (first modified according 11B) viewpoint image _MI 1 (j, i) and the second corrected viewpoint image _MI 2 (j, i) (the _{N LF} modified viewpoint image from the first corrected viewpoint image) ..

Equation (11A) expands the difference between the first viewpoint image and the second viewpoint image (plurality of viewpoint images) in the region where the first intensity parameter distribution (image shift difference amount distribution) is 0 or more, and parallax. Represents a process of sharpening (crosstalk correction, first sharpening). On the other hand, the equation (11B) reduces the difference between the first viewpoint image and the second viewpoint image (plurality of viewpoint images) in the region where the first intensity parameter distribution (image shift difference amount distribution) is less than 0. It represents a process of smoothing the parallax (crosstalk, first smoothing).

FIG. 16 shows an example of a process of enlarging the difference between the first viewpoint image and the second viewpoint image (a plurality of viewpoint images) to sharpen the parallax. The horizontal axis represents the pixel position, and the vertical axis represents the pixel value (signal level). In FIG. 16, an example of the first viewpoint image (before correction A) and the second viewpoint image (before correction B) before the sharpening process is shown by a broken line graph. Further, an example of the first modified viewpoint image (corrected A) and the second modified viewpoint image (corrected B) after being sharpened according to the equation (11A) is shown by a solid line graph. By applying the process of enlarging the difference between the viewpoint images and sharpening the parallax, the part where the difference between the viewpoint images was large before the process is enlarged, but the difference between the viewpoint images is small before the process. The part does not change much. In this way, it can be seen that the parallax between the viewpoint images is sharpened. On the other hand, by applying the smoothing process in the formula (11B), the difference between the first viewpoint image and the second viewpoint image (plurality of viewpoint images) is reduced, and the parallax between the viewpoint images is smoothed. .. As for the image processing according to the contrast distribution and the image shift amount distribution described above, either a sharpening process or a smoothing process may be applied, if necessary.

As described above, according to the equations (7A) to (7B), the equation (9), the equation (10), the equation (11A) and the equation (11B), the control unit 121 has a region where the difference between the contrasts for each viewpoint image is large. Sharpening and smoothing are strongly applied to each viewpoint image in the region where the difference between contrasts is smaller than that. In addition, sharpening and smoothing are strongly applied to each viewpoint image in a region where the contrast distribution value is larger than the region where the value is small. That is, by using the equations (9), (10), (11A) and (11B), the sharpening process is applied to the region where the difference from the predetermined shift amount of the image shift amount distribution is small. The smoothing process is applied to the region where the difference is large. Further, by using the equations (9), (10), and (11A), sharpening is strongly applied to the region where the difference is smaller than the region where the difference from the predetermined shift amount of the image shift amount distribution is large. On the other hand, by using the equations (9), (10), and (11B), the smoothing is strongly applied to the region where the difference from the predetermined shift amount of the image shift amount distribution is larger than the region where the difference is large.

In this way, from the formulas (11A) and (11B), the difference between the plurality of viewpoint images is enlarged and the parallax is sharpened for each pixel of the plurality of viewpoint images, or the difference between the plurality of viewpoint images is reduced. A process of reducing the size and smoothing the parallax is performed to generate a plurality of modified viewpoint images. The first sharpening process of the formula (11A) and the second smoothing process of the formula (11B) are the first viewpoints corresponding to the first photoelectric conversion unit included in each (j, i) pixel. This is an arithmetic process between the image I ₁ (j, i) and the second viewpoint image I ₂ (j, i) corresponding to the second photoelectric conversion unit.

<Setting of weighting factor>
In S6, the control unit 121 sets a weighting coefficient for each of the first modified viewpoint image and the second modified viewpoint image (from the first modified viewpoint image to the _NLF modified viewpoint image). Specifically, in a region where the value of the image shift amount distribution _MDIS (j, i) is near the predetermined image shift amount p (region where the difference is small), one viewpoint image so as to deepen the depth of field. The weighting coefficient is biased to (that is, the difference in weighting is increased). On the other hand, in the region where the value of the image shift amount distribution _MDIS (j, i) is not in the vicinity of the predetermined image shift amount p (region with a large difference), each viewpoint image is weighted evenly so as to make the depth of field shallow. (That is, reduce the difference in weighting). In the following processing, such a weighting coefficient is set based on the image shift difference amount distribution _MDIFF (j, i).

That is, the control unit 121 formulates the first weighting coefficient distribution W ₁ (j, i) of the first modified viewpoint image MI ₁ (j, i) from the image shift difference amount distribution M _DIFF (j, i). Calculate according to 12A). Further, the second weighting coefficient distribution W ₂ (j, i) of the second modified viewpoint image MI ₂ (j, i) is calculated according to the equation (12B).

As shown in the formulas (12A) and (12B), the difference between the first weight coefficient distribution W ₁ (j, i) and the second weight coefficient distribution W ₂ (j, i) is | W ₁ (j, i). ) -W ₂ (j, i) | = max (M _DIFF (j, i), 0). Therefore, in the present embodiment, the difference between the weighting coefficients is larger in the region where the difference is smaller than in the region where the difference from the predetermined shift amount of the image shift amount distribution is larger | W ₁ (j, i) -W ₂ (j, i) | is large. That is, the depth of field is corrected in the region where the value of the image shift amount distribution _MDIS (j, i) is in the vicinity of the predetermined image shift amount p (so that the difference between the weighting coefficients becomes large). There is. If necessary, the first weight coefficient distribution W ₁ (j, i) and the second weight coefficient distribution W ₂ (j, i) may be smoothed by performing a high-frequency cut (low-pass) filter process. good.

<Generation of output image by compositing processing>
In S7, the control unit 121 performs an addition process (composite process) to generate a depth-corrected output image. Specifically, the weighting coefficient is multiplied for each of the first modified viewpoint image and the second modified viewpoint image (from the first modified viewpoint image to the _NLF modified viewpoint image), and the addition process (composite process) according to the equation (13) is performed. This is performed to generate a depth-corrected output image based on the image shift distribution _MDIS (j, i).

First, the control unit 121 multiplies the first weighting coefficient distribution W ₁ (j, i) of the equation (12A) by the first modified viewpoint image MI ₁ (j, i), and the second weight of the equation (12B). The coefficient distribution W ₂ (j, i) is multiplied by the second modified viewpoint image MI ₂ (j, i). Then, by synthesizing these, the depth of field is corrected in the region where the value of the image shift amount distribution _MDIS (j, i) is near the predetermined image shift amount p. As described above, in the present embodiment, the image shift amount distribution is generated from the plurality of viewpoint images, and the weighting coefficient is changed for each of the plurality of viewpoint images based on the image shift amount distribution (the addition ratio of the plurality of viewpoint images is changed). And combine to generate an output image with a corrected depth of field. When the control unit 121 completes the generation of the output image, the control unit 121 ends a series of operations related to the depth correction process.

(Effect of modified viewpoint image)
Next, after explaining the depth correction range with reference to FIG. 17, the effect when the correction viewpoint image is used in the above-mentioned composition processing will be described. An image sensor 107 (not shown) is arranged on the image pickup surface 600, and the exit pupils of the imaging optical system are the first pupil partial region 501 and the second pupil as in the case of FIGS. 4, 7, and 8. It is divided into 2 × 1 subregions 502.

Assuming that the permissible circle of confusion diameter is δ and the aperture value of the imaging optical system is F, the depth of field at the aperture value F is ± F × δ. On the other hand, the effective aperture value F ₀₁ (or F ₀₂ ) in the pupil division direction (x direction) of the first pupil partial region 501 (or 502) divided into Nx × Ny (for example, 2 × 1) and narrowed. ) Becomes F ₀₁ = Nx × F (or F ₀₂ = Nx × F) and becomes dark. The effective depth of field for each first modified viewpoint image (or second modified viewpoint image) is ± Nx × F × δ, which is Nx times deeper and the focusing range is expanded Nx times. Within the range of the effective depth of field "± Nx × F × δ", a focused subject image is acquired for each first modified viewpoint image (or second modified viewpoint image). Therefore, by setting the weighting coefficient of one modified viewpoint image to 1 (the coefficient of the other modified viewpoint image is 0), the depth of field in the pupil division direction can be made approximately Nx times deeper.

However, as in the example of the pupil intensity distribution shown in FIG. 6, in the pupil division by the microlens having a diameter of several um formed for each pixel portion and the photoelectric conversion portion divided into a plurality of parts, the diffraction blur due to the wave nature of light is used. Because of this, the pupil division is gentle. Therefore, the depth of the pupil division direction (x direction) of the first viewpoint image and the second viewpoint image (plurality of viewpoint images) is not sufficiently deep, and the first viewpoint image and the second viewpoint image (plurality of viewpoint images) are displayed. Even if the depth correction process is performed using the image, the depth correction effect may not be sufficiently obtained.

Therefore, in the depth correction process described above, for the first viewpoint image and the second viewpoint image (plurality of viewpoint images), between the first viewpoint image and the second viewpoint image (plurality of viewpoint images) according to the equation (11A). The difference was enlarged to sharpen the parallax (crosstalk correction, first sharpening). That is, for each pixel in which the first intensity parameter distribution (image shift difference amount distribution) is 0 or more ( _Kct (j, i) = _kct × M _DIFF (j, i) ≧ 0), the first viewpoint image and Image processing is applied to sharpen the parallax by expanding the difference between the second viewpoint images (plural viewpoint images). As a result, even if there is diffraction blur, the effective aperture value F in the pupil division direction (x direction) of the generated first modified viewpoint image and the second modified viewpoint image (multiple modified viewpoint images) is increased (depth). Can be corrected (deeply), and the depth correction effect can be improved.

As described above, in the present embodiment, an image shift amount distribution is generated for a plurality of viewpoint images obtained from signals that have passed through different pupil region regions, and a weighting coefficient for each viewpoint image based on the image shift amount distribution is used. , Multiple viewpoint images are combined (output image is generated). That is, when the difference between the image shift amount distribution and the predetermined shift amount is small, the weighting coefficient is calculated so as to bias the weight to one viewpoint image, and a plurality of viewpoint images are combined to deepen the depth. On the other hand, when the difference between the image shift amount distribution and the predetermined shift amount is large, the weighting coefficient is calculated so that the weights of the plurality of viewpoint images are equal, and the plurality of viewpoint images are combined to make the depth shallow. By doing so, it is possible to adjust the effect of making the subject in the vicinity of focus stand out by contrasting with the blurred foreground or background (improving the depth correction effect of the composite image) after shooting. In other words, using a plurality of viewpoint images, it is possible to generate an image in which the focusing state of another region is changed by comparison with a predetermined region in the image after shooting.

(Second Embodiment)
Next, the second embodiment will be described. In the first embodiment, the image sensor 107 has a unit pixel having a first photoelectric conversion unit 301 and a second photoelectric conversion unit 302 having two divisions divided into two in the x direction and one in the y direction. The case has been described as an example. On the other hand, the second embodiment is different in that an image sensor having a structure in which the photoelectric conversion unit in the unit pixel is divided into two in the y direction is used. Therefore, the configuration of the digital camera 100 is the same as that of the first embodiment. Therefore, the same configurations and processes are given the same reference numerals, detailed description thereof will be omitted, and differences will be mainly described.

FIG. 18 shows a schematic diagram of the arrangement of the pixels and the sub-pixels of the image sensor 107 in the present embodiment. The definitions of the x direction (horizontal direction), the y direction (vertical direction), and the z direction (optical axis direction) shown in FIG. 18 are the same as those of FIG. 2 shown in the first embodiment. FIG. 18 shows the pixel arrangement of the image sensor 107 according to the present embodiment in a range of 4 columns × 4 rows (for the sub-pixel arrangement, a range of 8 columns × 8 rows). In the pixel group 200 of 2 columns × 2 rows, the pixel 200R having the spectral sensitivity of the first color R (red) is in the upper left position, and the pixel 200G having the spectral sensitivity of the second color G (green) is in the upper left position. Pixels 200B having a third color B (blue) spectral sensitivity are arranged in the upper right and lower left in the lower right. Furthermore, each pixel is divided into two (Nx divided) in the x direction, divided into two in the y direction (Ny divided) the first fourth subpixels 201 of the split number 4 (division number _N LF = Nx × Ny) sub It is composed of a plurality of sub-pixels of pixel 204 (first sub-pixel to _NLF sub-pixel).

In the example shown in FIG. 18, by arranging a large number of pixels of 4 columns × 4 rows (sub-pixels of 8 columns × 8 rows) on the surface, a captured image and a plurality of viewpoints having 4 divisions ( _NLF divisions) It is possible to acquire an input image signal for generating an image. In the image sensor of the present embodiment, the pixel period P is 4 μm (micrometer), and the number of pixels N is 5575 columns (horizontal) x 3725 rows (vertical) = about 20.75 million pixels. Also, the period _{P S} of the sub-pixel is 2 [mu] m, the sub-pixel number _{N S} horizontal 11150 rows × vertical 7450 lines = from about 83 million pixels.

FIG. 19A shows a plan view of one pixel 200G shown in FIG. 18 when viewed from the light receiving surface side (+ z side) of the image sensor. The definitions of the x-axis, y-axis, and z-axis in FIG. 19A are the same as those in the first embodiment. Further, FIG. 19 (B) shows a cross-sectional view of FIG. 19 (A) when viewed from the −y side along the aa cutting line.

As shown in FIGS. 19A and 19B, the pixel 200G is formed with a microlens 305 for condensing incident light on the light receiving surface side (+ z direction) of each pixel. Additionally, bisected (Nx divided) in the x direction, divided into two (Ny divided) were divided number 4 (division number _{N LF)} fourth from the first photoelectric converter 301 of the photoelectric conversion unit 304 (first photoelectric the y-direction A plurality of photoelectric conversion units (from the conversion unit to the _NLF photoelectric conversion unit) are formed. The first photoelectric conversion unit 301 to the fourth photoelectric conversion unit 304 (the first photoelectric conversion unit to the _NLF photoelectric conversion unit) are the first sub-pixel 201 to the fourth sub-pixel 204 (the first sub-pixel to the Nth sub-pixel), respectively. _LF sub-pixel).

(Multi-viewpoint image and captured image)
In S1 shown in FIG. 9, the control unit 121 generates a plurality of viewpoint images for each different pupil region of the imaging optical system from the LF data acquired by the image sensor 107 of the present embodiment, and forms an image. A captured image corresponding to the pupil region obtained by synthesizing the pupil region regions having different optical systems is generated. Alternatively, the control unit 121 may use the LF data previously photographed by the image sensor 107 of the present embodiment and stored in the recording medium.

Next, the control unit 121 generates a viewpoint image (from the first viewpoint image to the _NLF viewpoint image) from the signals of the first sub-pixel 201 to the fourth sub-pixel 204 for each different pupil region of the imaging optical system. To do. At this time, the sub-pixel signal in the column direction i _S (1 ≦ i _S ≦ Nx) th and row direction j _S (1 ≦ j _S ≦ Ny) th in each pixel signal of LF is set to k = Nx (j _S). -1) As + i _S (1 ≦ k ≦ N _LF ), the kth sub-pixel signal is used. The k-th viewpoint image Ik (j, i) corresponding to the k-th pupil partial region of the imaging optical system at the i-th column direction and the j-th row direction is generated according to the above equation (1).

This embodiment is an example of four divisions of Nx = 2, Ny = 2, and _NLF = 4, and a signal of a specific sub-pixel is selected for each pixel from the LF data corresponding to the pixel arrangement shown in FIG. To do. That is, a signal of a specific sub-pixel is output for each pixel from the four-divided first sub-pixel 201 to the fourth sub-pixel 204 (Nx × Ny-divided first sub-pixel to _NLF sub-pixel). select. In the example of FIG. 18, the control unit 121 is a Bayer array RGB signal having a resolution of the number of pixels N corresponding to a specific pupil region in the first pupil region to the fourth pupil region of the imaging optical system. A fourth viewpoint image is generated from a certain first viewpoint image.

Further, the control unit 121 generates an image captured by synthesizing the luminous flux passing through different pupil region regions of the imaging optical system. That is, the i-th captured image I (j, i) in the column direction and the j-th captured image I (j, i) in the row direction are generated according to the equation (2).

In this embodiment, in the example of four divisions of Nx = 2, Ny = 2, and _NLF = 4, the signals of all the sub-pixels are acquired and synthesized for each pixel from the above-mentioned LF data. That is, the control unit 121 synthesizes all the signals of the first sub-pixel 201 to the fourth sub-pixel 204 divided into four, and generates an captured image which is a Bayer array RGB signal having a resolution of the number of pixels N.

The series of operations after S2 shown in FIG. 9 is the same as that of the first embodiment. Therefore, the control unit 121 performs the processes of S2 to S7 to generate an output image, and ends a series of operations related to the depth correction process, as in the first embodiment. By doing so, it is possible to adjust the effect of making the subject in the vicinity of focus stand out by contrasting with the blurred foreground or background (improving the depth correction effect of the composite image) after shooting.

In the above-described embodiment, the case where the number of divisions of the photoelectric conversion unit in each pixel portion of the image sensor is set to 4 has been described as an example. However, an embodiment in which the number of divisions of the photoelectric conversion unit in each pixel unit is further increased (for example, 9 divisions of Nx = 3, Ny = 3, _NLF = 9 and Nx = 4, Ny = 4, _NLF = 16). 16 divisions, etc.) are also possible.

As described above, in the present embodiment, an image shift amount distribution is generated for four viewpoint images obtained from sub-pixels that divide each unit pixel into four, and a weighting coefficient for each viewpoint image based on the image shift amount distribution is used. I tried to combine multiple viewpoint images. That is, as in the first embodiment, when the difference between the image shift amount distribution and the predetermined shift amount is small, the weighting coefficient is calculated so as to bias the weight to one viewpoint image, and a plurality of viewpoint images are combined. Increase the depth. On the other hand, when the difference between the image shift amount distribution and the predetermined shift amount is large, the weighting coefficient is calculated so that the weights of the plurality of viewpoint images are equal, and the plurality of viewpoint images are combined to make the depth shallow. By doing so, it is possible to adjust the effect of making the subject in the vicinity of focus stand out by contrasting with the blurred foreground or background (improving the depth correction effect of the composite image) after shooting. In other words, using a plurality of viewpoint images, it is possible to generate an image in which the focusing state of another region is changed by comparison with a predetermined region in the image after shooting.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

107 ... image sensor, 121 ... control unit, 125 ... image processing circuit

Claims (8)

An acquisition means for acquiring an image signal obtained by an image pickup device in which a plurality of unit pixels provided with a plurality of sub-pixels for receiving a light flux passing through different pupil region regions of an imaging optical system are arranged.
Between the first viewpoint image and the second viewpoint image in which signals obtained from the first predetermined position of each unit pixel and the sub-pixels at the second predetermined position are selected from the image signals, respectively. A detection means that detects the amount of image shift indicating image shift for each region, and
A processing means for generating an image in which the depth is changed by synthesizing the first viewpoint image and the second viewpoint image at different ratios for each region based on the detected image shift amount for each region. and, the possess,
The processing means
A setting means for setting each weight for synthesizing the first viewpoint image and the second viewpoint image for each region based on the amount of image shift for each detected region.
It has a generation means for generating an image having a changed depth by adding the first viewpoint image and the second viewpoint image based on the respective weights of each set region.
The setting means sets each of the weights in a second region where the difference between the amount of image shift for each detected region and the amount of image shift in a predetermined region of the captured subject is smaller than that of the first region. An image processing device characterized in that the difference in weighting is set to be large . An acquisition means for acquiring an image signal obtained by an image pickup device in which a plurality of unit pixels provided with a plurality of sub-pixels for receiving a light flux passing through different pupil region regions of an imaging optical system are arranged.
Between the first viewpoint image and the second viewpoint image in which signals obtained from the first predetermined position of each unit pixel and the sub-pixels at the second predetermined position are selected from the image signals, respectively. A detection means that detects the amount of image shift indicating image shift for each region, and
A processing means for generating an image in which the depth is changed by synthesizing the first viewpoint image and the second viewpoint image at different ratios for each region based on the detected image shift amount for each region. And have
The processing means
A setting means for setting each weight for synthesizing the first viewpoint image and the second viewpoint image for each region based on the amount of image shift for each detected region.
It has a generation means for generating an image having a changed depth by adding the first viewpoint image and the second viewpoint image based on the respective weights of each set region.
The processing means, than the first region, in the region a small difference between the image shift amount in the predetermined area of the image shift amount and the imaged object for each detected region, the said first viewpoint image It further has a correction means for modifying the first viewpoint image and the second viewpoint image so that the difference in parallax from the second viewpoint image becomes large.
The generation unit adds the second viewpoint image and the first view image in which the parallax is corrected by said correction means, for generating a modified image depth, features and images treatment that apparatus. An acquisition means for acquiring an image signal obtained by an image pickup device in which a plurality of unit pixels provided with a plurality of sub-pixels for receiving a light flux passing through different pupil region regions of an imaging optical system are arranged.
Between the first viewpoint image and the second viewpoint image in which signals obtained from the first predetermined position of each unit pixel and the sub-pixels at the second predetermined position are selected from the image signals, respectively. A detection means that detects the amount of image shift indicating image shift for each region, and
A processing means for generating an image in which the depth is changed by synthesizing the first viewpoint image and the second viewpoint image at different ratios for each region based on the detected image shift amount for each region. And have
The processing means, the difference in contrast between the previous SL the first viewpoint image second viewpoint image is generated an image changing the depth in the small area than a predetermined value, characterized in that images processing device. The method according to any one of claims 1 to 3 , wherein the detection means detects the amount of image shift for each region except for a region where a plurality of subjects having different spatial frequency components are mixed. Image processing device. An image sensor in which a plurality of unit pixels provided with a plurality of sub-pixels for receiving a luminous flux passing through different pupil region regions of the imaging optical system are arranged, and an image sensor.
An image pickup apparatus comprising the image processing apparatus according to any one of claims 1 to 4 . An image signal is acquired by arranging a plurality of unit pixels provided with a plurality of sub-pixels that receive light flux passing through different pupil region regions of the imaging optical system, and the image signal is obtained with the first predetermined position of each unit pixel. An image sensor that outputs a first viewpoint image and a second viewpoint image obtained by selecting signals obtained from sub-pixels at a second predetermined position from the image signals, respectively.
A detection means for detecting the amount of image shift indicating image shift between the first viewpoint image and the second viewpoint image for each region.
A processing means for generating an image in which the depth is changed by synthesizing the first viewpoint image and the second viewpoint image at different ratios for each region based on the detected image shift amount for each region. and, the possess,
The processing means
A setting means for setting each weight for synthesizing the first viewpoint image and the second viewpoint image for each region based on the amount of image shift for each detected region.
It has a generation means for generating an image having a changed depth by adding the first viewpoint image and the second viewpoint image based on the respective weights of each set region.
The setting means sets each of the weights in a second region where the difference between the amount of image shift for each detected region and the amount of image shift in a predetermined region of the captured subject is smaller than that of the first region. An imaging device characterized in that the difference in weighting is set to be large . An acquisition step in which the acquisition means acquires an image signal obtained by an image pickup device in which a plurality of unit pixels provided with a plurality of sub-pixels for receiving light flux passing through different pupil region regions of the imaging optical system are arranged.
A first viewpoint image and a second viewpoint image in which signals obtained from the first predetermined position and the second predetermined position sub-pixels of the unit pixels are selected from the image signals by the detection means, respectively. A detection step that detects the amount of image shift indicating image shift between regions for each region,
Based on the amount of image shift for each detected region, the processing means synthesizes the first viewpoint image and the second viewpoint image at different ratios for each region, and obtains an image whose depth is changed. a processing step to be generated, was closed,
In the processing step,
A setting step of setting each weight for synthesizing the first viewpoint image and the second viewpoint image for each region based on the amount of image shift for each detected region.
It has a generation step of adding the first viewpoint image and the second viewpoint image based on the respective weights of each set region to generate an image having a changed depth.
In the setting step, each of the weights is applied in the second region where the difference between the image shift amount for each detected region and the image shift amount in the predetermined region of the imaged subject is smaller than that in the first region. , A control method of an image processing apparatus, characterized in that the difference in weighting is set to be large . A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 4 .