JP6789833B2 - Image processing equipment, imaging equipment, image processing methods and programs - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an image processing method and a program.

Conventionally, an imaging device has been proposed in which the exit pupil of a photographing lens is divided into a plurality of regions and a plurality of viewpoint images corresponding to the divided pupil regions can be simultaneously captured.
Patent Document 1 discloses an image pickup device using a two-dimensional image pickup element in which one microlens and a plurality of photoelectric conversion units are formed for one pixel. The photoelectric conversion unit divided into a plurality of parts is configured to receive light from different pupil region regions of the exit pupil of the photographing lens via one microlens, and performs pupil division. From each signal received by the photoelectric conversion unit divided into a plurality of parts, it is possible to generate a plurality of viewpoint images according to the divided pupil region. Patent Document 2 discloses an imaging device that generates an captured image by adding all the signals received by the divided photoelectric conversion unit.

The plurality of captured viewpoint signals are equivalent to the light field (Light Field) data which is the spatial distribution and the angular distribution information of the light intensity. Non-Patent Document 1 discloses a refocusing technique for changing the in-focus position of an captured image after shooting by synthesizing an image on a virtual image plane different from the image plane using the acquired light field data. There is.

U.S. Pat. No. 4,410,804 Japanese Unexamined Patent Publication No. 2001-083407

Aaron Isaksen, Leonard McMillan, Stephen J. et al. Görtler, "Dynamicly reparated limited graphics", SIGGRAPH '00 Proceedings of the 27th annual conference on Computer graphics, ent. 297-306

However, when a composite image in which the viewpoint is moved is generated, the weighting of a plurality of viewpoint images is changed, so that the blurred shape of a part of the composite image is deformed from a perfect circle, which is an image not intended by the user. Changes may occur.

An object of the present invention is to provide an image processing device that enables image processing intended by a user for each area when a composite image is generated from a plurality of viewpoint images.

In order to solve the above problems, the image processing apparatus of the present invention has an acquisition means for acquiring a plurality of viewpoint images and an image process for generating a composite image by performing image processing on image data based on the plurality of viewpoint images. It includes means, a designating means for designating an area for performing the image processing by the image processing means, and an adjusting means for setting an adjustable range of the image processing for each of the areas designated by the designated means.

According to the present invention, it is possible to provide an image processing device that enables image processing intended by the user for each area when a composite image is generated from a plurality of viewpoint images.

It is a block diagram which shows the structural example of the image pickup apparatus. It is a block diagram which shows the structural example of an image processing apparatus. It is a schematic diagram of a pixel array. It is a schematic plan view and a schematic sectional view of a pixel. It is the schematic explanatory drawing of the pixel and pupil division. It is a figure which shows the example of the light intensity distribution inside a pixel. It is a figure which shows the example of the pupil intensity distribution. It is a schematic explanatory drawing of an image sensor and pupil division. It is a schematic relationship diagram of the defocus amount and the image shift amount of the 1st viewpoint image and the 2nd viewpoint image. It is a schematic explanatory view of viewpoint movement. It is the schematic explanatory drawing of the pupil deviation at the peripheral image height of an image sensor. It is a figure which shows the example of the viewpoint image. It is a schematic diagram of a viewpoint image and a user interface. This is the main flowchart. It is a sub-flow chart of the viewpoint change process. It is a sub-flow chart of the development process.

Hereinafter, embodiments for carrying out the present invention will be described with reference to drawings and the like. In the following embodiments, a case where the device is applied to an image pickup device such as a digital camera will be described, but the present embodiment can be widely applied to an image processing device, an information processing device, an electronic device, or the like that executes image processing according to the present invention.

FIG. 1 shows a block diagram showing a configuration example of an image pickup device having an image pickup device. The image processing device 300 may be provided in the image pickup device, or may exist independently of the image pickup device.
The details of the image pickup apparatus will be described. The first lens group 101 arranged at the tip of the imaging optical system (imaging optical system) is held by the lens barrel so as to be able to move forward and backward in the optical axis direction. The aperture combined shutter 102 adjusts the amount of light at the time of shooting by adjusting the aperture diameter thereof, and also has a function as a shutter for adjusting the exposure seconds at the time of shooting a still image. The second lens group 103 moves forward and backward in the optical axis direction integrally with the shutter 102 that also serves as an aperture, and has a magnification-changing action (zoom function) in conjunction with the forward / backward operation of the first lens group 101. The third lens group 105 is a focus lens that adjusts the focus by advancing and retreating in the optical axis direction. The optical low-pass filter 106 is an optical element for reducing false color and moire of a captured image. The image sensor 107 is composed of, for example, a two-dimensional CMOS (complementary metal oxide semiconductor) photosensor and peripheral circuits, and is arranged on the image plane of the image pickup optical system. Each pixel of the image sensor 107 of the present embodiment has a plurality of sub-pixels (for example, a first sub-pixel and a second sub-pixel) corresponding to a plurality of photoelectric conversion units, and details of the configuration are shown in FIG. 3 to 5 will be described later.

The zoom actuator 111 rotates the cam cylinder (not shown) to move the first lens group 101 and the second lens group 103 in the optical axis direction to perform a magnification change operation. The aperture shutter actuator 112 controls the aperture diameter of the aperture / shutter 102 to adjust the amount of shooting light, and also controls the exposure time during still image shooting. The focus actuator 114 moves the third lens group 105 in the optical axis direction to perform a focus adjustment operation.

The electronic flash 115 for illuminating the subject is used at the time of photographing, and a flash illuminating device using a xenon tube or a illuminating device provided with an LED (light emitting diode) that continuously emits light is used. The AF (autofocus) auxiliary light source 116 projects an image of a mask having a predetermined aperture pattern onto the field of view via a projection lens. As a result, the focus detection ability for a low-luminance subject or a low-contrast subject is improved.

The CPU (Central Processing Unit) 121 that constitutes the control unit of the camera body has a control center function that controls various controls. The CPU 121 includes a calculation unit, a ROM (read only memory), a RAM (random access memory), an A (analog) / D (digital) converter, a D / A converter, a communication interface circuit, and the like. The CPU 121 drives various circuits in the camera according to a predetermined program stored in the ROM, and executes a series of operations such as AF control, imaging processing, image processing, and recording processing. Further, the CPU 121 may have the function of the image processing device 300 described later.

The electronic flash control circuit 122 controls the lighting of the electronic flash 115 in synchronization with the shooting operation in accordance with the control command of the CPU 121. The auxiliary light source circuit 123 controls the AF auxiliary light source 116 to be turned on in synchronization with the focus detection operation in accordance with the control command of the CPU 121. The image pickup element drive circuit 124 controls the image pickup operation of the image pickup element 107, and A / D-converts the acquired image pickup signal and transmits it to the CPU 121. The image processing circuit 125 performs processing such as gamma conversion, color interpolation, and JPEG (Joint Photographic Experts Group) compression of the image acquired by the image sensor 107 in accordance with the control command of the CPU 121.

The focus drive circuit 126 drives the focus actuator 114 based on the focus detection result in accordance with the control command of the CPU 121, and moves the third lens group 105 in the optical axis direction to adjust the focus. The aperture shutter drive circuit 128 drives the aperture shutter actuator 112 in accordance with a control command of the CPU 121 to control the aperture diameter of the aperture shutter 102. The zoom drive circuit 129 drives the zoom actuator 111 according to the zoom operation instruction of the photographer in accordance with the control command of the CPU 121.

The display unit 131 has a display device such as an LCD (liquid crystal display), and displays information on the shooting mode of the camera, a preview image before shooting and a confirmation image after shooting, an in-focus state display image at the time of focus detection, and the like. To do. The operation unit 132 includes a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, and the like as operation switches, and outputs an operation instruction signal to the CPU 121. The flash memory 133 is a recording medium that can be attached to and detached from the camera body, and records captured image data and the like.

Next, the configuration of the image processing apparatus 300 will be described with reference to FIG. FIG. 2 is a block diagram showing a configuration example of the image processing device 300.
The memory 321 stores image data. The image data to be saved is an captured image and a plurality of viewpoint images that are also used for display on the display unit 131 and recording in the flash memory 133. The captured image is an image (A + B image) in which all the signals of the first sub-pixel and the second sub-pixel are combined. The viewpoint image is, for example, a first viewpoint image (A image) generated by selecting a signal of the first sub-pixel for each pixel. The subtraction unit 322 generates a second viewpoint image (B image) by subtracting the first viewpoint image (A image) from the captured image (A + B image). The memory 321 is, for example, acquired from the image sensor 107 and recorded in the flash memory 133, and the image data is acquired from the flash memory 133.

The shading processing unit 323 corrects the change in the amount of light due to the image heights of the first viewpoint image and the second viewpoint image. The operation unit 324 generates a user interface for adjusting the viewpoint movement by the user, displays it on the display device at the time of non-figure via the output unit 314, and sets the viewpoint movement and focus adjustment by the user in the user interface. Receives the adjustment value of focus). Then, the adjustment value operated by the user is transferred to the viewpoint change processing unit 327. The viewpoint change processing unit 327 performs image processing using a plurality of viewpoint images based on the adjustment value acquired from the operation unit 324. The viewpoint change processing unit 327 generates an image in which the viewpoint is changed or an image in which the depth of field is changed by changing the addition ratio of the viewpoint image by image processing.

The area designation unit 325, which is an area designation means for designating a plurality of areas from an image, enables the user to specify an arbitrary area in the image by operating the UI on the display screen, such as the coordinate position, vertical and horizontal size of the designated area. The coordinate information of is stored and passed to the adjustment unit 326. The adjustment unit 326 receives information on the designated area from the area designation unit 325, and changes the adjustment range of image processing such as changing the viewpoint for each area.

Next, the components of the image processing apparatus 300 that perform the development process will be described. The white balance unit 308 performs a white balance process. Specifically, gain is applied to each color of R, G, and B so that R, G, and B in the white region have the same color. By performing the white balance processing before the demosaiking processing, it is possible to prevent the saturation from becoming higher than the saturation of the false color due to color cast or the like when calculating the saturation, and to prevent erroneous judgment. It becomes.

The demoizing unit 309 interpolates the color mosaic image data of two of the three primary colors missing in each pixel to obtain a color image in which the color image data of R, G, and B are aligned in all the pixels. Generate. Specifically, first, the pixel of interest is interpolated in each of the specified directions using the pixels around it, and then the direction is selected. As a result of the interpolation processing for each pixel, 3 of R, G, and B Generates a color image signal of the primary colors. The gamma conversion unit 310 performs gamma correction processing on the color image data of each pixel to generate basic color image data. The color adjustment unit 311 performs various color adjustment processes such as noise reduction, saturation enhancement, hue correction, and edge enhancement, which are processes for improving the appearance of the image.

The compression unit 312 compresses the color-adjusted color image data by a method such as JPEG to reduce the data size at the time of recording. The recording unit 313 records the image data compressed by the compression unit 312 on a recording medium such as a flash memory. The output unit 314 outputs the generated user interface or image in order to display the user interface or image on the display device at the time of non-figure. In the present embodiment, the image processing has been described as the processing in the image processing device 300, but the image processing device may have the above-mentioned image processing control program separately from the image processing device. In that case, the output unit 314 outputs the user interface and the image to the display unit 131 of the imaging device.

FIG. 3 is a diagram showing a schematic diagram of an arrangement of pixels and sub-pixels of the image sensor. The left-right direction of FIG. 3 is defined as the x-axis direction, the vertical direction is defined as the y-axis direction, and the x-axis direction and the direction orthogonal to the y-axis direction (direction perpendicular to the paper surface) are defined as the z-axis direction. FIG. 3 shows the pixel arrangement of the two-dimensional CMOS sensor (imaging element) of the present embodiment in the range of 4 columns × 4 rows, and the sub-pixel arrangement in the range of 8 columns × 4 rows.

In the pixel 200 of 2 columns × 2 rows shown in FIG. 3, the pixel 200R having the spectral sensitivity of R (red) is in the upper left position, and the pixel 200G having the spectral sensitivity of G (green) is in the upper right and lower left positions. , B (blue) pixel 200B having a spectral sensitivity is arranged at the lower right position. Further, each pixel has a first sub-pixel 201 and a second sub-pixel 202 that are divided into two in the x-axis direction and one in the y-axis direction. In other words, the number of division in the x-direction is denoted by Nx, when the division number in y-direction is denoted by Ny, it denoted the number of divisions _{N LF,} 3 _{Nx = 2, Ny = 1,} N LF = Nx × Ny An example of = 2 is shown. Each of the sub-pixels has a function as a focus detection pixel that outputs a focus detection signal.

In the example shown in FIG. 3, by arranging a large number of pixels of 4 columns × 4 rows (sub-pixels of 8 columns × 4 rows) on the surface, it can be displayed on the display unit 131 or recorded in the flash memory 133. It is possible to acquire a captured image (A + B image) to be used and a signal for generating a plurality of viewpoint images. In the present embodiment, the pixel period P is 4 μm, the number of pixels N is 5575 columns wide × 3725 rows = about 20.75 million pixels, the columnwise period PS of the sub-pixels is 2 μm, and the number of sub-pixels NS is 11150 columns wide × vertical. The description will be given as an image sensor with 3725 lines = about 41.5 million pixels.

FIG. 4A is a plan view of one pixel 200G of the image pickup device shown in FIG. 3 as viewed from the light receiving surface side (+ z side) of the image pickup device. The z-axis is set in the direction perpendicular to the paper surface of FIG. 4A, and the front side is defined as the positive direction of the z-axis. Further, the y-axis is set in the vertical direction orthogonal to the z-axis so that the upper side is the positive direction of the y-axis, the x-axis is set in the horizontal direction orthogonal to the z-axis and the y-axis, and the right side is the positive direction of the x-axis. Is defined as. FIG. 4B is a cross-sectional view taken from the −y side along the aa cutting line of FIG. 4A.

As shown in FIGS. 4 (A) and 4 (B), the pixel 200G is formed with a microlens 305 for condensing incident light on the light receiving surface side (+ z-axis direction) of each pixel. .. Further, a plurality of photoelectric conversion units having a number of divisions of 2 are formed, which are divided into two in the x direction and one in the y direction. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 correspond to the first sub-pixel 201 and the second sub-pixel 202, respectively. The number of divisions of the photoelectric conversion unit (sub-pixel) is not limited to 2. The direction of division is not limited to the x direction, and may be divided in the y direction.

The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are two independent pn junction photodiodes, and are pin structure photodiodes in which an intrinsic layer is sandwiched between a p-type layer and an n-type layer. Further, if necessary, the intrinsic layer may be omitted and a pn junction photodiode may be used. A color filter 306 is formed in each pixel between the microlens 305 and the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302. Further, if necessary, the spectral transmittance of the color filter 306 may be changed for each pixel or each photoelectric conversion unit (for each sub-pixel), or the color filter may be omitted.

The light incident on the pixel 200G is collected by the microlens 305, separated by the color filter 306, and then received by the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302, respectively. In the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302, electrons and holes (holes) are pair-produced according to the amount of light received, and after being separated by the depletion layer, negatively charged electrons are generated in the n-type layer (non-type layer). Accumulated in (shown). On the other hand, the hole is discharged to the outside of the image sensor through a p-type layer connected to a constant voltage source (not shown). The electrons accumulated in the n-type layer (not shown) of the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via the transfer gate and converted into a voltage signal. To.

FIG. 5 is a schematic explanatory view showing a correspondence relationship between the pixel structure and pupil division. FIG. 5 shows a cross-sectional view of the pixel structure shown in FIG. 4 (A) at the aa cutting line seen from the + y side, and the exit pupil surface of the imaging optical system seen from the −z axis direction. The figure is shown. In FIG. 5, the x-axis and the y-axis of the cross-sectional view are inverted with respect to the state shown in FIG. 4 in order to correspond to the coordinate axes of the exit pupil surface. The image pickup element is arranged near the imaging plane of the photographing lens (imaging optical system), and the luminous flux from the subject passes through the exit pupil 400 of the imaging optical system and is incident on each pixel. The surface on which the image sensor is arranged is defined as the image pickup surface.

The first pupil region 501 of the first sub-pixel 201 is substantially optically conjugated to the light receiving surface of the first photoelectric conversion unit 301 whose center of gravity is eccentric in the −x direction by the microlens 305. The first pupil region 501 represents a pupil region that can be received by the first sub-pixel 201. The center of gravity of the first pupil portion region 501 of the first sub-pixel 201 is eccentric to the + X side on the pupil surface.
The second pupil partial region 502 of the second sub-pixel 202 is substantially optically conjugated to the light receiving surface of the second photoelectric conversion unit 302 whose center of gravity is eccentric in the + x direction by the microlens 305. The second pupil region 502 represents a pupil region that can be received by the second sub-pixel 202. The center of gravity of the second pupil partial region 502 of the second sub-pixel 202 is eccentric to the −X side on the pupil surface.

The pupil region 500 has a substantially optically conjugated relationship with the light receiving surface in which the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are all combined, and the microlens 305. The pupil region 500 is a pupil region that can receive light in the entire pixel 200G in which the first sub-pixel 201 and the second sub-pixel 202 are all combined.

FIG. 6 is a diagram showing an example of a light intensity distribution when light is incident on a microlens formed in each pixel. FIG. 6A shows a light intensity distribution in a cross section parallel to the optical axis of the microlens. FIG. 6B shows the light intensity distribution in the cross section perpendicular to the optical axis of the microlens at the focal position of the microlens. The incident light is focused on the focal position by the microlens. However, due to the influence of diffraction due to the wave nature of light, the diameter of the focused spot cannot be made smaller than the diffraction limit Δ, and has a finite size. The size of the light receiving surface of the photoelectric conversion unit is about 1 to 2 μm, whereas the condensing spot of the microlens is about 1 μm. Therefore, the first pupil region 501 and the second pupil region 502 in FIG. 5, which are conjugate to the light receiving surface of the photoelectric conversion unit via the microlens, are not clearly divided into pupils due to diffraction blur. The light receiving rate distribution (pupil intensity distribution) depends on the incident angle of light.

FIG. 7 is a diagram showing an example of a light receiving rate distribution (pupil intensity distribution) depending on the incident angle of light. The horizontal axis represents the pupil coordinates, and the vertical axis represents the light receiving rate. The graph line L1 shown by a solid line in FIG. 7 represents the pupil intensity distribution along the X axis of the first pupil portion region 501 of FIG. The light receiving rate shown by the graph line L1 rises sharply from the left end, reaches a peak, then gradually decreases, and then the rate of change becomes gradual and reaches the right end. Further, the graph line L2 shown by the broken line in FIG. 7 represents the pupil intensity distribution along the X axis of the second pupil partial region 502. The light receiving rate shown by the graph line L2 is opposite to the graph line L1 (symmetrical), rises sharply from the right end, reaches a peak, then gradually decreases, and then the rate of change becomes gentle and moves to the left end. To reach. As shown in the figure, it can be seen that the pupils are gently divided.

FIG. 8 is a schematic view showing the correspondence between the image sensor and the pupil division. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 correspond to the first sub-pixel 201 and the second sub-pixel 202, respectively. In each pixel of the image sensor, the first sub-pixel 201 and the second sub-pixel 202 divided into 2 × 1 are different pupil region regions of the first pupil region 501 and the second pupil region 502 of the imaging optical system, respectively. Receives the light flux that has passed through. That is, the luminous flux that has passed through the different pupil region regions of the first pupil region 501 and the second pupil region 502 is incident on each pixel of the image sensor at a different angle, and the first sub-pixel 201 is divided into 2 × 1. Is received by the second sub-pixel 202, respectively.

By selecting a signal of a specific sub-pixel from the first sub-pixel 201 and the second sub-pixel 202 for each sub-pixel from the signal received by each sub-pixel, the first pupil partial region 501 of the imaging optical system is selected. And a viewpoint image corresponding to a specific pupil region in the second pupil region 502 can be generated. For example, by selecting the signal of the first sub-pixel 201 in each pixel, it is possible to generate a first viewpoint image having a resolution of the number of pixels N corresponding to the first pupil partial region 501 of the imaging optical system. The same applies to other sub-pixels. The image pickup device of the present embodiment has a structure in which a plurality of pixels provided with a plurality of photoelectric conversion units (sub-pixels) for receiving light flux passing through different pupil region regions of the imaging optical system are arranged, and the pupils are different. A plurality of viewpoint images can be generated for each subregion.

In the present embodiment, the first viewpoint image and the second viewpoint image are Bayer array images, respectively. If necessary, the first viewpoint image and the second viewpoint image may be demosized. Further, by adding and reading the signals of the first sub-pixel 201 and the second sub-pixel 202 for each pixel of the image sensor, it is possible to generate an captured image having a resolution of N effective pixels.

Next, the relationship between the defocus amount and the image shift amount of the first viewpoint image and the second viewpoint image acquired by the image sensor of the present embodiment will be described. FIG. 9 is a diagram showing a schematic relationship between the defocus amount of the first viewpoint image and the second viewpoint image and the image shift amount between the first viewpoint image and the second viewpoint image. An image sensor (not shown) is arranged on the image pickup surface 800, and as in the cases of FIGS. 5 and 8, the exit pupils of the imaging optical system are 2 × in the first pupil portion region 501 and the second pupil portion region 502. It is divided into one.

The amount of defocus d has a magnitude | d | representing the distance from the imaging position of the subject image to the imaging surface 800. The front pin state where the image formation position of the subject is on the subject side of the imaging surface is a negative sign (d <0), and the rear pin state where the image formation position of the subject is on the opposite side of the subject from the imaging surface is a positive sign (d> 0). ) To define the orientation. The focusing state in which the imaging position of the subject is on the imaging surface (focusing position) is d = 0. In FIG. 9, the subject 801 shows an example of the position corresponding to the in-focus state (d = 0), and the subject 802 shows an example of the position corresponding to the front pin state (d <0). In the following, the front pin state (d <0) and the rear pin state (d> 0) are collectively referred to as a defocus state (| d |> 0).

In the front pin state (d <0), among the luminous fluxes from the subject 802, the luminous flux passing through the first pupil region 501 (or the second pupil region 502) is focused once and then the position of the center of gravity of the luminous flux. It extends to a width of Γ1 (Γ2) centered on G1 (G2). In this case, a blurred image is obtained on the imaging surface 800. The blurred image is received by the first sub-pixel 201 (or the second sub-pixel 202) constituting each pixel arranged in the image sensor, and the first viewpoint image (or the second viewpoint image) is generated. Therefore, in the first viewpoint image (or the second viewpoint image), the subject 802 is a subject image (blurred image) having a blurred width Γ1 (Γ2) at the center of gravity position G1 (or G2) on the imaging surface 800. Recorded as image data.

The blur width Γ1 (or Γ2) of the subject image increases substantially proportionally as the magnitude | d | of the defocus amount d increases. Similarly, the magnitude of the image shift p (= difference in the position of the center of gravity of the luminous flux G1-G2) of the subject image between the first viewpoint image and the second viewpoint image | p | is also the magnitude of the defocus amount d | d. As | increases, it increases roughly in proportion. In the rear pin state (d> 0), the image shift direction of the subject image between the first viewpoint image and the second viewpoint image is opposite to that in the front pin state, but there is the same tendency.

Therefore, in the present embodiment, as the magnitude of the defocus amount of the first viewpoint image and the second viewpoint image or the image pickup signal obtained by adding the first viewpoint image and the second viewpoint image increases, the first viewpoint image The amount of image shift between the image and the second viewpoint image increases.

Next, the principle of image processing of viewpoint movement will be described with reference to FIG. FIG. 10 is a schematic explanatory view of viewpoint movement. In FIG. 10, the image pickup device (not shown) of the present embodiment is arranged on the image pickup surface 800, and the exit pupil of the imaging optical system is the first pupil portion region 501 as in the cases of FIGS. 5, 8 and 9. And the second pupil partial region 502 are divided into two. The viewpoint movement is performed using a plurality of parallax images acquired by an image sensor having a plurality of photoelectric conversion units. In the present embodiment, the viewpoint is moved by the first viewpoint image and the second viewpoint image, and a composite image is generated.

FIG. 10A shows an example in which the focused image p1 of the main subject q1 is photographed by superimposing the blurred image Γ1 + Γ2 of the subject q2 in the foreground, and perspective conflict (front blurring on the main subject) occurs in the captured image. Is. In this example, the luminous flux passing through the first pupil region 501 of the imaging optical system and the luminous flux passing through the second pupil region 502 are divided into FIGS. 10 (B) and 10 (C), respectively. ).

In FIG. 10B, the luminous flux from the main subject q1 passes through the first pupil portion region 501 and is imaged on the image p1 in the focused state. On the other hand, the luminous flux from the subject q2 in the foreground passes through the first pupil portion region 501, spreads to the blurred image Γ1 in the defocused state, and is received by the first sub-pixel 201 of each pixel of the image sensor. Then, the first viewpoint image is generated from the received signal of the first sub-pixel 201. In the first viewpoint image, the image p1 of the main subject q1 and the blurred image Γ1 of the subject q2 in the foreground do not overlap and are imaged at different positions.

On the other hand, in FIG. 10C, the luminous flux from the main subject q1 passes through the second pupil portion region 502 and is imaged on the image p1 in the focused state. On the other hand, the luminous flux from the subject q2 in the foreground passes through the second pupil portion region 502, spreads to the blurred image Γ2 in the defocused state, and is received by the second sub-pixel 202 of each pixel of the image sensor. Then, a second viewpoint image is generated from the received signal of the second sub-pixel 202. In the second viewpoint image, the image p1 of the main subject q1 and the blurred image Γ2 of the subject q2 in the foreground overlap each other.

In FIGS. 10B and 10C, a region near the image p1 of the main subject q1 is defined as a predetermined region r. Comparing the first viewpoint image generated in FIG. 10 (B) and the second viewpoint image generated in FIG. 10 (C), the blurred image Γ1 of the subject q2 in front of the first viewpoint image is obtained in the predetermined region r. , The range of the blurred image is narrower than the blurred image Γ2 of the subject q2 in front of the second viewpoint image. Further, in the predetermined region r of the first viewpoint image, the blurred image Γ1 is less likely to be captured, and the blurred image Γ1 does not overlap the image p1 of the main subject q1, so that the contrast evaluation value of the predetermined region r becomes large. On the other hand, in the predetermined region r of the second viewpoint image, the blurred image Γ2 is often captured, and the image p1 of the main subject q1 and the blurred image Γ2 are imaged in an overlapping manner, so that the contrast evaluation value of the predetermined region r becomes small.

In the present embodiment, in a predetermined region r of the composite image, among the plurality of viewpoint images, the viewpoint image in which the closest subject is photographed in the widest range has the smallest weight coefficient, or the closest subject has the smallest weight coefficient. The weighting coefficient of the viewpoint image captured in the narrowest range is maximized. In other words, in the present embodiment, in the predetermined region r of the composite image, among the plurality of viewpoint images, the viewpoint image having the smallest contrast evaluation value has the smallest weighting coefficient, or the viewpoint image having the largest contrast evaluation value. Make the weighting factor the largest.

Therefore, in the present embodiment, in the predetermined region r, the first weighting coefficient Wa of the first viewpoint image with less overlap between the image p1 and the blurred image Γ1 is set, and the second viewpoint image with more overlap between the image p1 and the blurred image Γ2 is used. 2 The weight coefficient Wb is set to be larger than the Wb, and a composite image is generated. By moving the viewpoint in this way, it is possible to generate a composite image in which the front blurring on the main subject is reduced in a predetermined area.

Here, the pupil shift at the peripheral image height of the image sensor will be described. FIG. 11 is a schematic explanatory view of pupil deviation at the peripheral image height of the image sensor. Specifically, the first pupil region 501 that receives light from the first photoelectric conversion unit 301 of each pixel arranged at the peripheral image height of the image sensor, the first that receives light from the second photoelectric conversion unit 302, and the imaging optical system. The relationship between the exit pupils 400 and 400 is shown. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 correspond to the first sub-pixel 201 and the second sub-pixel 202, respectively.

FIG. 11A shows a case where the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the image sensor substantially match. In this case, as with the central image height, the exit pupil 400 of the imaging optical system is substantially evenly divided by the first pupil region 501 and the second pupil region 502 in the peripheral image height of the image sensor.

On the other hand, FIG. 11B shows a case where the exit pupil distance Dl of the imaging optical system is shorter than the set pupil distance Ds of the image sensor. In this case, the peripheral image height of the image pickup device causes a pupil shift between the exit pupil of the image pickup optical system and the entrance pupil of the image pickup device, and the first pupil portion region 501 and the second pupil portion region 502 cause the imaging optical system to shift. The exit pupil 400 is unevenly divided into pupils. In the example of FIG. 11B, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is smaller (brighter) than the effective aperture value of the second viewpoint image corresponding to the second pupil region 502. It becomes a value. On the opposite side, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is larger than the effective aperture value of the second viewpoint image corresponding to the second pupil region 502 (dark). ) Value.

FIG. 11C shows a case where the exit pupil distance Dl of the imaging optical system is longer than the set pupil distance Ds of the image sensor. In this case as well, at the peripheral image height of the image sensor, the exit pupil of the imaging optical system and the entrance pupil of the image sensor are displaced from each other at the peripheral image height of the image sensor, and the exit pupil 400 of the image pickup optical system is the first. The pupil is unevenly divided by the pupil region 501 and the second pupil region 502. In the example of FIG. 11C, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is larger than the effective aperture value of the second viewpoint image corresponding to the second pupil region 502 (dark). It becomes a value. On the opposite side, the effective aperture value of the first viewpoint image corresponding to the first pupil region 501 is smaller than the effective aperture value of the second viewpoint image corresponding to the second pupil region 502 (bright). ) Value.

As the pupil division becomes non-uniform due to the peripheral image height due to the pupil shift, the effective F-numbers of the first-view image and the second-view image also become non-uniform. Therefore, either the first-view image or the second-view image The spread of the blur becomes large, and the spread of the other blur becomes small. Therefore, in a predetermined area of the composite image, the weighting coefficient of the viewpoint image having the smallest effective aperture value is the smallest, or the weighting coefficient of the viewpoint image having the largest effective aperture value is the largest among the plurality of viewpoint images. Is desirable. With the above configuration, it is possible to reduce the front blurring on the main subject by performing image processing for moving the viewpoint after shooting.

Next, the depth change process will be described. In FIG. 11B, the image that has passed through the first pupil portion region 501 is the first viewpoint image, and the image that has passed through the second pupil portion region 502 is the second viewpoint image. Each viewpoint image is an image that has passed through half of the original pupil region, and in the case of the pupil portion region divided into two in the horizontal direction, the aperture diameter in the horizontal direction is halved, so the depth of field in the horizontal direction is 4. Double. In the example of this embodiment, since the pupil is not divided in the vertical direction, the change in the depth of field in the vertical direction does not change. Therefore, the first-viewpoint image or the second-viewpoint image is an image having a depth of field that is double the vertical and horizontal average of the depth of field of the image (A + B image) obtained by combining the first and second viewpoint images.

As described above, the viewpoint change processing unit 327 generates a composite image in which the addition ratio of the first viewpoint image and the second viewpoint image is changed to other than 1: 1 to generate an image with an enlarged depth of field. Can be done. Further, the viewpoint change processing unit 327 performs unsharp mask processing using the contrast distribution and the image shift distribution on the image in which the addition ratio of the first viewpoint image and the second viewpoint image is changed to expand the depth. Moreover, it is possible to generate a composite image with contour enhancement.

Next, the area designation will be described with reference to FIGS. 12 and 13. FIG. 12 is a diagram showing an example of a blurred shape in a viewpoint image and a composite image. FIG. 13 is a diagram showing an example in which the user specifies an area and moves the viewpoint or changes the depth.

FIG. 12A is an image (A + B image) obtained by combining the first viewpoint image and the second viewpoint image. FIG. 12B is a first viewpoint image (A image). FIG. 13C is a second viewpoint image (B image). The blurred shapes of the A + B image, the A image, and the B image are compared in the area 1200 surrounded by the dotted line in each figure. In contrast to FIG. 13 (A), which is an image of the A + B image (composite image), FIG. 13 (B), which is the image of the A image (first viewpoint image), is an image in which the right side of the blurred shape of the region 1200 is missing. It has become. Further, FIG. 13 (C), which is an image of the B image (second viewpoint image), is an image in which the left side of the blurred shape of the region 1200 is missing. Since the first viewpoint image and the second viewpoint image are light fluxes that have passed through the first pupil region 501 and the second pupil region 502, which are part of the pupil region, respectively, the effective F value is large and the shape is semicircular. Therefore, the bokeh shape is deformed from a perfect circle.

Therefore, in the present embodiment, in order to suppress unwanted changes in the blurred shape, the viewpoint is moved or the depth is expanded only in the area specified by the user, and the effect of image processing using the viewpoint image is prohibited or reduced in other areas. To do. Therefore, the user can specify the area in which the viewpoint is to be moved, and perform image processing with different parameters in the specified area and other than the specified area. Further, in the present embodiment, in the region where the viewpoint movement process is not performed, that is, in the region other than the designated region, the weighting coefficient (first weighting coefficient) for each of the plurality of viewpoint images is not changed so that the blurred shape of the imaging optical system is not changed. , Second weighting factor) is added almost evenly to generate a composite image.

FIG. 13 shows an example of moving the viewpoint by designating an area.
First, the A + B image is displayed as shown in FIG. 13 (A). Then, the user is made to specify the designated area 1001 for moving the viewpoint and expanding the depth from the image. The image pickup device of the present embodiment has a horizontal viewpoint image because the pupil is divided into two in the horizontal direction. Therefore, the slider bar 1002 and the slider 1003 are arranged in the horizontal direction as a UI for allowing the user to operate in the direction in which the viewpoint changes. If the value at the right end of the slider is defined as 1, the value at the center is 0, and the value at the left end is -1, and the slider is at an arbitrary position x, the ratio of the first viewpoint image to the second viewpoint image is (1 + x). ): The addition ratio is changed so as to be (1-x), and a composite image in which the viewpoint is moved is generated.

FIG. 13A is an image when the slider position is in the center, and the composite ratio of the first viewpoint image and the second viewpoint image in this case is 1: 1. FIG. 13B is an image when the slider position is set to the left end, and the composite ratio of the first viewpoint image and the second viewpoint image in this case is 0: 2. FIG. 13C is an image when the slider position is set to the right end, and the composite ratio of the first viewpoint image and the second viewpoint image in this case is 2: 0. Therefore, the designated area 1001 in FIG. 13B is generated only by the second viewpoint image, and the designated area 1001 in FIG. 13C is generated only by the first viewpoint image. On the other hand, since the addition ratio of the viewpoint image is changed only in the designated area 1001, the composition ratio of each viewpoint image in the area other than the designated area 1001 (for example, the area 1200 in the dotted line) remains 1: 1, that is, the original A + B image. Remains. Therefore, the bokeh shape of the region 1200 is not deformed from a perfect circle.

In the present embodiment, an example in which the addition ratio of the viewpoint image is not changed with respect to an area other than the designated area 1001 has been described, but an area other than the designated area 1001 may be set as an adjustment range different from the designated area 1001. For example, the addition ratio of the viewpoint image is set from the maximum (0:10) to the minimum (10: 0) in the designated area 1001 and from the maximum (3: 7) to the minimum (7: 3) in the areas other than the designated area 1001. May be limited to an adjustment range that can reduce the blurring that is deformed from a perfect circle.
As described above, when synthesizing a plurality of viewpoint images by changing the weighting according to the divided pupil region, it is possible to reduce the deformation of the blurred shape from a perfect circle, and to move the viewpoint or depth only to a desired subject (region). Can be expanded.

Next, the image processing of the viewpoint movement in the present embodiment will be described. By moving the viewpoint, for example, blurring can be adjusted to reduce front blurring on the main subject.
First, the image processing device 300 acquires the captured image and the first viewpoint image acquired by the image pickup device 107 and inputs them to the subtraction unit 322 to generate the second viewpoint image. Then, the first viewpoint image and the second viewpoint image are input to the viewpoint change processing unit 327. The viewpoint change processing unit 327 generates a composite image from the acquired plurality of viewpoint images (first viewpoint image and second viewpoint image). This composite image is an image in which the viewpoint can be moved by changing the composite ratio (weight) of each viewpoint image. As the image acquired by the subtraction unit 322, an image previously captured by the image sensor of the present embodiment and stored in the recording medium may be used.
Hereinafter, the j-th and i-th positions in the row direction and the i-th column direction of the first viewpoint image and the second viewpoint image are defined as (j, i), where j and i are integers, and the first viewpoint of the pixel at the position (j, i). Let the image be A (j, i) and the second viewpoint image be B (j, i).

As the first step, the adjusting unit 326 sets the designated area R = [j1, j2] × [i1, i2] for moving the viewpoint, and the boundary width σ of the designated area. The designated area R is an arbitrary area designated by the user by a UI operation or the like on the display screen. The area designation unit 325 acquires the coordinate information of the area designated by the user and inputs it to the adjustment unit 326. The adjusting unit 326 calculates the table function T (j, i) according to the boundary width σ between the designated area R and the designated area by the equation (1).

The table function T (j, i) becomes 1 inside the designated area R and 0 outside the designated area R, and changes continuously from approximately 1 to 0 at the boundary width σ of the designated area R. If desired, the designated area may have a circular shape or any other shape. Moreover, you may set a plurality of designated areas and a boundary width as needed.

As the second step, the viewpoint change processing unit 327 calculates the weighting coefficient of each viewpoint image in the designated area R for moving the viewpoint. Specifically, the actual coefficient w (-1 ≦ w ≦ 1) is used, and the first weighting coefficient Wa (j, i) of the first viewpoint image A (j, i) is calculated by the equation (2A). Further, the second weighting coefficient Wb (j, i) of the second viewpoint image B (j, i) is calculated by the equation (2B).

As the third step, the viewpoint change processing unit 327 generates a composite image in which the viewpoint is moved in the designated area R. Specifically, from the first viewpoint image A (j, i), the second viewpoint image B (j, i), the first weight coefficient Wa (j, i), and the second weight coefficient Wb (j, i), The composite image I (j, i) is generated by the formula (3).

The viewpoint change processing unit 327 may generate the composite image Is (j, i) by the formula (4A) or the formula (4B) in combination with the refocus processing as the shift amount s, if necessary.

In the present embodiment, a plurality of viewpoint images are generated from signals acquired by an image pickup device in which a plurality of pixels provided with a plurality of sub-pixels for receiving light flux passing through different pupil region regions of the imaging optical system are arranged. .. Then, each of the plurality of viewpoint images is multiplied by a weighting coefficient and combined to generate a composite image. The weighting coefficient for each of the plurality of viewpoint images changes continuously according to the region of the composite image. In the present embodiment, a weight coefficient is applied to each of a plurality of viewpoint images and addition or shift addition is performed to generate a composite image.

Finally, with reference to FIGS. 14 to 16, a flow of processing for generating a composite image using a plurality of viewpoint images will be described.
FIG. 14 is a main flowchart of a process for generating a composite image. Start at S100 and proceed to S101. In S101, the image sensor 107 captures a viewpoint image (A + B image and A image). Then, in S102, the viewpoint images (A + B image and A image) are output from the image sensor 107 and stored in the flash memory 133 as image data in one file format. In S103, the image data stored in the flash memory 133 in S102 is read into the memory 321 of the image processing device 300. At this time, the subtraction unit 322 generates a second viewpoint image (B image) from the captured image (A + B image) and the first viewpoint image (A image) read into the memory 321, and the memory 321 is generated by the original balance. The second viewpoint image is also read. In S104, the image processing device 300 performs image processing (viewpoint image manipulation processing), proceeds to S105, and is completed. The image processing (viewpoint image operation processing) in S104 will be described with reference to the sub-flow chart of FIG.

Next, image processing (viewpoint image manipulation processing) by specifying an area will be described with reference to the sub-flow chart of FIG. Here, as an example of image processing, a case where viewpoint movement processing is performed will be described.
The viewpoint image operation process is started in S200, and the process proceeds to S201. In S201, the operation unit 324 displays an image and a user interface (hereinafter, UI) on the display device via the output unit 314. The image displayed at this time is an image corresponding to image data based on a plurality of viewpoint images, and the first displayed image is an image in which the addition ratio of each viewpoint image is 1: 1, that is, a captured image (A + B image). Is. In S202, the operation unit 324 determines whether or not to move the viewpoint based on the user's selection in the UI, and if the viewpoint is moved, the process proceeds to S203. On the other hand, if the viewpoint is not moved, the process proceeds to S209 and the process ends.

In S203, the area where the user moves the viewpoint is designated on the image displayed on the display device, and the area designation unit 325 acquires the coordinate information such as the coordinates and size of the designated area. In S204, the adjusting unit 326 sets the adjustable range of the parameters of each area specified in S203. The parameter of this embodiment is the addition ratio of each viewpoint image, but other image processing such as sharpness may be used as a parameter. In S205, the user operates the viewpoint movement UI, and the operation unit 324 acquires the adjustment value according to the slider position set by the user. The viewpoint movement UI is, for example, the slider bar 1002 and the slider 1003 shown in FIG. The range of the slider bar 1002 is the adjustable range set in S204, and the adjustment value that can be set by the user by operating the slider is within the adjustable range. In S206, the viewpoint change processing unit 327 changes the addition ratio of the viewpoint image according to the adjustment value acquired in S205. The first viewpoint image (A image), which is a viewpoint image, is an image of the left viewpoint, and the second viewpoint image (B image) is an image of the right viewpoint. Therefore, an image in which the viewpoint is moved is generated by generating a composite image in which the addition ratio of the first viewpoint image and the second viewpoint image is changed according to the slider position. If the value at the right end of the slider is defined as 1, the value at the center is 0, and the value at the left end is -1, and the slider is at an arbitrary position x, the ratio of the first viewpoint image to the second viewpoint image is (1 + x). ): Change the addition ratio so that it becomes (1-x). Further, at this time, the addition ratio of the boundary area of the designated area may be determined to be between the addition ratio of the designated area and the addition ratio of the area adjacent to the designated area.

In S207, development processing is performed on the image data that has undergone image processing, that is, an image in which the addition ratio of the viewpoint image is changed in a region designated by the user, and a composite image is generated. Details of the development process will be described later using the sub-flow chart of FIG. In S208, the output unit 314 outputs the composite image developed in S207 to the display device and displays it on the display device. Then, the process proceeds to S209, and the viewpoint image operation process is completed.

In the present embodiment, an example is shown in which the addition ratio of the first viewpoint image and the second viewpoint image is 1: 1 without adjusting the region other than the designated region, but the present invention is not limited to this. Absent. For example, in an area other than the designated area, the viewpoint may be moved within an adjustment range limited from the designated area.

Next, the development process will be described with reference to FIG. The process is started in S300 and proceeds to S301. In S301, the white balance unit 308 performs the white balance process. The white balance process is a process of applying a gain to each color of R, G, and B so that R, G, and B in the white region have the same color. In S302, the demosaiking unit 309 performs the demosaiking process. The demosaiking process is a process of generating color image signals of the three primary colors R, G, and B as the result of interpolation processing for each pixel by performing interpolation in each specified direction and then selecting a direction. In S303, the gamma conversion unit 310 performs gamma processing. In S304, the color adjustment unit 311 performs the color adjustment process. The color adjustment process is various processes such as noise reduction, saturation enhancement, hue correction, and edge enhancement, which are processes for improving the appearance of an image. In S305, the compression unit 312 compresses the color-adjusted color image data by a method such as JPEG. In S306, the recording unit 313 records the image data compressed by the compression process on the recording medium. Then, the process is completed in S307, and the process returns to the sub-flow chart of the viewpoint image operation.

Although the example of moving the viewpoint by designating the area has been described above, the image processing (viewpoint image manipulation processing) is not limited to the viewpoint movement. For example, a region may be specified to perform focus adjustment (refocus) or a process of changing the depth of field. Also in this case, the weighting coefficients (addition ratios) of the plurality of viewpoint images are changed only in the designated area, and it is possible to suppress the occurrence of unintended changes in the areas other than the designated area. In addition, an example was shown in which the viewpoint image operation is performed for the specified area and not for the other areas, but conversely, the area where the blur is desired to be left is designated as the area where the viewpoint image operation is not performed, and other areas are not performed. The viewpoint image operation may be performed in the area of.

As described above, according to the present embodiment, it is possible to provide a composite image intended by the user when a composite image is generated by changing the weighting of a plurality of viewpoint images according to the divided pupil regions.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and modifications can be made within the scope of the gist thereof.

300 Image processing device 305 Area designation unit 307 Viewpoint change processing unit

Claims (12)

An acquisition method for acquiring multiple viewpoint images,
An image processing means for generating a composite image by performing image processing on image data based on the plurality of viewpoint images.
Designating means for designating an area for performing the image processing by the image processing means, and
An image processing apparatus comprising: an adjusting means for setting an adjustable range of the image processing for each of the areas designated by the designated means. The designation means designates the area based on the designation from the user.
The image processing apparatus according to claim 1, wherein the designation from the user is the designation of the area where the image processing is performed or the designation of the area where the image processing is not performed. The image processing according to claim 1 or 2, wherein the adjusting means sets the adjustable range of another region of the image processing region to be smaller than the adjustable range of the image processing region. apparatus. The image processing means synthesizes the plurality of viewpoint images to generate the composite image, and based on the adjustment value specified within the adjustable range set for each region, for each region. The image processing apparatus according to any one of claims 1 to 3, wherein the addition ratio of the plurality of viewpoint images is determined. The image processing means bases the addition ratio of the plurality of viewpoint images in the boundary region of the predetermined region on which the image processing is performed based on the adjustment value of the predetermined region and the adjustment value of the region adjacent to the predetermined region. The image processing apparatus according to claim 4, wherein the image processing apparatus is determined. The image processing apparatus according to any one of claims 1 to 5, wherein the image processing is a process of moving a viewpoint. The image processing apparatus according to any one of claims 1 to 6, wherein the image processing is a refocusing process or a depth of field changing process. Any one of claims 1 to 7, further comprising an image for the user to specify the area for performing the image processing, and an output unit for outputting the user interface indicating the adjustable range to the display device. The image processing apparatus according to the section. The image processing apparatus according to any one of claims 1 to 8.
An image pickup device including an image pickup device that captures an image of a subject. The image sensor has a plurality of microlenses and a plurality of photoelectric conversion units, and each microlens corresponds to the plurality of photoelectric conversion units.
The imaging device according to claim 9, wherein the plurality of viewpoint images based on signals output by the plurality of photoelectric conversion units corresponding to the respective microlenses are output. The process of acquiring multiple viewpoint images and
The process of specifying the area to perform image processing and
The process of setting the adjustable range of the image processing for each designated area, and
A step of acquiring an adjustment value set within the adjustable range for each region and performing image processing on image data based on the plurality of viewpoint images based on the adjustment value to generate a composite image. An image processing method characterized by having. A program for causing a computer of an image processing apparatus to execute each step according to claim 11.