JP2019057908A - Imaging apparatus and control method thereof - Google Patents

Abstract

To provide an imaging apparatus capable of suppressing capacity of data to be recorded.SOLUTION: An imaging apparatus includes: an image acquisition unit 151 that acquires a plurality of viewpoint images; an operation information acquisition unit 154 that acquires imaging conditions when the viewpoint image is captured; and a compression unit 161 that compresses the viewpoint image to be recorded on a flash memory 133. The compression unit 161 compresses the viewpoint image according to the imaging conditions such as ISO sensitivity and an aperture value or a value of contrast information corresponding to a contrast distribution. The compression unit 161 changes a compression rate for compressing the viewpoint image according to the imaging conditions or the value of the contrast information.SELECTED DRAWING: Figure 19

Description

  The present invention relates to an imaging apparatus and a control method thereof.

  In the imaging surface phase difference method employed in the imaging apparatus, focus detection by the phase difference method is performed by focus detection pixels formed on the imaging element. The imaging apparatus disclosed in Patent Document 1 uses a two-dimensional imaging element in which one microlens and a plurality of photoelectric conversion units are formed for one pixel. The photoelectric conversion units divided into a plurality are configured to receive different regions of the exit pupil of the photographing lens through one microlens, and perform pupil division. A viewpoint signal is generated from the light received by each photoelectric conversion unit. By calculating an image shift amount from the parallax between a plurality of viewpoint signals and converting it to a defocus amount, phase difference type focus detection is performed. Patent Document 2 discloses that an imaging signal is generated by adding a plurality of viewpoint signals generated by a plurality of divided photoelectric conversion units.

  When a file including a plurality of viewpoint signals is generated, the data size of the image file becomes larger than usual, and the recording medium capacity is compressed. Japanese Patent Application No. 2006-094020 discloses a control method for deleting a viewpoint signal when it is no longer necessary after executing a predetermined process based on the viewpoint signal.

US Pat. No. 4,410,804 JP 2001-083407 A

  However, in Patent Document 3, since the presence / absence of recording of the viewpoint signal is determined after executing predetermined processing based on the viewpoint signal in the camera, the recording medium capacity is pressed immediately after shooting. Further, when the predetermined processing based on the viewpoint signal is not executed in the camera, the viewpoint signal can only be stored in the recording medium, and thus the capacity of the recording medium is compressed.

  An object of the present invention is to provide an imaging device capable of suppressing the data volume to be recorded.

  In order to solve the above-described problems, an imaging apparatus according to the present invention records an acquisition unit that acquires a plurality of viewpoint images, an imaging condition acquisition unit that acquires imaging conditions when the viewpoint image is captured, and a recording unit. Compression means for compressing the viewpoint image. The compression means compresses the viewpoint image according to the photographing condition.

  In order to solve the above problems, an imaging apparatus according to the present invention includes an acquisition unit that acquires a plurality of viewpoint images, an imaging condition acquisition unit that acquires imaging conditions when the viewpoint images are captured, and a recording unit. Determining means for determining data to be recorded, and the determining means determines whether or not to record the viewpoint image in the recording means in accordance with the photographing condition.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging device which can suppress the data capacity to record can be provided.

It is a block diagram which shows schematic structure of an imaging device. It is a block diagram which shows the structure of an image processing circuit. It is the schematic of a pixel arrangement | sequence. It is the schematic plan view and schematic sectional drawing of a pixel. It is a schematic explanatory drawing of a pixel and pupil division. It is a figure which shows the example of light intensity distribution inside a pixel. It is a figure which illustrates pupil intensity distribution. It is a schematic explanatory drawing of an image pick-up element and pupil division. FIG. 6 is a schematic relationship diagram between a defocus amount and an image shift amount. It is a figure which shows the example of the contrast distribution of a captured image. It is a figure explaining the example of the parallax emphasis which expanded the difference between viewpoint images. It is a figure explaining the outline of a refocus process. It is a figure explaining the outline | summary of an unsharpness process. It is a figure explaining the refocus possible range. It is a figure explaining the principle of a viewpoint movement process. It is a figure explaining the pupil shift in the peripheral image height of an image sensor. It is a flowchart which shows the process which records information. It is a flowchart which shows the process which determines the information to record. It is a flowchart which shows the process which compresses a viewpoint image.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration example of an imaging apparatus having an imaging element. In the following, an embodiment in which the present invention is applied to an imaging device such as an arbitrary digital camera capable of acquiring LF data will be described. Etc. can be widely applied. Also, an arbitrary device transmits LF data and operation contents to a server device (including a virtual machine) provided with processing means such as a processor on the network, and a part or all of the processing for the LF data is executed by the server device. A configuration may be included.

  The imaging apparatus includes a first lens group 101, a diaphragm / shutter 102, a second lens group 103, a third lens group 105, an optical low-pass filter 106, an imaging element 107, an electronic flash 115, and an AF auxiliary light source 116. The imaging apparatus also includes a zoom actuator 111, an aperture shutter actuator 112, a focus actuator 114, a CPU 121, various circuits 122 to 129, a display unit 131, an operation unit 132, and a flash memory 133.

  The first lens group 101 is disposed at the front end of the imaging optical system (imaging optical system), and is held by a lens barrel so as to be able to advance and retract in the optical axis direction. The iris / shutter 102 adjusts the light amount at the time of shooting by adjusting the aperture diameter, and also has a function as an exposure time adjustment shutter at the time of still image shooting. The second lens group 103 moves forward and backward in the optical axis direction integrally with the diaphragm / shutter 102 and performs a zooming operation in conjunction with the forward / backward movement of the first lens group 101 to realize a zoom function. The third lens group 105 is a focus lens that performs focus adjustment by advancing and retreating in the optical axis direction. The optical low-pass filter 106 is an optical element for reducing false colors and moire in the captured image. The imaging element 107 includes, for example, a two-dimensional CMOS (complementary metal oxide semiconductor) photosensor and a peripheral circuit, and is disposed on the imaging plane of the imaging optical system.

  The zoom actuator 111 rotates the cam barrel of the lens barrel to move the first lens group 101 and the second lens group 103 in the optical axis direction and perform a zooming operation. The aperture shutter actuator 112 controls the aperture diameter of the aperture / shutter 102 to adjust the amount of photographing light, and controls the exposure time during still image photographing. The focus actuator 114 performs the focus adjustment operation by moving the third lens group 105 in the optical axis direction.

  The electronic flash 115 for illuminating the subject is used at the time of photographing, and a flash illumination device using a xenon tube or an illumination device including an LED (light emitting diode) that emits light continuously is used. An AF (autofocus) auxiliary light source 116 projects an image of a mask having a predetermined opening pattern onto a subject field via a light projection lens. This improves the focus detection capability for low-luminance subjects or low-contrast subjects.

  A CPU (Central Processing Unit) 121 that constitutes a control unit of the camera body has a control center function that controls various controls. The CPU 121 includes an arithmetic unit, a ROM (read only memory), a RAM (random access memory), an A (analog) / D (digital) converter, a D / A converter, a communication interface circuit, and the like. The CPU 121 drives various circuits in the imaging apparatus according to a predetermined program stored in the ROM, and executes a series of operations such as AF control, imaging processing, image processing, and recording processing. In AF control, focus state detection and focus adjustment of the imaging optical system are performed.

  The electronic flash control circuit 122 controls lighting of the electronic flash 115 in synchronization with the photographing operation in accordance with a control command from the CPU 121. The auxiliary light source driving circuit 123 controls lighting of the AF auxiliary light source 116 in synchronization with the focus detection operation in accordance with a control command from the CPU 121. The imaging element driving circuit 124 controls the imaging operation of the imaging element 107, A / D converts the acquired imaging signal, and outputs it to the CPU 121. The image processing circuit 125 performs processing such as gamma conversion, color interpolation, and JPEG (Joint Photographic Experts Group) compression of an image acquired by the image sensor 107 in accordance with a control command of the CPU 121.

  The focus drive circuit 126 drives the focus actuator 114 based on the focus detection result in accordance with the control command of the CPU 121, and moves the third lens group 105 in the optical axis direction to perform focus adjustment. The aperture shutter drive circuit 128 drives the aperture shutter actuator 112 in accordance with a control command from the CPU 121 to control the aperture diameter of the aperture / shutter 102. The zoom drive circuit 129 drives the zoom actuator 111 in accordance with a photographer's zoom operation instruction in accordance with a control command from the CPU 121.

  The display unit 131 includes a display device such as an LCD (Liquid Crystal Display), and information on the shooting mode of the imaging apparatus, a preview image before shooting and a confirmation image after shooting, a focus state display image at the time of focus detection, and the like. Is displayed. The operation unit 132 includes various operation switches and outputs an operation instruction signal to the CPU 121. The flash memory 133 is a recording medium that can be attached to and detached from the camera body, and records captured image data and the like. Predetermined image data is displayed on the screen of the display unit 131 or recorded in the flash memory 133. The predetermined image data is, for example, a plurality of viewpoint image data captured by the image sensor 107 and then processed by the image processing circuit 125, or a plurality of viewpoint image data synthesized in the image sensor 107 or the image processing circuit 125. This is composite image data.

  FIG. 2 is a block diagram illustrating a configuration example of the image processing circuit 125. The image processing circuit 125 includes an image acquisition unit 151, a subtraction unit 152, a shading processing unit 153, an operation information acquisition unit 154, a viewpoint change processing unit 155, and a refocus processing unit 156. The image processing circuit 125 includes a white balance unit 157, a demosaicing unit 158, a gamma conversion unit 159, a color adjustment unit 160, a compression unit 161, and an output unit 163.

  The image acquisition unit 151 acquires image data from the flash memory 133 and stores the acquired image data. The image data is a captured image (A + B image) and a first viewpoint image (or a second viewpoint image) obtained by combining a first viewpoint image and a second viewpoint image described later. The subtraction unit 152 generates a second viewpoint image (B image) by subtracting the first viewpoint image (A image) from the captured image (A + B image).

  The shading processing unit 153 corrects the light amount change due to the image heights of the first viewpoint image and the second viewpoint image. The operation information acquisition unit 154 receives adjustment values for viewpoint movement and focus adjustment (refocus) set by the user on the user interface. Then, the operation information acquisition unit 154 transfers the adjustment value operated by the user to the viewpoint change processing unit 155 and the refocus processing unit 156. The operation information acquisition unit 154 also has a function as a shooting condition acquisition unit that acquires shooting conditions at the time of shooting. The shooting conditions include an aperture value, an ISO value, a subject distance, and the like. The viewpoint change processing unit 155 performs image processing using a plurality of viewpoint images based on the adjustment value acquired from the operation information acquisition unit 154. The viewpoint change processing unit 155 generates an image in which the viewpoint is changed or an image in which the depth of field is changed by changing the addition ratio (weighting) of the first viewpoint image and the second viewpoint image. The refocus processing unit 156 generates a composite image by shifting and adding the first viewpoint image and the second viewpoint image in the pupil division direction, and generates images at different focus positions.

  Next, components that perform development processing in the image processing circuit 125 will be described. The white balance unit 157 performs white balance processing. Specifically, a gain is applied to each color of R, G, and B so that R, G, and B in the white region have the same color. By performing the white balance process before the demosaicing process, it is possible to avoid a saturation higher than the false color saturation due to a color cast or the like when calculating the saturation, thereby preventing erroneous determination.

  The demosaicing unit 158 generates a color image in which R, G, and B color image data are arranged in all pixels by interpolating two color mosaic image data out of the three primary colors missing in each pixel. To do. Specifically, first, interpolation is performed on the target pixel in the respective specified directions using pixels around the target pixel, and then the direction is selected, so that R, G, and B of 3 as interpolation results for each pixel are obtained. A primary color image signal is generated.

  The gamma conversion unit 159 performs gamma correction processing on the color image data of each pixel to generate basic color image data. For example, the gamma conversion unit 159 generates color image data matched with the display characteristics of the display unit 131 as basic color image data. The color adjustment unit 160 applies various color adjustment processes such as noise reduction, saturation enhancement, hue correction, and edge enhancement, which are processes for improving the appearance of the image, to the color image data.

  The compression unit 161 compresses color-adjusted color image data by a method based on a predetermined compression method such as JPEG, and reduces the data size of the color image data when recording. The output unit 163 outputs various image data such as color image data, compressed image data, and display data for a user interface to the display unit 131 of the imaging device, an external display device, the flash memory 133, and the like.

  FIG. 3 is a diagram illustrating a schematic diagram of an array of pixels and sub-pixels of the image sensor. The left-right direction in FIG. 3 is defined as the x-axis direction, the up-down direction is defined as the y-axis direction, and the direction perpendicular to the x-axis direction and the y-axis direction (direction perpendicular to the paper surface) is defined as the z-axis direction. FIG. 3 shows a pixel array of the two-dimensional CMOS sensor (imaging device) of this embodiment in a range of 4 columns × 4 rows and a sub-pixel array in a range of 8 columns × 4 rows.

  In the two columns × two rows of pixels 200 shown in FIG. 3, the pixel 200R having R (red) spectral sensitivity is in the upper left position, and the pixel 200G having G (green) spectral sensitivity is in the upper right and lower left positions. , B (blue) has a pixel 200B having a spectral sensitivity at the lower right position. Further, each pixel has a first subpixel 201 and a second subpixel 202 that are divided into two in the x-axis direction and one in the y-axis direction. That is, if the number of divisions in the x direction is expressed as Nx, the number of divisions in the y direction is expressed as Ny, and the number of divisions is expressed as NLF, FIG. 3 shows Nx = 2, Ny = 1, NLF = Nx × Ny = 2. An example is shown. Each of the sub-pixels has a function as a focus detection pixel that outputs a focus detection signal.

  In the example shown in FIG. 3, by arranging a large number of pixels of 4 columns × 4 rows (8 columns × 4 rows of sub-pixels) on the surface, it is possible to display on the display unit 131 or record in the flash memory 133. A captured image (A + B image) to be used and a signal for generating a plurality of viewpoint images can be acquired. In the present embodiment, the pixel period P is 4 μm, the number of pixels N is 5575 columns × 3725 rows = approximately 2075 million pixels, the column direction period PS of the subpixels is 2 μm, and the number of subpixels NS is 11150 columns × vertical. The description will be made assuming that 3725 rows = an image sensor having about 41.5 million pixels.

  4A is a plan view of one pixel 200G of the image sensor shown in FIG. 3 as viewed from the light receiving surface side (+ z side) of the image sensor. The z axis is set in a direction perpendicular to the paper surface of FIG. 4A, and the near side is defined as the positive direction of the z axis. Also, the y-axis is set in the vertical direction perpendicular to the z-axis, the upper direction is the positive direction of the y-axis, the x-axis is set in the horizontal direction perpendicular to the z-axis and the y-axis, and the right direction is the positive direction of the x-axis It is defined as FIG. 4B is a cross-sectional view when viewed from the −y side along the line aa in FIG.

  As shown in FIGS. 4A and 4B, the micro lens 305 for condensing incident light is formed on the pixel 200G on the light receiving surface side (+ z-axis direction) of each pixel. . Further, a plurality of photoelectric conversion units having two divisions, that is, two divisions in the x direction and one division in the y direction, are formed. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 correspond to the first subpixel 201 and the second subpixel 202, respectively. Note that the number of divisions of the photoelectric conversion unit (subpixel) is not limited to two. The division direction is not limited to the x direction and may be divided in the y direction.

  The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are two independent pn junction photodiodes, which are pin structure photodiodes in which an intrinsic layer is sandwiched between a p-type layer and an n-type layer. Further, if necessary, the intrinsic layer may be omitted to form a pn junction photodiode. In each pixel, a color filter 306 is formed between the microlens 305 and the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302. Further, if necessary, the spectral transmittance of the color filter 306 may be changed for each pixel or each photoelectric conversion unit (for each sub-pixel), or the color filter may be omitted.

  The light incident on the pixel 200 </ b> G is collected by the microlens 305, dispersed by the color filter 306, and then received by the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302. In the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302, a pair of electrons and holes are generated according to the amount of received light and separated by a depletion layer. Accumulated in the figure). On the other hand, the holes are discharged to the outside of the image sensor through a p-type layer connected to a constant voltage source (not shown). Electrons accumulated in the n-type layers (not shown) of the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via the transfer gate and converted into a voltage signal. The

  FIG. 5 is a schematic explanatory diagram showing the correspondence between the pixel structure and pupil division. FIG. 5 is a cross-sectional view of the pixel structure shown in FIG. 4A taken along the line aa, viewed from the + y side, and the exit pupil plane of the imaging optical system viewed from the −z-axis direction. The figure is shown. In FIG. 5, in order to correspond to the coordinate axis of the exit pupil plane, the x axis and the y axis of the cross-sectional view are shown inverted with respect to the state shown in FIG. 4. The imaging element is disposed in the vicinity of the imaging surface of the photographing lens (imaging optical system), and the light flux from the subject passes through the exit pupil 400 of the imaging optical system and enters each pixel. The surface on which the image sensor is arranged is defined as an image capturing surface.

The first pupil partial region 501 of the first subpixel 201 is substantially optically conjugate with the light receiving surface of the first photoelectric conversion unit 301 whose center of gravity is decentered in the −x direction and the microlens 305. The first pupil partial area 501 represents a pupil area that can be received by the first subpixel 201. The first pupil partial region 501 of the first subpixel 201 has a center of gravity decentered on the + X side on the pupil plane.
The second pupil partial region 502 of the second subpixel 202 is substantially optically conjugate with the light receiving surface of the second photoelectric conversion unit 302 whose center of gravity is decentered in the + x direction and the microlens 305. The second pupil partial area 502 represents a pupil area that can be received by the second subpixel 202. The center of gravity of the second pupil partial region 502 of the second subpixel 202 is decentered toward the −X side on the pupil plane.

  The pupil region 500 has a substantially optically conjugate relationship by the light receiving surface where all of the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are combined with the microlens 305. The pupil region 500 is a pupil region that can receive light in the entire pixel 200G including all of the first subpixel 201 and the second subpixel 202.

  FIG. 6 is a diagram illustrating an example of a light intensity distribution when light is incident on the microlens formed in each pixel. FIG. 6A shows a light intensity distribution in a cross section parallel to the optical axis of the microlens. FIG. 6B shows the light intensity distribution in a cross section perpendicular to the optical axis of the microlens at the focal position of the microlens. Incident light is condensed at the focal position by the microlens. However, due to the influence of diffraction due to the wave nature of light, the diameter of the focused spot cannot be made smaller than the diffraction limit Δ, and has a finite size. The size of the light receiving surface of the photoelectric conversion unit is about 1 to 2 μm, whereas the condensing spot of the microlens is about 1 μm. Therefore, the first pupil partial region 501 and the second pupil partial region 502 in FIG. 5 that are in a conjugate relationship with the light receiving surface of the photoelectric conversion unit via the microlens are not clearly divided into pupils due to diffraction blur. The light reception rate distribution (pupil intensity distribution) depends on the incident angle of light.

  FIG. 7 is a diagram illustrating an example of a light reception rate distribution (pupil intensity distribution) depending on the incident angle of light. The horizontal axis represents pupil coordinates, and the vertical axis represents the light reception rate. A graph line L1 indicated by a solid line in FIG. 7 represents a pupil intensity distribution along the X axis of the first pupil partial region 501 in FIG. The light reception rate indicated by the graph line L1 rises steeply from the left end and gradually decreases after reaching the peak, and then the rate of change becomes gentle and reaches the right end. A graph line L2 indicated by a broken line in FIG. 7 represents the pupil intensity distribution along the X axis of the second pupil partial region 502. The light reception rate indicated by the graph line L2 is opposite to the graph line L1 (symmetrical), and rises sharply from the right end and gradually decreases after reaching the peak, and then the rate of change becomes gradual to the left end. And so on. It can be seen that the pupil is gently divided as shown.

  FIG. 8 is a schematic diagram illustrating a correspondence relationship between the image sensor and pupil division. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 correspond to the first subpixel 201 and the second subpixel 202, respectively. In each pixel of the imaging element, the first subpixel 201 and the second subpixel 202 divided by 2 × 1 are different pupil partial regions of the first pupil partial region 501 and the second pupil partial region 502 of the imaging optical system, respectively. The light beam that has passed through is received. That is, light beams that have passed through different pupil partial regions of the first pupil partial region 501 and the second pupil partial region 502 are incident on the pixels of the image sensor at different angles, and are divided into 2 × 1 first subpixels 201. And the second sub-pixel 202 receive light.

  By selecting a signal of a specific subpixel from the first subpixel 201 and the second subpixel 202 for each pixel from signals received by each subpixel, the first pupil partial region 501 of the imaging optical system is selected. And a viewpoint image corresponding to a specific pupil partial region in the second pupil partial region 502 can be generated. For example, by selecting the signal of the first subpixel 201 in each pixel, a first viewpoint image having a resolution of N pixels corresponding to the first pupil partial region 501 of the imaging optical system can be generated. The same applies to other sub-pixels. The imaging device of the present embodiment has a structure in which a plurality of pixels provided with a plurality of photoelectric conversion units (sub-pixels) that receive light beams passing through different pupil partial regions of the imaging optical system are arranged, and different pupils A plurality of viewpoint images can be generated for each partial region.

  In the present embodiment, each of the first viewpoint image and the second viewpoint image is a Bayer array image. If necessary, demosaicing processing may be performed on the first viewpoint image and the second viewpoint image. Further, by adding and reading the signals of the first subpixel 201 and the second subpixel 202 for each pixel of the image sensor, a captured image having a resolution of N effective pixels can be generated.

  Next, the relationship between the defocus amount and the image shift amount between the first viewpoint image and the second viewpoint image acquired by the image sensor of the present embodiment will be described. FIG. 9 is a diagram illustrating a schematic relationship between the defocus amounts of the first viewpoint image and the second viewpoint image and the image shift amount between the first viewpoint image and the second viewpoint image. An imaging element (not shown) is arranged on the imaging surface 600, and the exit pupil of the imaging optical system is 2 × in the first pupil partial region 501 and the second pupil partial region 502, as in the case of FIGS. Divided into one.

  The size | d | of the defocus amount d represents the distance from the imaging position of the subject image to the imaging surface 600. A negative sign (d <0) indicates a front pin state where the imaging position of the subject is on the subject side from the imaging surface, and a positive sign indicates a rear pin state where the imaging position of the subject is on the opposite side of the subject from the imaging surface (d> 0). ) To define the orientation. An in-focus state where the imaging position of the subject is on the imaging surface (in-focus position) is d = 0. In FIG. 9, the subject 801 shows an example of the position corresponding to the in-focus state (d = 0), and the subject 802 shows the example of the position corresponding to the front pin state (d <0). Hereinafter, the front pin state (d <0) and the rear pin state (d> 0) are collectively referred to as a defocus state (| d |> 0).

  In the front pin state (d <0), the luminous flux that has passed through the first pupil partial area 501 (or the second pupil partial area 502) out of the luminous flux from the subject 802 is once condensed and then the gravity center position of the luminous flux. G1 (G2) extends around width Γ1 (Γ2). In this case, the image is blurred on the imaging surface 600. The blurred image is received by the first sub-pixel 201 (or the second sub-pixel 202) that constitutes each pixel arranged in the image sensor, and a first viewpoint image (or a second viewpoint image) is generated. Therefore, in the first viewpoint image (or second viewpoint image), the subject 802 is a subject image (blurred image) having a blur width Γ1 (Γ2) at the gravity center position G1 (or G2) on the imaging surface 600. Recorded as image data.

  The blur width Γ1 (or Γ2) of the subject image increases substantially in proportion as the magnitude | d | of the defocus amount d increases. Similarly, the magnitude | p | of the image shift amount p of the subject image between the first viewpoint image and the second viewpoint image (= difference G1-G2 in the center of gravity of the light beam) is also the magnitude | d of the defocus amount d. As | increases, it increases approximately proportionally. In the rear pin state (d> 0), the image shift direction of the subject image between the first viewpoint image and the second viewpoint image is opposite to that in the front pin state, but there is a similar tendency.

  Therefore, in this embodiment, as the magnitude of the defocus amount of the first viewpoint image and the second viewpoint image or the imaging signal obtained by adding the first viewpoint image and the second viewpoint image increases, The amount of image shift between the image and the second viewpoint image increases.

Next, image processing using viewpoint images will be described. Image processing using viewpoint images includes refocus processing, viewpoint movement processing, blur adjustment, unnecessary component reduction processing, and the like.
The viewpoint image correction process and the refocus process will be described. In the refocus process of the present embodiment, three stages of processes are executed. As a first stage, the viewpoint change processing unit 155 calculates a contrast distribution representing the level of contrast based on each pixel value of the captured image. As the second stage, based on the contrast distribution calculated in the first stage, the viewpoint change processing unit 155 enlarges the difference between a plurality of viewpoint images (first viewpoint image and second viewpoint image) for each pixel, thereby reducing the parallax. Perform emphasis conversion. By this second stage process, a plurality of corrected viewpoint images (first corrected viewpoint image and second corrected viewpoint image) are generated. As a third stage, the refocus processing unit 156 relatively shift-adds a plurality of corrected viewpoint images (first corrected viewpoint image and second corrected viewpoint image) to generate a refocus image.

  Hereinafter, the j-th position in the row direction and the i-th position in the column direction of the image sensor 107 are denoted as (j, i). j and i are integer variables. The first viewpoint image of the pixel at the position (j, i) is denoted as A0 (j, i), and the second viewpoint image is denoted as B0 (j, i). The captured image is expressed as I (j, i), and I (j, i) = A0 (j, i) + B0 (j, i).

First, calculation of contrast distribution will be described. The viewpoint change processing unit 155 matches the color centroids of the respective colors RGB for each position (j, i) with respect to the Bayer array captured image I (j, i) according to the equation (1), thereby obtaining luminance Y (j, i) is calculated.

  Next, the viewpoint change processing unit 155 [1, 2, -1, -4, -1, 2] in the horizontal direction (column i direction) that is the pupil division direction with respect to the luminance Y (j, i). , 1] and the like, and a horizontal high-frequency component dY (j, i) is calculated. The viewpoint change processing unit 155 performs high-frequency cut filter processing such as [1, 1, 1, 1, 1, 1, 1] in the vertical direction (row j direction) that is not the pupil division direction, as necessary. The high frequency noise in the vertical direction may be suppressed.

Next, the viewpoint change processing unit 155 calculates the normalized (normalized) horizontal high-frequency component dZ (j, i) according to the equation (2). Here, by adding a non-zero constant Y0 to the denominator, the division by 0 prevents the expression (2) from diverging. The viewpoint change processing unit 155 may suppress high-frequency noise by performing high-frequency cut filter processing on the luminance Y (j, i) as necessary before normalization by the equation (2).

The viewpoint change processing unit 155 calculates the contrast distribution C (j, i) according to the equation (3). The first line of Expression (3) indicates that the contrast distribution C (j, i) is set to 0 when the luminance of the captured image is lower than the predetermined luminance Yc and low. On the other hand, the third line of Equation (3) indicates that the contrast distribution C (j, i) is set to 1 when the normalized high-frequency component dZ (j, i) is larger than the predetermined value Zc. Otherwise, as shown in the second line of Equation (3), the value normalized by dividing dZ (j, i) by Zc is the contrast distribution C (j, i). .
Thus, the contrast distribution C (j, i) takes a value in the range of [0, 1] (0 to 1). The closer the value of C (j, i) is to 0, the lower the contrast is, and the closer it is to 1, the higher the contrast is.

  FIG. 10 is a diagram illustrating an example of the contrast distribution C (j, i) of the captured image obtained by Expression (3). In the distribution chart shown in FIG. 10, the contrast level index is represented by the gray scale display on the right side. The white part indicates that there are many horizontal high-frequency components and the contrast is high, and the black part indicates that there are few horizontal high-frequency components and the contrast is low.

Next, the parallax enhancement process will be described. In the parallax enhancement processing, first, an image shift distribution of the viewpoint image is calculated. The image shift distribution is calculated by performing a correlation operation on a pair of images of the first viewpoint image A0 and the second viewpoint image B0 and calculating a relative positional shift amount of the pair of images. Various known methods are known for the correlation calculation. The viewpoint change processing unit 155, for example, adds the absolute values of the difference between a pair of images as shown in Expression (4), thereby, A correlation value is calculated.
Here, A0 _i and B0 _i represent the luminance of the i-th pixel of the first viewpoint image A0 and the second viewpoint image B0, respectively. Ni is a number representing the number of pixels used in the calculation, and is appropriately set according to the minimum calculation range of the image shift distribution.

  For example, the viewpoint change processing unit 155 calculates k that minimizes COR (k) in Expression (4) as the image shift amount. That is, with the pair of images shifted by k pixels, the absolute value of the difference between each i-th A0 pixel and B0 pixel in the row direction is taken, and the absolute value is added to a plurality of pixels in the row direction. Then, the viewpoint change processing unit 155 considers the added value, that is, k when COR (k) is smallest, as the image shift amount of A0 and B0, and calculates the shift amount k pixels.

On the other hand, when the two-dimensional image is moved by k pixels only in the pupil division direction so as to take the difference between the pixels of the first viewpoint image A0 and the second viewpoint image B0, and the addition is performed for a plurality of columns, the correlation calculation is It is defined by equation (5).
Here, A0 _ij and B0 _ij represent the luminance of the i-th pixel in the j-th column of the first viewpoint image A0 and the second viewpoint image B0, respectively. Ni represents the number of pixels used in the calculation, and nj represents the number in the column direction of a pair of images on which the correlation calculation is performed.

  The viewpoint change processing unit 155 calculates k that minimizes COR (k) in Expression (5) as the image shift amount, similarly to Expression (4). Note that the subscript k is added only to i and is independent of j. This corresponds to performing the correlation calculation while moving the two-dimensional image only in the pupil division direction. The viewpoint change processing unit 155 calculates an image shift amount of each region of the first viewpoint image A0 and the second viewpoint image B0 according to the equation (5), and calculates an image shift distribution. In the refocus process of the present embodiment described later, the refocus process is performed by performing the sharpness process described later only on the high contrast portion. Therefore, in the above-described contrast distribution calculation process, the correlation calculation according to the equation (5) is not performed on the region where the contrast distribution C (j, i) is 0 (that is, a position having a luminance lower than the predetermined luminance Yc). It may be.

  Next, a specific example of parallax enhancement processing will be described. As shown in the example of the pupil intensity distribution in FIG. 7, the pupil division by the microlens formed for each pixel and the photoelectric conversion unit divided into a plurality is a gentle pupil division due to diffraction blur. Therefore, the point images of the plurality of viewpoint images (the first viewpoint image and the second viewpoint image) are also gently divided, are not easily parallax, and the effective F value in the pupil division direction is not sufficiently dark (large). Effective depth of focus is unlikely to be deep. Therefore, in the present embodiment, the viewpoint change processing unit 155 increases the difference between viewpoint images for each pixel and enhances parallax with respect to a plurality of viewpoint images (first viewpoint image and second viewpoint image). I do. A plurality of corrected viewpoint images (a first corrected viewpoint image and a second corrected viewpoint image) are generated by the process of enhancing the parallax.

The viewpoint change processing unit 155 calculates the difference between the viewpoint images according to Expression (6) and Expression (7) for the first viewpoint image A ₀ (j, i) and the second viewpoint image B ₀ (j, i). A process of enlarging and enhancing the parallax is performed. Then, a first modified viewpoint image A (j, i) and a second modified viewpoint image B (j, i) are generated. Here, the coefficient k is a real number that satisfies “0 ≦ k ≦ 1”, and the coefficient α is a real number that satisfies “0 ≦ α ≦ 1”.

Note that k ₁ and k ₂ , A 1 (j, i) and B 1 (j, i) are calculated in the process of calculating the first modified viewpoint image A (j, i) and the second modified viewpoint image B (j, i). It is a variable defined by equation (6). A1 (j, i) is calculated as the sum of terms obtained by multiplying the first viewpoint image A ₀ (j, i) and the second viewpoint image B ₀ (j, i) by k ₁ and k ₂ respectively. The B ₁ (j, i) is a sum of terms obtained by multiplying the first viewpoint image A ₀ (j, i) and the second viewpoint image B ₀ (j, i) by k ₂ and k ₁ , respectively. Calculated.

Each corrected viewpoint image is calculated by the sum of the first term on the right side and the second term on the right side of Equation (7). That is, the first term on the right side is a term indicating an average value of A ₁ (j, i) or B ₁ (j, i) and their absolute values. The second term on the right _side, B 1 _(j, i) or _A 1 (j, i) of the absolute _value, B 1 _(j, i) or _A 1 (j, i) by dividing the difference between the 2 This is a term multiplied by a coefficient α.

FIG. 11 is a diagram illustrating an example in which the difference between the viewpoint images is enlarged by the parallax enhancement processing. The horizontal axis indicates the 1152st to 1156th pixels in subpixel units, and the vertical axis indicates the magnitude of parallax in each pixel. That is, the horizontal axis represents the pixel position, and the vertical axis represents the pixel value (signal level). In FIG. 11, the first viewpoint image A ₀ (before correction) A and the second viewpoint image B ₀ (before correction B) before performing the parallax enhancement processing are indicated by broken lines. Moreover, the solid line shows an example of the first corrected viewpoint image A (after correction A) and the second corrected viewpoint image B (after correction B) after performing the parallax enhancement processing according to the expressions (4) and (5). The portion where the difference between the viewpoint images is large before the conversion is further enlarged and the parallax is emphasized, but the portion where the difference between the viewpoint images is small before the conversion is not changed so much.

  As described above, in the present embodiment, the viewpoint change processing unit 155 generates a plurality of corrected viewpoint images for each of the plurality of viewpoint images by a process of enlarging the difference between the plurality of viewpoint images and enhancing the parallax. . The viewpoint change processing unit 155 performs conversion by weighting calculation between signals of a plurality of sub-pixels included in a pixel, as in Expression (6) and Expression (7), in order to suppress the load of parallax enhancement processing. .

  In Expression (6), when the value of k is increased to increase the degree of parallax enhancement by conversion, the parallax between a plurality of corrected viewpoint images (first corrected viewpoint image and second corrected viewpoint image) increases, and the division direction The effective F value of can be deeply corrected. On the other hand, if the degree of parallax enhancement is excessively increased, the noise of the corrected viewpoint image increases and the S / N ratio (signal-to-noise ratio) decreases. Therefore, in the present embodiment, the strength of the parallax enhancement conversion is adaptively adjusted for each region based on the contrast distribution C (j, i). That is, the viewpoint change processing unit 155 adjusts to increase the degree of parallax enhancement in order to increase the parallax and darken (increase) the effective F value in the division direction in a region where the contrast is relatively high. On the other hand, in the region where the contrast is low, in order to maintain the S / N ratio, the parallax enhancement intensity is adjusted to be weak. As a result, it is possible to improve the refocus effect while suppressing a decrease in the S / N ratio.

  The viewpoint change processing unit 155 performs processing for strongly adjusting the intensity of parallax enhancement in a region with a higher luminance than a region with a low luminance of a captured image, as necessary, and suppresses a decrease in S / N. In addition, if necessary, a process for adjusting the intensity of parallax enhancement more strongly in a region where the high frequency component is larger than a region where the high frequency component of the captured image is small can be performed to suppress the S / N decrease. As described above, by increasing the parallax between the plurality of corrected viewpoint images (the first corrected viewpoint image and the second corrected viewpoint image), the effective F value in the dividing direction is darkened (increased), and the effective in the dividing direction is increased. Deep depth of focus can be corrected. Further, in the refocus processing described later, a refocus image is generated using a plurality of corrected viewpoint images (first corrected viewpoint image and second corrected viewpoint image), thereby improving the refocus effect (change in image due to refocusing). Can be emphasized).

  Next, the refocus process will be described. FIG. 10 is an explanatory diagram showing an overview of the refocusing process (third generation process) in the pupil division direction (horizontal direction) using a plurality of corrected viewpoint images (first corrected viewpoint image and second corrected viewpoint image). The imaging surface 600 in FIG. 12 corresponds to the imaging surface 600 shown in FIG. In FIG. 12, the first modified viewpoint image of the i-th pixel in the column direction of the image sensor arranged on the imaging surface 600 is denoted as Ai, and the second modified viewpoint image is denoted as Bi, where i is an integer variable. This is schematically shown. The first modified viewpoint image Ai includes a light reception signal of a light beam incident on the i-th pixel at the principal ray angle θa (corresponding to the first pupil partial region 501 in FIG. 8). The second modified viewpoint image Bi includes a light reception signal of a light beam incident on the i-th pixel at the principal ray angle θb (corresponding to the second pupil partial region 502 in FIG. 8). That is, the first modified viewpoint image Ai and the second modified viewpoint image Bi have incident angle information in addition to the light intensity distribution information.

  The first modified viewpoint image Ai and the second modified viewpoint image Bi have not only light intensity distribution information but also incident angle information. Therefore, the refocus processing unit 156 can generate a refocus image on a predetermined virtual imaging plane (virtual imaging plane 610). The refocus processing unit 156 generates a refocus signal on the virtual imaging plane 610 by performing parallel movement processing and addition processing.

  Specifically, first, the refocus processing unit 156 translates the first corrected viewpoint image Ai along the principal ray angle θa to the virtual imaging plane 610 and sets the second corrected viewpoint image Bi to the principal ray angle θb. A process of translating along the virtual imaging plane 610 along the line is performed. Next, the refocus processing unit 156 performs a process of adding the first modified viewpoint image Ai and the second modified viewpoint image Bi that have been translated.

  Translating the first modified viewpoint image Ai along the angle θa to the virtual imaging plane 610 corresponds to shifting the first modified viewpoint image Ai by +0.5 pixels in the column direction. Further, translating the second corrected viewpoint image Bi to the virtual imaging plane 610 along the angle θb corresponds to shifting the second corrected viewpoint image Bi by −0.5 pixels in the column direction. Therefore, the first modified viewpoint image Ai and the second modified viewpoint image Bi are relatively shifted by +1 pixel, and Ai and Bi + 1 are added in correspondence with each other, thereby generating a refocus signal on the virtual imaging plane 610. . That is, the first modified viewpoint image Ai and the second modified viewpoint image Bi are shifted by an integer number of pixels and added for each pixel, thereby generating a refocused image on each virtual imaging plane corresponding to the integer shift amount. it can.

The refocus processing unit 156 shift-adds the first modified viewpoint image A and the second modified viewpoint image B according to the equation (8), thereby refocusing the image I on each virtual imaging plane according to the integer shift amount s. (J, i; s) is generated.
In the present embodiment, since the first corrected viewpoint image A and the second corrected viewpoint image B are configured in a Bayer array, the refocus processing unit 156 has a shift amount s = 2n (n: integer) that is a multiple of 2. Shift addition according to the equation (8) is performed for each color. That is, the refocus processing unit 156 generates the refocus image I (j, i; s) while maintaining the Bayer array of the image, and then demosaicing the generated refocus image I (j, i; s). Apply processing.

  Note that the refocus processing unit 156 first performs demosaicing processing on the first modified viewpoint image A and the second modified viewpoint image B as necessary, and the first modified viewpoint image and the second modified viewpoint image after the demosaicing processing. Shift addition processing may be performed using the viewpoint image. In addition, the refocus processing unit 156 generates an interpolated signal between the pixels of the first corrected viewpoint image A and the second corrected viewpoint image B as necessary, and generates a refocus image corresponding to the non-integer shift amount. It may be generated. Thereby, it is possible to generate a refocus image in which the position of the virtual imaging plane is changed with a more detailed granularity.

  Next, the sharpness process applied by the refocus processing unit 156 to generate a more effective refocus image will be described. The sharpness process is a process for performing contour coordination of a subject. As described above, in the refocus process, the first corrected viewpoint image A and the second corrected viewpoint image B are shift-added to generate a refocus image on the virtual imaging plane. Since the images of the first corrected viewpoint image A and the second corrected viewpoint image B are shifted by shift addition, the relative shift amount (also referred to as image shift amount) with respect to the image before the refocus processing can be known. The integer shift amount s by the refocus processing corresponds to this image shift amount. The refocus processing unit 156 can realize contour enhancement of the subject in the refocus image by performing sharpness processing on the area corresponding to the image shift amount s.

  In this embodiment, for example, unsharp mask processing is used as sharpness processing. FIG. 13 is a diagram for explaining the outline of the unsharp mask process. In the unsharp mask process, a blur filter is applied to a local region (original signal) centered on the pixel of interest, and a signal after application of the blur filter is generated. Then, the edge enhancement is realized by reflecting the difference between the pixel values before and after applying the blurring process on the pixel value of the target pixel.

The unsharp mask process for the pixel value P to be processed is calculated according to Equation (9). In Equation (9), P ′ (i, j) is the pixel value after application of the processing, R is the radius of the blur filter, and T (i, j) is the application amount (%). The size of the radius R of the blur filter is related to the wavelength of the frequency on the image to which the sharpness processing is to be applied. That is, as R is smaller, a finer pattern is emphasized, and as R is larger, a gentler pattern is emphasized.

  F (i, j, R) is a pixel value obtained by applying a blurring filter having a radius R to the pixel P (i, j). A known method such as Gaussian blur can be used for the blur filter. Gaussian blur is a process of averaging by applying weighting according to a Gaussian distribution according to the distance from the pixel to be processed, and a natural processing result can be obtained.

  The application amount T (i, j) is a value that changes the application amount of contour enhancement by unsharp mask processing according to the image shift distribution. The image shift amount at the position of each pixel is pred (i, j), and the shift amount s by refocus processing is set. The application amount T is increased in a region where | s-pred (i, j) | is a small value (for example, within one pixel of image deviation), that is, a region in a focused state on the virtual imaging plane. On the other hand, the application amount T is decreased in a region where | s-pred (i, j) | is a large value (for example, when the image shift amount is 3 pixels or more). By doing this, it is possible to enhance the outline in the focus position where the defocus amount is small or in the vicinity of the focus, and not to apply the unsharp mask process to the blurred region where the defocus amount is large. be able to. That is, the effect of moving the focus position by the refocus processing can be further emphasized.

Next, the refocusable range will be described. The refocusable range is a range of focus positions that can be changed by the refocus processing. FIG. 14 is a diagram schematically illustrating a refocusable range according to the present embodiment. If the allowable circle of confusion is δ and the aperture value of the imaging optical system is F, the depth of field at the aperture value F is ± F × δ. On the other hand, the effective aperture value F _{01 in} the horizontal direction of the first pupil partial region 501 (or the second pupil partial region 502) narrowed by being divided into N _H × N _V (for example, 2 × 1) (or F ₀₂ ) becomes darker as F ₀₁ = N _H × F (or F02 = NH × F). The effective depth of field for each first modified viewpoint image (or second modified viewpoint image) is ± N _H × F × δ, which is N _H times deeper, and the focus range is widened N _H times. That is, within the range of the effective depth of field “± N _H × F × δ”, a focused subject image is acquired for each first modified viewpoint image (or second modified viewpoint image). Therefore, the refocus processing unit 156 performs imaging by the refocus processing that translates and adds the first corrected viewpoint image (second corrected viewpoint image) along the principal ray angle θa (or θb) shown in FIG. Later, the focus position can be readjusted (refocused).

The defocus amount d from the imaging surface that can readjust the focus position (refocus) after shooting is limited. The refocusable range of the defocus amount d is approximately the range of Expression (10).

  The allowable circle of confusion δ is defined by, for example, δ = 2 · ΔX (the reciprocal of the Nyquist frequency 1 / (2ΔX) of the pixel period ΔX). As described above, by calculating the refocusable range, it is possible to correspond to the operable range when the focus position is changed (refocused) by the user operation. Further, since the light beam (subject) that can be focused by the refocus processing can be grasped in advance, for example, imaging of the state of the imaging optical system or the like so that the predetermined subject is included in the refocusable range. It is also possible to take a picture again by controlling the conditions.

  Next, viewpoint movement processing will be described. The viewpoint movement process is a process executed by the viewpoint change processing unit 155 in order to reduce the blur due to the non-main subject when the non-main subject is blurred on the near side. FIG. 15 is a diagram for explaining the outline of the viewpoint movement process. In FIG. 15, the image sensor 107 is arranged on the imaging surface 600, and the exit pupil of the imaging optical system is divided into a first pupil partial region 501 and a second pupil partial region 502, as in FIG. The An image that has passed through the first pupil partial region 501 becomes a first viewpoint image, and an image that has passed through the second pupil partial region 502 becomes a second viewpoint image. The viewpoint movement is performed using a plurality of viewpoint images acquired by an image sensor having a plurality of photoelectric conversion units. In the present embodiment, viewpoint movement is performed using the first viewpoint image and the second viewpoint image to generate a composite image.

  FIG. 15A shows an example in which a focused image p1 of the main subject q1 is overlaid with a blurred image Γ1 + Γ2 of the subject q2 in front, and perspective conflict (front blurring on the main subject) occurs in the captured image. It is. The example shown in FIG. 15A is divided into a light beam that passes through the first pupil partial region 501 and a light beam that passes through the second pupil partial region 502 of the imaging optical system. ) And FIG. 10 (C).

  In FIG. 15B, the light beam from the main subject q1 passes through the first pupil partial region 501 and forms an image p1 in a focused state. On the other hand, the light beam from the subject q2 on the near side passes through the first pupil partial region 501 and spreads in the blurred image Γ1 in the defocused state. Each light flux is received by the first sub-pixel 201 of each pixel of the image sensor 107, and a first viewpoint image is generated. As shown in FIG. 15B, in the first viewpoint image, the image p1 of the main subject q1 and the blurred image Γ1 of the subject q2 in front are received without overlapping.

  In FIG. 15C, the light beam from the main subject q1 passes through the second pupil partial region 502 and forms an image p1 in a focused state. On the other hand, the light beam from the subject q2 on the near side passes through the second pupil partial region 502 and spreads in the blurred image Γ2 in the defocused state. Each light beam is received by the second sub-pixel 202 of each pixel of the image sensor 107, and a second viewpoint image is generated. As shown in FIG. 15C, in the second viewpoint image, the image p1 of the main subject q1 and the blurred image Γ2 of the subject q2 in front are overlapped and received.

  In FIGS. 15B and 15C, a region near the image p1 of the main subject q1 is set as a predetermined region. In FIG. 15B, the range of the blurred image Γ1 of the subject q2 on the near side in the predetermined area is narrow, and there is little reflection. Therefore, the contrast evaluation value in the predetermined area becomes large. On the other hand, in FIG. 15C, the range of the blurred image Γ1 of the subject q2 on the near side in the predetermined area is wide and there are many reflections. Therefore, the contrast evaluation value in the predetermined area becomes large. For this reason, the contrast evaluation value in the predetermined area becomes small. Therefore, in the predetermined area, the weight of the first viewpoint image with little overlap between the image p1 and the blurred image Γ1 is increased, and the weight of the second viewpoint image with much overlap between the image p1 and the blurred image Γ2 is decreased and added. Thereby, the front blurring with respect to the main subject can be reduced.

  Next, processing in which the viewpoint change processing unit 155 superimposes the first viewpoint image and the second viewpoint image using weights will be described. The viewpoint change processing unit 155 inputs the first viewpoint image A (j, i) and the second viewpoint image B (j, i) described above.

First, the viewpoint change processing unit 155 sets a predetermined area R = [j1, j2] × [i1, i2] for moving the viewpoint and a boundary width σ of the predetermined area. Then, a table function T (j, i) corresponding to the boundary width σ between the predetermined region R and the predetermined region is calculated according to the equation (11).
The value of the table function T (j, i) is 1 inside the predetermined region R, 0 outside the predetermined region R, and continuously changes from approximately 1 to 0 with the boundary width σ of the predetermined region R. Note that the viewpoint change processing unit 155 may set the predetermined area to be circular or any other shape as necessary. A plurality of predetermined areas and a plurality of boundary widths may be set.

Next, the viewpoint change processing unit 155 calculates a weighting coefficient. As the real coefficient w (−1 ≦ w ≦ 1), the first weighting coefficient Wa (j, i) of the first viewpoint image A (j, i) is calculated by the equation (12A). The second weighting coefficient Wb (j, i) of the second viewpoint image B (j, i) is calculated by the equation (12B).
When the depth of field is corrected by increasing the addition ratio of the first viewpoint image A (j, i) in the predetermined area, the setting is performed in the range of −1 ≦ w <0. On the other hand, when the depth of field is corrected by increasing the addition ratio of the second viewpoint image B (j, i), the setting is made in the range of 0 <w ≦ 1. In some cases, w = 0 is set to W1≡W2≡1 and the depth of field is not corrected.

Next, the viewpoint change processing unit 155 generates an output image I (j, i). The viewpoint change processing unit 155 weights the first viewpoint image A (j, i) weighted with the first weighting coefficient Wa (j, i) and the second weighting coefficient Wb (j, i) according to the equation (13). The second viewpoint image B (j, i) is added to generate an output image I (j, i).

Further, the viewpoint change processing unit 155 may generate the output image Is (j, i) according to the formula (14A) or the formula (14B) in combination with the refocus processing using the shift amount s.
The generated output image I _s (j, i) is an image in which the viewpoint is moved and an image in which the focus position is readjusted (refocused).

In this way, using a weighting factor that continuously changes in accordance with the region of the output image, a plurality of viewpoint images are multiplied and combined to generate an output image. In a predetermined area, the first weighting factor W _a of the first viewpoint image in which the overlap between the image p1 and the blurred image Γ1 is small is made larger than the second weighting factor W _b , thereby generating an output image. An image with reduced blur coverage can be generated. That is, in order to reduce the front blur, the viewpoint change processing unit 155 reduces the weighting coefficient of the viewpoint image in which the closest subject is photographed in the widest range or the closest subject is the narrowest in a predetermined area. What is necessary is just to enlarge the weighting coefficient of the viewpoint image image | photographed in the range. Similarly, the viewpoint change processing unit 155 reduces the weight coefficient of the viewpoint image with the smallest contrast evaluation value or increases the weight coefficient of the viewpoint image with the largest contrast evaluation value in the predetermined area in order to reduce the front blur. do it.

  Note that the viewpoint change processing unit 155, as necessary, does not change the blurring shape of the imaging optical system in a region other than the predetermined area where the viewpoint movement process is not performed. The output image may be generated by adding the coefficients and the second weighting coefficient substantially evenly. Further, although a method for generating an output image in which the weighting coefficient (that is, the addition ratio) is changed according to the user's designation will be described later, the user may designate a predetermined area for performing the viewpoint movement process.

  Next, pupil shift in the peripheral image height of the image sensor 107 will be described. FIG. 16 is a schematic explanatory diagram of pupil shift at the peripheral image height of the image sensor. Specifically, FIG. 16 shows a first pupil partial region 501 received by the first subpixel 201 of each pixel arranged in the image sensor, a second pupil partial region 502 received by the second subpixel 202, and imaging. The relationship with the exit pupil 400 of the optical system is shown.

  FIG. 16A shows a case where the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the image sensor 107 are the same. In this case, the exit pupil 400 of the imaging optical system is divided into pupils approximately equally by the first pupil partial region 501 and the second pupil partial region 502 regardless of the central image height or the peripheral image height.

  On the other hand, FIG. 16B shows a case where the exit pupil distance Dl of the imaging optical system is shorter than the set pupil distance Ds of the image sensor 107. In this case, at the peripheral image height, the exit pupil 400 of the imaging optical system is non-uniformly divided by the first pupil partial region 501 and the second pupil partial region 502. In the example of FIG. 16B, the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is smaller (brighter) than the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502. Value. At the opposite image height (not shown), conversely, the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502. Larger (darker) value.

  FIG. 16C shows a case where the exit pupil distance Dl of the imaging optical system is longer than the set pupil distance Ds of the image sensor 107. Also in this case, at the peripheral image height, the exit pupil 400 of the imaging optical system is non-uniformly divided by the first pupil partial region 501 and the second pupil partial region 502. In the example of FIG. 16C, the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is larger (darker) than the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502. Value. At the opposite image height (not shown), conversely, the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502. Smaller (brighter) value.

  That is, as the pupil division becomes nonuniform at the peripheral image height due to the pupil shift, the effective F values of the first viewpoint image and the second viewpoint image also become nonuniform. For this reason, one of the first viewpoint image and the second viewpoint image has a larger spread of blur, and the other blur has a smaller spread. Therefore, the viewpoint change processing unit 155 reduces the weight coefficient of the viewpoint image having the smallest effective aperture value or the viewpoint image having the largest effective aperture value in a predetermined area of the output image as necessary. It is desirable to make it the largest. By performing such viewpoint movement processing, it is possible to reduce front blurring on the main subject.

  Next, depth expansion processing by the viewpoint change processing unit 155 will be described. As described with reference to FIGS. 15 and 16, an image that has passed through the first pupil partial region 501 is a first viewpoint image, and an image that has passed through the second pupil partial region 502 is a second viewpoint image. Since each viewpoint image is an image obtained by passing through half of the original pupil region, in the case of a pupil division region divided in two in the horizontal direction, the aperture diameter in the horizontal direction is halved. For this reason, the depth of field in the horizontal direction is quadrupled. In addition, since this embodiment does not have a configuration in which pupils are divided in the vertical direction, there is no change in the depth of field in the vertical direction. Therefore, the first viewpoint image or the second viewpoint image has a depth of field that is twice as long as the vertical and horizontal average of the depth of field of the image (A + B image) obtained by combining the first viewpoint image and the second viewpoint image. It becomes the image which has.

  The viewpoint change processing unit 155 can generate an image with an increased depth of field by generating a composite image by changing the addition ratio of the first viewpoint image or the second viewpoint image to other than 1: 1. . Further, the viewpoint change processing unit 155 applies an unsharp mask process using the contrast distribution and the image shift distribution to the image in which the addition ratio of the first viewpoint image or the second viewpoint image is changed, so that the object scene is changed. A composite image in which the depth is enlarged and the contour is emphasized can be generated. Further, in the depth expansion process, as in the viewpoint movement process, a predetermined area may be processed according to the user's designation. Note that the above-described development processing is applied to the composite image output from the viewpoint change processing unit 155, and the image to which the development processing is applied is output from the image processing circuit 125.

  In addition, the viewpoint changing process using the first viewpoint image and the second viewpoint image reduces the foreground blurring that causes the main subject to be hidden due to large blurring of the foreground located in front of the main subject (blur adjustment processing). Is possible. In addition, the viewpoint changing process using the first viewpoint image and the second viewpoint image separates an unnecessary component such as a ghost in which a part of light incident on the imaging optical system appears in the captured image from a parallax component, and the ghost. The process which reduces only the unnecessary component (unnecessary component reduction process) is possible.

  Next, FIG. 17 to FIG. 19 show how to select and save information necessary for image processing such as refocus processing, viewpoint movement processing, blur adjustment, and unnecessary component reduction processing based on shooting conditions, contrast information, and processing information. It explains using. FIG. 17 is a flowchart showing a process for recording information. FIG. 18 is a flowchart showing processing for determining information to be recorded. FIG. 19 is a flowchart illustrating processing for compressing a viewpoint image. Note that the processing in FIGS. 17 to 19 is executed by the image processing circuit 125 and the CPU 121.

  In step S <b> 100, the image sensor driving circuit 124 controls the image sensor 107 according to the set shooting condition, and acquires a viewpoint image. The imaging conditions include an aperture value, shutter speed, ISO sensitivity, focal length, and subject distance. The exposure may be set based on a value calculated by automatically setting the exposure according to the imaging condition and the brightness of the subject, or the value may be set manually by the user.

  In step S200, it is determined whether or not to record the viewpoint image in the flash memory 133 from the shooting conditions set in step S100. Here, a method for determining whether to record the viewpoint image in the flash memory 133 will be described with reference to FIG.

  In step S201, it is determined whether to record the viewpoint image in the flash memory 133 based on the shooting conditions. When the effect obtained by the image processing using the viewpoint image is large, the viewpoint image is recorded, and when the effect is small, the viewpoint image is recorded. Whether or not the effect of the image processing is great is determined based on the shooting conditions. The imaging conditions here are, for example, an aperture value, ISO sensitivity, focal length, and the like.

  First, a determination method using the aperture value will be described. The larger the aperture value, the deeper the depth of field and the smaller the parallax between the first viewpoint image and the second viewpoint image. If the parallax between the first viewpoint image and the second viewpoint image is small, the effect of image processing (particularly, blur adjustment processing and unnecessary component reduction processing) is small. Therefore, when the aperture value is equal to or greater than a predetermined value, it is determined that the viewpoint image is not recorded. On the other hand, when the aperture value is smaller than a predetermined value, it is determined that the viewpoint image is recorded.

  Next, a determination method using ISO sensitivity will be described. As the ISO sensitivity increases, the S / N decreases (that is, deteriorates), and changes before and after image processing become difficult to visually recognize. A small change before and after the image processing means that the effect of the image processing is small. Therefore, when the ISO sensitivity is equal to or higher than a predetermined value, it is determined that the viewpoint image is not recorded. On the other hand, when the ISO sensitivity is lower than a predetermined value, it is determined that the viewpoint image is recorded.

  Finally, a determination method based on the focal length will be described. The actual distance converted value of the defocus amount in the refocus processing is proportional to the vertical magnification (the square of the horizontal magnification), and the short distance subject has a small actual distance converted value of the defocus change amount due to the refocus processing. Is difficult to see. Also, at a long distance (for example, at infinity), the actual distance converted value of the defocus change amount by the refocus process is large, and therefore the effect of appropriate image processing (particularly, refocus process) according to the subject size cannot be applied. . Therefore, when the focal length is out of the predetermined focal length range, it is determined that the viewpoint image is not recorded. On the other hand, when the focal length is within a predetermined focal length range, it is determined to record the viewpoint image. Note that a predetermined focal length range in which the effect of the defocus change amount by the refocus processing can be sufficiently obtained is set for each focal length of the lens.

  If it is determined in step S201 that a viewpoint image is to be recorded, the process proceeds to step S201. On the other hand, if it is determined not to record the viewpoint image, the process proceeds to step S206. In step S201, the aperture value, ISO sensitivity, and focal length have been described as shooting conditions. However, the present invention is not limited to these. For example, the shading processing unit 153 determines whether to record the viewpoint image in the flash memory 133 when the change in the light amount due to the image height of the first viewpoint image and the second viewpoint image has not been corrected. Can do. Further, it may be determined whether or not to record the viewpoint image in the flash memory 133 based on the camera shake or panning operation of the photographer or the focus state of the lens.

  In step S202, contrast information of the captured image that is parallax-related information is generated. The contrast information is information corresponding to the contrast distribution. The contrast information is generated by synthesizing the first viewpoint image and the second viewpoint image. After the contrast information is generated, the process proceeds to step S203.

  In step S203, image shift information that is parallax-related information is generated. The image shift information is information corresponding to the image shift amount distribution. The image shift information is generated from the viewpoint image. After generating the image shift information, the process proceeds to step S204.

  In step S204, image shift / contrast information that is parallax-related information is generated. The image shift / contrast information is information obtained by integrating the contrast information generated in step S202 and the image shift information generated in step S203. After the image shift / contrast information is generated, the process proceeds to step S205.

  In step S205, based on the image shift / contrast information generated in step S204, it is determined whether or not the viewpoint image is to be recorded in the flash memory 133. Whether or not to record the viewpoint image is determined depending on whether or not there is a region having a predetermined contrast or more in the image shift / contrast information at a predetermined value (the size of the predetermined region) or more. If there is an area of a predetermined contrast or more in the image shift / contrast information, the image processing is highly effective, and it is determined to record the viewpoint image. On the other hand, if the region having a predetermined contrast or higher is less than the predetermined value, it is determined that the viewpoint image is not recorded because the effect of the image processing is small.

  When there is a region having a predetermined contrast or more in the image shift / contrast information at a predetermined value or more, that is, when recording a viewpoint image, contrast information or image shift / contrast information may be further recorded in the flash memory 133. Good. If it is determined in step S205 that a viewpoint image is to be recorded, the process proceeds to step S206. On the other hand, if it is determined not to record the viewpoint image, the process proceeds to step S208.

  In step S206, image processing is performed on the viewpoint image to generate a composite image. After the generation, the process proceeds to step S207. The image processing includes, for example, refocus processing, viewpoint movement processing, blur adjustment, unnecessary component reduction processing, and the like.

  In step S207, it is determined whether to record the viewpoint image based on the image processing. In the image processing of the viewpoint image performed in step S206, at least one of predetermined image processing using the viewpoint image, for example, refocus processing, viewpoint movement processing, blur adjustment, and unnecessary component reduction processing has been performed. For example, it is determined that the viewpoint image is not recorded. Even when the above-described image processing is not performed, it may be determined that the viewpoint image is not recorded when the predetermined image processing using the viewpoint image is not performed in the image processing after step S207.

  On the other hand, even when the predetermined image processing using the viewpoint image is performed in the image processing in step S206, the predetermined image processing using the viewpoint image is performed in the image processing after step S207. May determine that a viewpoint image is to be recorded. Further, when performing refocus processing using the viewpoint image in the image processing after step S207, the viewpoint image may not be recorded, and the captured image and the image shift / contrast information may be stored. In step S207, it may be determined whether or not to record a viewpoint image according to a user instruction. After determining whether or not to record the viewpoint image, the process proceeds to step S208.

  In step S208, information to be recorded in the flash memory 133 is determined. Information to be recorded in the flash memory 133 is determined based on the determinations in step S201, step S205, and step S207. If it is determined to record the viewpoint image in each step, for example, the first viewpoint image or the second viewpoint image, the captured image obtained by combining the first viewpoint image and the second viewpoint image, and the image shift generated in step S204. • Record contrast information. On the other hand, when it is determined not to record the viewpoint image, only the captured image obtained by combining the first viewpoint image and the second viewpoint image is recorded. After determining information to be recorded in the flash memory 133, the present subroutine is terminated, and the process proceeds to step S300 of the main routine.

  Regardless of whether or not to record the viewpoint image, the image shift / contrast information generated in step S204 may be recorded. By recording the image shift / contrast information, for example, even when it is determined in step S201, step S205, or step S207 that the viewpoint image is not recorded, image processing using the captured image and the image shift / contrast information is performed. It becomes possible to do.

  Returning to the description of FIG. In step S300, if it is determined in step S200 that the viewpoint image is to be recorded, it is determined whether or not the viewpoint image to be recorded in the flash memory 133 is to be compressed. Here, determination as to whether or not to perform viewpoint image compression in step S300 and compression processing will be described with reference to FIG. In step S300, the user may select whether to compress the viewpoint image.

  In step S301, it is determined whether the ISO sensitivity, which is a photographing condition at the time of photographing a viewpoint image, is equal to or higher than a predetermined value. When the ISO sensitivity is equal to or higher than a predetermined value, the S / N becomes small (deteriorates), so the S / N is increased by compressing the viewpoint image by horizontal and vertical addition. If the ISO sensitivity is a predetermined value (for example, ISO 6400) or more, the process proceeds to S303. On the other hand, if the ISO sensitivity is not equal to or higher than the predetermined value, the process proceeds to S302.

  In S302, using the contrast information generated in S202, it is determined whether the contrast is equal to or higher than a predetermined value. If the contrast is not equal to or higher than a predetermined value, it can be determined that the viewpoint image does not have a high-frequency subject. Therefore, the viewpoint image may be compressed by horizontal and vertical addition. If the contrast is not equal to or greater than the predetermined value, the process proceeds to S303. On the other hand, if the contrast is equal to or higher than a predetermined value, the viewpoint image compression determination is terminated without compressing the viewpoint image, and the process returns to the main flow.

  In S303, the viewpoint image is compressed for each RGB in the horizontal or vertical direction to reduce the number of pixels. By compressing and recording the viewpoint image, it is possible to generate an image shift distribution with improved S / N and image shift accuracy when generating the image shift distribution in post-processing. Further, the data capacity can be reduced by compressing the viewpoint image. In the present embodiment, the viewpoint image compression method has been described by horizontal and vertical addition, but low-pass filter processing or demosaicing processing may be used.

  You may change the horizontal or vertical addition number at the time of compressing a viewpoint image, ie, the compression rate at the time of compression, according to imaging conditions and the value of contrast. For example, as the ISO sensitivity is higher, the number of horizontal and vertical addition pixels is increased and compression is performed at a higher compression rate. Further, as the contrast value is higher, the number of horizontal and vertical added pixels is increased and compression is performed at a higher compression rate. Furthermore, the larger the aperture value, the larger the number of horizontal and vertical addition pixels may be, and compression may be performed at a high compression rate.

  As shown in FIG. 19, the ISO sensitivity is set as a determination condition having a higher priority than the contrast value. When the ISO sensitivity is high, even when the contrast value is high, the S / N is small, so that the accuracy of image shift is lowered. Therefore, when the ISO sensitivity is high, it is better to perform the correlation calculation after improving the S / N by performing horizontal or vertical addition of the viewpoint images regardless of the contrast value. After the compression processing in step S303 is completed, the viewpoint image compression determination is terminated and the process returns to the main flow.

  In step S400, the recording information prepared in steps S200 and S300 is recorded in the flash memory 133. After recording, the main flow is terminated.

  In the above-described embodiment, the example in which the viewpoint image is not recorded depending on the shooting conditions and the result of the image processing has been described. Alternatively, the compression rate may be controlled. That is, as a result of the determination of each condition in step S201, step S205, and step S207, the CPU 121 controls the presence or absence of compression or the compression rate, not the presence or absence of recording. At this time, the branch where the viewpoint image is not recorded (NO in each determination flow) corresponds to the branch where the viewpoint image is not compressed or the branch where the compression rate is set higher. A branch in which the viewpoint image is recorded (YES in each determination flow) corresponds to a branch in which the viewpoint image is not compressed or a lower compression rate is set.

  As described above, according to the present embodiment, the data to be recorded is determined and compressed according to the shooting conditions and the execution of image processing, so that the viewpoint image can be recorded with an appropriate capacity. Therefore, it is possible to suppress the data volume recorded in the flash memory 133.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

121 CPU
125 Image Processing Circuit 151 Image Acquisition Unit 154 Operation Information Acquisition Unit 155 Viewpoint Change Processing Unit 156 Refocus Processing Unit 161 Compression Unit

Claims (19)

Acquisition means for acquiring a plurality of viewpoint images;
Shooting condition acquisition means for acquiring shooting conditions when shooting the viewpoint image;
Compression means for compressing the viewpoint image to be recorded in the recording means,
The image pickup apparatus, wherein the compression unit compresses the viewpoint image according to the shooting condition.   The imaging apparatus according to claim 1, wherein the photographing condition is an ISO sensitivity or an aperture value.   The imaging apparatus according to claim 1, wherein the photographing condition is ISO sensitivity, and the compression unit compresses the viewpoint image when the ISO sensitivity is equal to or higher than a predetermined value.   The imaging apparatus according to claim 1, wherein the compression unit changes a compression rate for compressing the viewpoint image in accordance with ISO sensitivity.   The imaging apparatus according to claim 1, wherein the compression unit changes a compression rate for compressing the viewpoint image according to an aperture value. A calculation means for calculating contrast information corresponding to a contrast distribution based on the viewpoint image;
6. The imaging apparatus according to claim 1, wherein the compression unit compresses the viewpoint image when a region having a predetermined contrast or higher in the contrast information is less than a predetermined value. 7. .   The imaging apparatus according to claim 6, wherein the compression unit changes a compression rate for compressing the viewpoint image in accordance with a value of the contrast information. Acquisition means for acquiring a plurality of viewpoint images;
Shooting condition acquisition means for acquiring shooting conditions when shooting the viewpoint image;
Determining means for determining data to be recorded in the recording means,
The determination unit determines whether to record the viewpoint image in the recording unit according to the shooting condition.   The imaging apparatus according to claim 8, wherein the photographing condition is any one of an aperture value, an ISO sensitivity, and a subject distance.   The imaging apparatus according to claim 8 or 9, wherein the photographing condition is an aperture value, and the determination unit determines that the viewpoint image is not recorded when the aperture value is equal to or greater than a predetermined value. .   10. The imaging apparatus according to claim 8, wherein the photographing condition is ISO sensitivity, and the determination unit determines not to record the viewpoint image when the ISO sensitivity is equal to or higher than a predetermined value. .   The imaging apparatus according to claim 8 or 9, wherein the shooting condition is a subject distance, and the determination unit determines not to record the viewpoint image when the subject distance is outside a predetermined range. . Image processing means for performing image processing using the viewpoint image,
The imaging apparatus according to claim 8, wherein the determination unit determines whether to record the viewpoint image in the recording unit according to the image processing.   The determination means determines that the viewpoint image is not to be recorded when any of refocus processing, viewpoint movement processing, blur adjustment, and unnecessary component reduction processing is performed as the image processing. The imaging device according to 13.   The determination unit, when performing any of refocus processing, viewpoint movement processing, blur adjustment, and unnecessary component reduction processing as the image processing, determines to record the viewpoint image. The imaging device according to 13. A calculation means for calculating contrast information corresponding to a contrast distribution based on the viewpoint image;
The said determination means determines that the said viewpoint image is not recorded when the area | region more than predetermined | prescribed contrast in the said contrast information is less than predetermined | prescribed value, It is characterized by the above-mentioned. Imaging device.   17. The imaging according to claim 16, wherein the determination unit determines to record the contrast information in the recording unit when a region having a predetermined contrast or more in the contrast information is a predetermined value or more. apparatus. Control means for the imaging device,
An acquisition step of acquiring a plurality of viewpoint images;
A shooting condition acquisition step of acquiring shooting conditions when shooting the viewpoint image;
A compression step of compressing the viewpoint image to be recorded in the recording means,
In the compression step, the viewpoint image is compressed according to the photographing condition. A method for controlling an imaging apparatus,
An acquisition step of acquiring a plurality of viewpoint images;
A shooting condition acquisition step of acquiring shooting conditions when shooting the viewpoint image;
A determination step for determining data to be recorded in the recording means,
In the determining step, it is determined whether to record the viewpoint image in the recording unit according to the photographing condition.