JP2017216649A - Imaging device, imaging apparatus and imaging signal processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of performing image generation and focus detection based on signals obtained by three or more kinds of output conditions in photography of a minimum requirement frequency.SOLUTION: An imaging device 102 has a plurality of pixel units and can perform signal processing of focus detection and image generation by acquiring signals from respective pixel units. Each pixel unit respectively includes first and second photoelectric conversion sections and is capable of independently set an output condition of a signal. A video signal processing unit 104 acquires signals of first to third output conditions in first and second photography and performs dynamic range expansion processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging device and an imaging apparatus capable of acquiring signals from a plurality of pixel units and performing image generation and focus detection.

  There is a process of generating an image (hereinafter also referred to as an HDR image) having an apparently wider dynamic range than the dynamic range of an image that can be acquired by one shooting in the imaging apparatus. A technique for generating an HDR image by regularly arranging pixels having different aperture ratios constituting an image sensor is disclosed (Patent Document 1). In addition, a technique for generating an HDR image using an imaging element having a pupil division pixel portion in which a plurality of pixels are assigned to one microlens is disclosed (Patent Document 2).

  On the other hand, in the configuration of the imaging apparatus that can acquire images of different viewpoints by one shooting, a pair of subject images formed by light beams that have passed through different pupil regions can be acquired. A technique for performing focus detection from a pair of acquired subject images by correlation calculation is disclosed (Patent Document 3).

JP 2003-179819 A Japanese Patent Laying-Open No. 2015-144416 JP 2013-072906 A

  In the apparatus disclosed in Patent Document 1, one HDR image is generated from images acquired with a plurality of imaging frames and having different imaging acquisition periods. Therefore, for example, when trying to obtain an HDR composite image under three different exposure conditions of proper exposure, short-second exposure, and long-second exposure, it is necessary to shoot three images, and the number of photographic images increases, and subject blurring tends to occur. Become. In the apparatus disclosed in Patent Document 2, an HDR image is generated by a combination of pixels having different aperture ratios of pixels with respect to the microlens. In the case of this method, phase difference detection is performed also using a pixel from a pixel in which incident light is blocked and the pixel signal is greatly reduced, or a signal from a saturated pixel. Therefore, there may be a case where sufficient accuracy of phase difference detection cannot be obtained. In addition, the apparatus disclosed in Patent Document 3 performs focus detection using an imaging element having a pupil-divided pixel unit in which a plurality of pixels are assigned to one microlens, but mentions processing for generating an HDR image. It has not been.

  An object of the present invention is to provide an image pickup device and an image pickup apparatus capable of performing image generation and focus detection based on signals obtained under three or more kinds of output conditions with the minimum number of times of shooting. .

  An image sensor according to an embodiment of the present invention is an image sensor that can acquire signals from a plurality of pixel units and perform signal processing for focus detection and image generation, and includes first and second pixel units. Setting means for setting signal output conditions for the first and second photoelectric conversion units included in the first and second photoelectric conversion units, and the first and second pixel units obtained from the first and second pixel units for the first and second imaging, Or signal processing means for performing signal processing of the image generation based on the pixel signal of the third output condition.

  According to the present invention, it is possible to perform image generation and focus detection based on signals obtained under three or more types of output conditions with the minimum number of times of photographing.

1 is a block diagram illustrating a system configuration of an imaging apparatus according to the present invention. It is a block diagram which shows the structure of the image pick-up element based on this invention. It is a figure which shows the pixel structure and arrangement | sequence of an image pick-up element which concern on 1st Example of this invention. It is a schematic diagram which shows the relationship between an exit pupil and the light reception of an image pick-up element. It is a circuit block diagram of an image sensor. It is a figure which shows the pattern of the exposure conditions of a pixel. It is a figure which shows the pattern of the exposure conditions of a pixel. It is a figure explaining a dynamic range expansion process. It is a flowchart explaining the operation | movement process in a present Example. FIG. 10 is a diagram illustrating a pattern of pixel exposure conditions in Example 2.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The imaging apparatus of this embodiment can be applied to an electronic still camera with a moving image function, a video camera, or the like.
(Outline of the embodiment)
First, an overview of an imaging device, an imaging apparatus, and an imaging signal processing method according to an embodiment of the present invention will be described, and then each example will be described in detail. The present invention can be applied to an image sensor in which a pixel unit includes a plurality of photoelectric conversion units and an image pickup apparatus including the image sensor. Each of the plurality of pixel units has a first photoelectric conversion unit and a second photoelectric conversion unit, and output conditions of pixel signals are set independently. The pixel signal output conditions are, for example, conditions such as the ISO sensitivity of the pixel, exposure time, optical aperture value, gain amplifier gain, and a combination of a plurality of conditions. The exposure setting value can be changed arbitrarily. Is possible. The signal processing unit in the imaging device or the imaging apparatus performs first and second signal processing described below.

  The first signal processing is performed using the signals of the first and second photoelectric conversion units set to the first to third output conditions. In the following embodiments, a dynamic range expansion process (HDR process) of an image signal will be described as a specific example of signal processing for image generation. For example, a wide dynamic range image signal can be generated by combining a plurality of signals having different output conditions. In the second signal processing, the signals of the first and second photoelectric conversion units set to any one of the first to third output conditions are processed. In the following embodiments, focus detection processing of an imaging optical system will be described as a specific example. A focus detection signal is generated by the second signal processing, and focus adjustment control is enabled. The first signal processing is executed based on a plurality of output condition signals obtained in a plurality of shooting frames related to the first and second shootings. Further, the second signal processing is executed based on signals having the same output condition obtained in a single frame.

  As for the control mode of the image sensor, there are a first control mode in which only focus detection calculation is performed and a second control mode in which focus detection calculation and HDR processing are performed. The control unit changes the content of the signal processing for the signal acquired from the imaging unit by switching the control mode. For example, when the focus detection calculation of the phase difference detection method is performed in the first control mode, the phase difference is detected from images (parallax images) having parallax output from a plurality of photoelectric conversion units in the pixel unit. When an A image signal is acquired from the output of the first photoelectric conversion unit and a B image signal is acquired from the output of the second photoelectric conversion unit, a correlation operation is performed on the A image signal and the B image signal, and the calculation result From this, the amount of defocus is calculated. In the second control mode, the calculation of the phase difference detection method and the HDR process are performed on the image signal acquired by the imaging operation. The embodiment of the present invention is not limited to the phase difference detection method. The present invention can be applied to focus detection processing and contrast detection processing based on refocus processing by shift addition, or a combined method combining a plurality of detection methods.

  Regarding the configuration of the pixel unit, a photoelectric conversion unit that is divided into two in the horizontal direction, which is the pupil division direction, is illustrated. Of course, the present invention can be applied to a photoelectric conversion unit that is divided into two in the horizontal direction and in the vertical direction. As an embodiment of the present invention, there is an example in which the number of divisions is further increased to 6, 9 or the like. Moreover, the shape of the photoelectric conversion unit is not limited to a rectangle, and can be applied to an embodiment in which the photoelectric conversion unit is designed in a polygon such as a hexagon.

(First embodiment)
The imaging apparatus according to the present exemplary embodiment can generate two or more stages of HDR images while performing phase difference detection from images acquired in the first and second imaging. The image pickup apparatus of this embodiment uses an image pickup element in which a plurality of pixel portions are arranged in a two-dimensional array. Each pixel unit includes two photoelectric conversion units that respectively receive light passing through different pupil partial regions in the imaging optical system.

  With reference to FIG. 1, the configuration of the imaging apparatus will be described. The optical barrel 101 includes a plurality of lenses, a focus mechanism unit 1011, a zoom mechanism unit 1012, a diaphragm mechanism unit 1013, a shutter mechanism unit 1014, and the like. The light from the subject passing through the lens in the optical barrel 101 is adjusted to an appropriate amount of light by the aperture mechanism unit 1013 and imaged on the imaging surface on the imaging element 102. The focus mechanism unit 1011 is driven based on a control signal from the control unit 107, and performs focus adjustment by movement control of a focus lens for focus adjustment. The zoom mechanism unit 1012 is driven based on a control signal from the control unit 107, and changes the angle of view by controlling the movement of the zoom lens according to the zoom operation of the user. The aperture mechanism unit 1013 and the shutter mechanism unit 1014 are driven based on a control signal from the control unit 107, and change the aperture value and the shutter time, respectively.

  In the imaging element 102, an imaging operation such as exposure, signal readout, and reset is performed in accordance with a control signal from the control unit 107, and a corresponding image signal is output. Details thereof will be described later. The video signal processing unit 104 acquires an image signal from the image sensor 102 and corrects pixel defects and signal variations caused by the image sensor 102, HDR processing, white balance adjustment processing, gamma processing, color Various signal processing such as correction processing is performed to output a video signal. The video signal processing unit 104 also detects the brightness of each region for exposure control of the image sensor 102 and performs AE (automatic exposure) processing for calculating appropriate exposure.

  The compression / decompression unit 105 operates under the control of the control unit 107, and performs compression coding processing on the video signal from the video signal processing unit 104 in a still image data format of a predetermined method. For example, there is a JPEG (Joint Photographic Coding Experts Group) method. The compression / decompression unit 105 performs decompression / decoding processing on the encoded data of the still image supplied from the image recording unit 111 via the control unit 107. Furthermore, it is possible to execute compression encoding / decompression decoding processing of moving images by the MPEG (Moving Picture Experts Group) method or the like.

  The phase difference signal processing unit 106 performs phase difference detection processing based on a phase difference signal that is a pixel signal corresponding to a different pupil plane from the image sensor 102 in accordance with a control signal from the control unit 107. When calculating the phase difference detection signal, output level correction is performed when using a signal that has a difference in output level due to factors other than the difference in pupil plane between pixel signals corresponding to different pupil planes. Then, the phase difference signal processing unit 106 performs phase difference detection processing. Since the generation process of the phase difference detection signal is known, the description thereof will be omitted and the output level correction process will be described later. The phase difference detection signal (image shift amount) calculated by the phase difference signal processing unit 106 is sent to the control unit 107.

  The control unit 107 is a microcontroller including, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 107 executes a program stored in a ROM or the like, and comprehensively controls each unit of the imaging apparatus. For example, the control unit 107 calculates a defocus amount indicating the focus state of the imaging optical system based on the acquired phase difference detection signal. The control unit 107 calculates a drive amount of the focus lens necessary for obtaining the in-focus state from the calculated defocus amount, and sends a drive control signal to the focus mechanism unit 1011. In accordance with a drive control signal from the control unit 107, the focus mechanism unit 1011 drives an AF (automatic focus adjustment) mechanism to move the focus lens to a target position.

  The light emitting unit 108 is a device that emits light to the subject when the exposure value of the subject is determined to be low by the AE processing of the video signal processing unit 104. The light emitting unit 108 is a strobe device using a xenon tube, an LED light emitting device, or the like.

  The operation unit 109 includes various operation keys such as a shutter release button, a lever, a dial, a touch panel, and the like. The operation unit 109 outputs a control signal corresponding to the input operation by the user to the control unit 107. The image display unit 110 includes a display device such as an LCD (Liquid Crystal Display) and an interface circuit for the device. The image display unit 110 generates a signal to be displayed on the display device from the video signal supplied from the control unit 107, supplies the signal to the display device, and displays an image on the screen.

  The image recording unit 111 includes, for example, a recording medium such as a portable semiconductor memory, an optical disc, an HDD (Hard Disk Drive), and a magnetic tape. The image recording unit 111 acquires the image data file encoded by the compression / decompression unit 105 from the control unit 107 and records it on a recording medium. Further, the image recording unit 111 reads designated data from the recording medium based on a control signal from the control unit 107 and outputs it to the control unit 107.

  Next, the configuration of the image sensor 102 in this embodiment will be described with reference to FIG. FIG. 2 is a diagram illustrating a configuration example of the image sensor 102. The image sensor 102 includes a pixel array unit 202 in which a plurality of pixel units 201 are arranged in a matrix on a semiconductor substrate using, for example, silicon (Si), and a peripheral circuit unit thereof. The peripheral circuit section includes a vertical drive circuit 203, a column signal processing circuit 204, a horizontal drive circuit 205, a timing control circuit 206, and the like.

  The pixel unit 201 includes a photodiode as a photoelectric conversion unit and a plurality of pixel transistors. The plurality of pixel transistors are, for example, МOS (metal oxide semiconductor) transistors such as a transfer transistor, an amplification transistor, a selection transistor, and a reset transistor.

  The vertical drive circuit 203 is configured by a shift register, for example. The vertical drive circuit 203 selects the pixel drive wiring 208, supplies a pulse for driving the pixel portion 201 to the selected pixel drive wiring 208, and drives the pixel portion 201 in units of rows. The vertical drive circuit 203 selectively scans each pixel unit 201 on the pixel array unit 202 in the vertical direction sequentially in units of rows. A pixel signal based on a signal charge generated according to the amount of incident light in the photoelectric conversion unit of each pixel unit 201 is supplied to the column signal processing circuit 204 through the vertical signal line 207.

  The column signal processing circuit 204 is arranged for each column of the pixel unit 201, and performs signal processing such as noise removal for each pixel column on the pixel signal output from the pixel unit 201 for one row. For example, the column signal processing circuit 204 performs signal processing such as CDS processing for removing fixed pattern noise peculiar to the pixel, amplification processing of the pixel signal of the pixel unit 201 output through the vertical signal line 207, and AD conversion. CDS is an abbreviation for “Correlated Double Sampling”.

  The horizontal drive circuit 205 is constituted by, for example, a shift register, and sequentially selects each of the column signal processing circuits 204 by sequentially outputting horizontal scanning pulses. A pixel signal is output from each of the column signal processing circuits 204 to the horizontal signal line 209. The timing control circuit 206 receives data for instructing an input clock signal and an operation mode from the control unit 107. The timing control circuit 206 generates a clock signal and a control signal that serve as a reference for operations of the vertical drive circuit 203, the column signal processing circuit 204, the horizontal drive circuit 205, and the like based on the vertical synchronization signal, the horizontal synchronization signal, and the master clock signal. .

  With reference to FIG. 3, the configuration and pixel arrangement of the pixel unit 201 of the image sensor 102 will be described. FIG. 3A is a schematic diagram illustrating a configuration example of photoelectric conversion units 201L and 201R that receive light that has passed through different pupil partial regions of the imaging optical system. The photoelectric conversion units 201L and 201R include photodiodes 211 and 212, respectively. L means that it is arranged on the left side when viewed from the front, and R means that it is arranged on the right side when viewed from the front. Hereinafter, the photodiode is abbreviated as “PD”.

  The photoelectric conversion unit 201L receives light that has passed through a part of the pupil region (first pupil partial region) of the imaging optical system. The photoelectric conversion unit 201R receives light that has passed through a part of the pupil region (second pupil partial region) different from the first pupil partial region. The photoelectric conversion unit 201L and the photoelectric conversion unit 201R are configured under one microlens 210, and each have one PD 211 and one PD 212. The transfer transistors 213 and 214 read the pixel signals of the PD 211 and PD 212, respectively. The PDs 211 and 212 have floating diffusion (FD) units 215 and 216 that temporarily store pixel signals. The configurations of the two photoelectric conversion units are the same except that the PD 211 and PD 212 read out signals that are photoelectrically converted from light that has passed through different pupil partial regions of the imaging optical system.

  Pixel signals acquired from the photoelectric conversion unit 201L and the photoelectric conversion unit 201R pass through the vertical signal line 207 and the column signal processing circuit 204, respectively (FIG. 2), and are read out to the horizontal signal line 209 as needed by the horizontal drive circuit 205 in units of rows. It is. Each pixel unit includes a plurality of constituent elements which will be described later in addition to the constituent elements shown in the figure, but these are not important in the description of the present invention and will be omitted.

  FIG. 3B is a plan view schematically showing a pixel array in the image sensor 102. In order to provide a two-dimensional image, a plurality of pixel portions 201 having the configuration shown in FIG. 3A are arranged in a two-dimensional array along a predetermined direction. The predetermined directions are horizontal and vertical directions. The PDs 301 </ b> L and 301 </ b> R in the pixel unit 301 constitute a photoelectric conversion unit that is divided into two in the pupil division direction (the horizontal direction in this example). A row 305 indicates a row including the pixels 301, 302, 303, and 304. Taking the row 305 as an example, 301L, 302L, 303L, and 304L correspond to the PD 211 in FIG. 3A, and 301R, 302R, 303R, and 304R correspond to the PD 212 in FIG.

  FIG. 4 is a conceptual diagram showing a state in which a light beam (subject light) emitted from the exit pupil of the photographing lens enters the image sensor 102. A cross-sectional portion 401 of the pixel portion shows a microlens 210, a color filter 403, and PD211 and PD212. The exit pupil 406 shows partial areas 407 and 408 of the exit pupil of the photographic lens, and these areas are pupil partial areas when viewed from each PD that receives light. For the pixel having the central microlens 210, an optical axis 409 that is the center of the light beam emitted from the exit pupil 406 is indicated by a one-dot chain line. Rays 410 and 411 are outermost rays of light passing through the pupil partial region 407, and rays 412 and 413 are outermost rays of light passing through the pupil partial region 408.

  The light emitted from the exit pupil 406 enters the image sensor 102 with the optical axis 409 as the center. As can be seen from FIG. 4A, among the light beams emitted from the exit pupil 406, the upper light beam with the optical axis 409 as a boundary passes through the microlens 210 and the color filter 403 and enters the PD 212. Further, the lower light beam with the optical axis 409 as a boundary passes through the microlens 210 and the color filter 403 and enters the PD 211. That is, the PD 211 and the PD 212 respectively receive light that has passed through different pupil partial regions 408 and 407 of the exit pupil 406 of the photographing lens.

  Referring to FIG. 3, for example, in the case of the pixel unit 301 included in the row 305, the PD 301 </ b> L corresponds to the PD 211 that receives a light beam emitted from one exit pupil (partial region) across the optical axis 409. An image obtained from the PD 301L is referred to as an A image, and the image signal is referred to as an A image signal. The PD 301R corresponds to the PD 212 that receives a light beam emitted from the other exit pupil (partial region) with the optical axis 409 interposed therebetween. An image obtained from the PD 301R is referred to as a B image, and the image signal is referred to as a B image signal.

  As described above, the PD 211 and the PD 212 receive light by dividing the exit pupil 406 equally into the pupil partial regions 407 and 408 around the optical axis 409, thereby causing a phase difference in the output signal between the PD 211 and the PD 212. Occurs. Due to this phase difference, the A image signal and the B image signal corresponding to the pupil partial regions 407 and 408 change in the address of the pixel portion 201 where the pixel signal derived from the same subject image appears, and is detected as an address interval. . By detecting this address interval (phase difference detection), the defocus amount is calculated. The PD 211 and the PD 212 are equally divided with respect to the optical axis 409. In other words, since the exit pupil 406 is not decentered, even if a part of the light beam is blocked by the components in the optical barrel 101, the signal loss (shading) of the A image or the B image is dealt with. There is an advantage that it is easy to do.

FIG. 5 is a diagram illustrating an example of the circuit configuration and basic operation of the image sensor.
In FIG. 5, a circuit configuration in the pixel 201L will be described as an example, but the same circuit configuration is used in the pixel 201R.

  As illustrated in FIG. 5, the pixel 201 </ b> L includes a PD 211, a transfer transistor 213, an amplification transistor 510, a selection transistor 511, and a reset transistor 512. In this example, each transistor is an n-channel MOSFET (MOS Field-Effect Transistor).

  A transfer signal line 513L, a row selection signal line 514L, and a reset control signal line 515L are connected to the gates of the transfer transistor 213, the selection transistor 511, and the reset transistor 512, respectively. Similarly, the transfer signal line 513R, the row selection signal line 514R, and the reset control signal line 515R are connected to the pixel 201R. In FIG. 2, these, the transfer signal line 513 </ b> L, the row selection signal line 514 </ b> L, and the reset control signal line 515 </ b> L are collectively illustrated by the pixel drive wiring 208. These signal lines extend in the horizontal direction and drive pixels included in the same row at the same time. As a result, a line-sequential operation type rolling shutter or an all-row simultaneous operation type global shutter can be used. It is possible to control the operation. Further, the transfer signal line 513 is configured separately for the pixels 201L and 201R, so that the exposure time can be set separately for the pixel 201L and the pixel 201R. Further, a vertical signal line 207L is connected to the source of the selection transistor 511, and one end of the vertical signal line 207L is grounded via a constant current source 516. In FIG. 2, the vertical signal lines 207L and 207R connected to the pixels 201L and 201R, respectively, are collectively shown as the vertical signal lines 207.

  The PD 211 accumulates charges generated by photoelectric conversion. In the PD 211, the P side is grounded and the N side is connected to the source of the transfer transistor 213. When the transfer transistor 213 is turned on, the charge of the PD 211 is transferred to the FD portion 215. Since the FD 215 has the parasitic capacitance C51, the charge is accumulated in this portion. The drain of the amplification transistor 510 is set to the power supply voltage Vdd, and the gate is connected to the FD 215. The amplification transistor 510 converts the voltage of the FD 215 into an electric signal. The selection transistor 511 selects a pixel from which a signal is read out in units of rows. The drain of the selection transistor 511 is connected to the source of the amplification transistor 510, and the source is connected to the vertical signal line 207. When the selection transistor 511 is turned on, the amplification transistor 510 and the constant current source 516 constitute a source follower, so that a voltage corresponding to the voltage of the FD 215 is output to the vertical signal line 207. The drain of the reset transistor 512 is the power supply voltage Vdd, and the source is connected to the FD 215. The reset transistor 512 resets the voltage of the FD 215 to the power supply voltage Vdd.

  6 and 7 are diagrams showing exposure condition patterns of the pixels 201L and 201R. In this embodiment, two frames are photographed, and the pattern of the exposure condition is different for every even / odd frame. FIG. 6 shows a pattern of exposure conditions in the first frame (first frame). FIG. 7 shows an exposure condition pattern in the second frame (second frame).

  Each pixel shown in FIGS. 6 and 7 is provided with red, blue, and green color filters. The pixels 201L and 201RR corresponding to the red color component color filter are R (hereinafter referred to as R pixel), and the pixels 201L and 201RB corresponding to the blue color component color filter are referred to as BL (hereinafter referred to as BL pixel). The pixels 201L and 201RG corresponding to the green color component color filter are defined as Gr or Gb (hereinafter referred to as Gr pixel or Gb pixel, respectively). The Gr pixel indicates a G pixel in a row with an R pixel, and the Gb pixel indicates a G pixel in a row with a BL pixel. In the nth row, the mth column pixel is repeatedly configured as an R pixel, the m + 1th column is a Gr pixel, and the m + 2 column is an R pixel. In the (n + 1) th row, the pixel in the m-th column is repeatedly configured as a Gb pixel, the m + 1-th column as a BL pixel, and the m + 2 column as a Gb pixel.

In this embodiment, a pixel photographed at an appropriate exposure time is a medium output pixel, a pixel photographed with a long exposure time for proper exposure is a high output pixel, and a pixel photographed with a short exposure is a low output pixel. To do. In FIG. 6 and FIG. 7, each part shown with three different hatching lines is as follows.
A photoelectric conversion unit indicated by a horizontal line: a pixel (medium output pixel) at a medium output level (appropriate output level).
A photoelectric conversion unit indicated by a rough oblique line: a low output level pixel (low output pixel).
A photoelectric conversion unit indicated by a vertical line: a high output level pixel (high output pixel).

  The low output pixel, the medium output pixel, and the high output pixel have the same configuration as the pixel 201L or the pixel 201R, respectively. Therefore, the R pixel, the Gb pixel, the Gr pixel, and the BL pixel described above are configured by a medium output pixel and a low output pixel, or a medium output pixel and a high output pixel, as shown in FIGS. In one pixel portion, the plurality of pixels are arranged in a second direction different from the arrangement direction (first direction) of the plurality of pixel portions.

  Further, in this embodiment, the exposure time of the low output pixel is set to half of the exposure time of the medium output pixel, and the exposure time of the high exposure pixel is set to double the appropriate exposure, so that the post-HDR processing time is set. The dynamic range can be extended by ± 1 step, but other settings may be used.

  The first frame shown in FIG. 6 includes low output pixels and medium output pixels. In the nth and n + 1th rows, the pixel corresponding to the pixel 201L is a medium output pixel, and the pixel corresponding to the pixel 201R is a low output pixel. In the n + 2 and n + 3 rows, the R, Gr, Gb, and B pixels are repeatedly configured in the same way as the n and n + 1 rows, but the pattern of the exposure conditions is different. In the n + 2 and n + 3 rows, the pixel corresponding to the pixel 201L is a low output pixel, and the pixel corresponding to the pixel 201R is a medium output pixel. The n + 4th and subsequent lines are repeatedly configured with the same exposure conditions and color filter patterns as the nth, n + 1th, n + 2th, and n + 3th lines.

  The second frame shown in FIG. 7 includes high output pixels and medium output pixels. In the nth and n + 1th rows, the pixel corresponding to the pixel 201L is a medium output pixel, and the pixel corresponding to the pixel 201R is a high output pixel. In the n + 2 and n + 3 rows, the R, Gr, Gb, and B pixels are repeatedly configured in the same manner as the n and n + 1 rows, but the pattern of exposure conditions is different. In the n + 2 and n + 3 rows, the pixel corresponding to the pixel 201L is a high output pixel, and the pixel corresponding to the pixel 201R is a medium output pixel. The n + 4th and subsequent lines are repeatedly configured in the same pattern as the nth, n + 1th, n + 2 and n + 3th lines. The first frame may be composed of high output pixels and medium output pixels, and the second frame may be composed of low output pixels and medium output pixels.

  In the present invention, in order to perform HDR signal processing, photographing is performed so that an exposure difference between the A image signal and the B image signal is generated. In the case where an exposure difference is generated in the A image signal and the B image signal, for example, the exposure time may be set separately for the A image and the B image, or a column for each output of the A image and the B image. In the signal processing circuit 204, the amplification process may be set to a different multiplication factor to generate an exposure difference.

  In FIG. 6, the pixels used for phase difference detection are, for example, the A image pixels in the nth row, the n + 4th row, the n + 8th row, and the B image pixels in the n + 2, the n + 6th row, and the n + 10th row. All pixels are used for HDR signal processing. An imaging signal read out with an exposure difference between the A image signal and the B image signal is output from the imaging element 102 and then passed to the video signal processing unit 104 and the phase difference signal processing unit 106 through the signal selection unit 103. The HDR signal processing and the phase difference detection processing are executed. The video signal processing unit 104 converts, for example, a medium output pixel signal, a low output pixel signal for the first frame shown in FIG. 6, and a high output pixel signal for the second frame shown in FIG. Based on this, HDR signal processing is executed. Note that the control unit 107 (FIG. 1) determines whether to perform HDR signal processing, and if it is determined to perform HDR signal processing, the video signal processing unit 104 may execute HDR signal processing. Good.

  From the AE calculation result of the previous frame by the video signal processing unit 104, the control unit 107 determines an appropriate exposure condition when detecting the phase difference. Based on the conditions determined by the control unit 107, for the first and second frames, imaging is set so that the pixels used for phase difference detection have the same output condition (exposure), and a plurality of signals are read out. Specifically, the phase difference between the A image signal of the R pixel in the nth, n + 4, and n + 8 rows and the B image signal of the R pixel in the n + 2, n + 6, and n + 10 rows. The signal processing unit 106 generates a phase difference detection signal by executing a known phase difference detection process. That is, the phase difference signal processing unit 106 performs correlation calculation while relatively shifting the two images using the A image signal and the B image signal, and performs known phase difference detection for detecting the image shift amount from the correlation calculation result. . In this embodiment, as can be seen with reference to FIGS. 6 and 7, the arrangement relationship of a plurality of pixels (for example, medium output pixels) used for the phase difference detection processing is the same in the first and second imaging. .

  The signal selection unit 103 selects the A image signal and the B image signal of the R pixel from the video signals and sends them to the phase difference signal processing unit 106. In the present embodiment, the phase difference detection process is performed on the R pixel signal, but may be performed on the RGB pixel signals and the signals obtained by calculating the respective color pixel signals. Data for one frame is output from the image sensor 102 by sequentially reading out the signal of the pixel 201 as described above from the top to the bottom row. Then, the signal selection unit 103 distributes the HDR processing video signal and the phase difference detection signal, and outputs them simultaneously to the video signal processing unit 104 and the phase difference signal processing unit 106, respectively. In this way, in the pixels used for phase difference detection, by determining appropriate exposure conditions for phase difference detection using information of the video signal processing unit 104 included in the video signal processing unit 104, signal whiteout and SN It is possible to easily avoid exposure conditions that are inferior.

Next, the dynamic range expansion process will be described with reference to FIG. FIG. 8 is a diagram illustrating the relationship between the incident light amount (horizontal axis X) and the pixel signal amount (vertical axis Y) in the dynamic range expansion process. The horizontal axis X represents X ^* 1 to X3 and X1 to X3, and the vertical axis Y represents Y1 to Y3. Y1 represents a noise level, and Y2 represents a saturation signal amount. X ^* 1 to 3 represent incident light amounts when the pixel signal levels of the high output pixel, the medium output pixel, and the low output pixel reach the noise level Y1, respectively. X1 to X3 represent incident light amounts when the pixel signal levels of the high output pixel, the medium output pixel, and the low output pixel reach the saturation signal amount Y2, respectively.

In the high output pixel, the saturation signal amount Y2 is reached when the amount of incident light reaches X1. In the medium output pixel, the saturation signal amount Y2 is reached when the incident light quantity becomes X2. In the low output pixel, the saturation signal amount Y2 is reached when the incident light quantity becomes X3. On the other hand, when the amount of pixel signal obtained by light reception is Y1 or less, it corresponds to the noise level, so that the pixel signal cannot be used. Therefore, the dynamic range of the high output pixel is a range where the incident light amount is from X ^* 1 to X1. The dynamic range of the medium output pixel is a range where the incident light amount is from X ^* 2 to X2. The dynamic range of the low output pixel is a range where the incident light amount is from X ^* 3 to X3.

As an example, a case will be described in which the ratio of the output levels of low output pixels, medium output pixels, and high output pixels is 1: 2: 4. The video signal processing unit 104 obtains the pixel signal HDR after the HDR processing for the pixel unit 201 according to the following formulas (1) to (5) according to the amount of incident light.
When X ^* 1 <(incident light amount) ≦ X ^* 2, pixel signal HDR = (high output pixel signal) (1)
When X ^* 2 <(incident light amount) ≦ X ^* 3, pixel signal HDR = (high output pixel signal) × (1−α) + (medium output pixel signal) × α × 2
(2)
When X ^* 3 <(incident light amount) ≦ X1, pixel signal HDR = (high output pixel signal) × β + (medium output pixel signal) × γ × 2 + (low output pixel signal) × (1−β−γ) × 4 (3)
When X1 <(incident light quantity) ≦ X2, pixel signal HDR = (medium output pixel signal) × (1-δ) × 2 + (low output pixel signal) × δ × 4
(4)
When X2 <(incident light quantity) ≦ X3, pixel signal HDR = (low output pixel signal) × 4 (5)

Α, β, γ, δ, β + γ in the above equation are coefficients for synthesis, and all of these values are positive real numbers of 1 or less. A signal after HDR processing is generated from the low output pixel signal, the medium output pixel signal, and the high output pixel signal according to the amount of incident light. The video signal processing unit 104 calculates the pixel signal after the dynamic range expansion processing using Equations (1) to (5) according to the signal amount (incident light amount) of each pixel unit 201 of the pixel array unit 202. To do. As a result, by acquiring and synthesizing signals of three different output levels taken, the signal amount is expanded from Y1 to Y3, and an image with a wide dynamic range that can handle X ^* 1 to X3 as the incident light amount is generated. it can. In this embodiment, in two frames taken with the exposure condition patterns of FIGS. 6 and 7, a low output pixel signal, a medium output pixel signal, and a high output pixel signal from each of the R, Gr, Gb, and BL pixels. The above-described HDR signal processing is performed from these signals.

FIG. 9 is a flowchart for explaining operation processing in the present embodiment.
When shooting is started, the control unit 107 sets shooting conditions in step S801. If this shooting condition is initial, the initial condition is applied, but the shooting condition is updated by feedback from step S807 described later.

  In step S <b> 802, the control unit 107 drives the image sensor 102 with the shooting conditions set in step S <b> 801, and performs shooting of the first frame with the pattern of exposure conditions for each pixel in the sensor array illustrated in FIG. 6. In step S <b> 803, the control unit 107 captures a second frame with a pattern of exposure conditions for each pixel in the sensor array illustrated in FIG. 7.

  In step S804, the video signal processing unit 104 calculates an AE value from the captured first frame and second frame signals. Subsequently, in S805, the video signal processing unit 104 executes HDR signal processing. The HDR signal processing is executed based on signals of three types of pixels (medium output pixel, low output pixel, and high output pixel) in the first frame and the second frame shown in FIGS. In this example, the medium output pixel indicated by a horizontal hatching in FIG. 6 or 7 is a pixel (phase difference pixel) used for phase difference detection. The phase difference pixel signal (phase difference pixel signal) is subjected to HDR signal processing by reducing the random noise by averaging the signal values in the first and second frames.

  In step S806, the phase difference signal processing unit 106 performs phase difference detection from the phase difference pixel signal in the second frame imaged in step S803. The phase difference detection processing unit 106 may execute phase difference detection from the phase difference pixel signal in the first frame. Further, the phase difference detection processing unit 106 may perform phase difference detection from the phase difference pixel signal in which random noise is reduced by averaging the signal values of the first and second frames. Note that the processes of S804, S805, and S806 may be executed in parallel or in any order.

  Next, in step S807, the control unit 107 determines whether or not to end shooting. When shooting is ended, the state transits to a standby state. In the case of continuing shooting, the process returns to S801, and the control unit 107 updates the exposure condition from the AE value obtained in S804. Further, the focus mechanism unit 1011 executes autofocus control from the phase difference detection result obtained in S806. As described above, the shooting conditions are reset and the shooting is repeated.

  By performing the operation of the present embodiment as described above, HDR imaging with simultaneous reduction of HDR-processed video signal data and phase difference detection operation and reduction of random noise under exposure conditions larger than the number of shots is performed. Can do. In addition, in the phase difference detection pixel, by determining appropriate exposure conditions for phase difference detection using the AE information of the video signal processing unit 104, it is possible to easily avoid exposure conditions in which signals are overexposed or SN is inferior. it can.

(Example 2)
Next, Example 2 will be described. The imaging apparatus according to the second embodiment has the same configuration as the imaging apparatus according to the first embodiment. Therefore, the same parts as those in the first embodiment are denoted by the same reference numerals and the description thereof is omitted.

  FIG. 10 is a diagram showing a pattern of pixel exposure conditions in the second embodiment. FIG. 10 shows a sensor array according to the first frame. The low output pixels are the same as the sensor array shown in FIG. 6, but in the second embodiment, the pixels 201 </ b> R and 201 </ b> L that do not read out pixel signals are provided. In the example shown in FIG. 10, the pixel 201R and the pixel 201L from which the pixel signal is not read out are white (not hatched), and for the characters of the R pixel, the Gr pixel, the Gb pixel, and the BL pixel, Strike through.

  In the second embodiment, as in the pattern shown in FIG. 10, only the pixels used for HDR imaging are read without reading the pixel signals of the pixels used for phase difference detection of the first frame. This eliminates the need to put the column signal processing circuit 204 for pixels that are not read out to sleep or operate the vertical driving circuit 203 or the horizontal driving circuit 205 in an extra manner, so that a power saving effect can be obtained. That is, for the second imaging, the imaging element reads signals from all the pixels included in the plurality of pixel units, and for the first imaging, reads signals only from the pixels used for the phase difference detection process. From the above, according to the second embodiment, HDR imaging and phase difference detection can be made compatible, HDR imaging can be performed under an exposure condition larger than the number of images to be captured, and a power saving effect can be aimed at. Of course, the image sensor reads out signals from all the pixels of the plurality of pixel units for the first imaging, and reads out signals only from the pixels used for the phase difference detection processing for the second imaging. Also good.

  In the first and second embodiments, the configuration example in which the video signal processing unit 104 and the phase difference signal processing unit 106 are provided in the imaging device with respect to the imaging element 102 has been described. Functions of these signal processing units A configuration in which at least a part of the image sensor is provided in the image sensor may be used. In this case, for example, a pixel array unit (imaging unit) in which a large number of pixel units are arranged in a matrix and a signal processing unit that processes a signal of each pixel unit are mounted on the integrated circuit chip. For example, in the case of a stacked imaging device, the imaging device has a configuration in which a second integrated circuit chip that constitutes an imaging unit is stacked on a first integrated circuit chip that constitutes a signal processing unit. At this time, the signal processing unit configured on the first integrated circuit chip includes a correlation calculation unit that performs correlation calculation of two images for focus detection and a calculation unit that calculates an image shift amount from the correlation calculation result. Good. As a result, since the image shift amount (or defocus amount) and distribution thereof may be output as the output from the image sensor 102, it is possible to reduce the manufacturing cost of the sensor or to secure the bandwidth of the image processing unit at the subsequent stage. it can. At this time, for HDR processing, a correction processing unit that corrects pixel defects or signal variations caused by the image sensor 102 and a synthesis processing unit that performs HDR synthesis may be provided. As a result, an image signal for one frame after synthesis is output as an output from the image sensor 102. In addition to the same effects as described above, an important process for determining the image quality can be performed with higher accuracy analysis and It can be left to an image processing unit at a later stage that can be processed. Of course, the present invention is not limited to the above, and a part or all of other processing may be provided in the signal processing unit in the image sensor 102 for focus detection and HDR processing. In addition, as a specific configuration example of the signal processing unit, one signal processing that executes in parallel the first signal processing that performs bit range expansion processing in HDR processing and the second signal processing that performs phase difference detection A part may be provided.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

102 Image sensor 104 Video signal processing unit 106 Phase difference signal processing unit 107 Control unit

Claims (12)

An image sensor capable of acquiring signals from a plurality of pixel units and performing signal processing for focus detection and image generation,
Setting means for setting signal output conditions for the first and second photoelectric conversion units respectively included in the first and second pixel units;
Signal processing means for performing signal processing of the image generation based on pixel signals of the first to third output conditions obtained from the first and second pixel units for the first and second imaging; An image pickup device comprising:   The signal processing means further outputs a plurality of signals obtained from photoelectric conversion units having the same output condition setting among the first and second photoelectric conversion units included in the first and second pixel units, respectively. The imaging device according to claim 1, wherein signal processing of the focus detection is performed.   The positional relationship between the photoelectric conversion unit included in the first pixel unit and the photoelectric conversion unit included in the second pixel unit, which is used for the focus detection signal processing, is the same in the first and second imaging. The image pickup device according to claim 2, wherein the image pickup device is provided. Readout means for reading out pixel signals from the plurality of pixel portions,
The reading means reads signals from all the photoelectric conversion units included in the plurality of pixel units for the second imaging, and the photoelectric conversion unit used for the focus detection signal processing for the first imaging. The image sensor according to any one of claims 1 to 3, wherein a signal is read out only from the image sensor. The setting means includes
For the first imaging, the first photoelectric conversion unit included in the first pixel unit is set to the second output condition, the second photoelectric conversion unit is set to the first output condition,
For the second imaging, the first photoelectric conversion unit included in the first pixel unit is set to the second output condition, and the second photoelectric conversion unit is set to the third output condition. The imaging device according to any one of claims 1 to 4, wherein The first and second pixel units are arranged in a first direction, and the first and second photoelectric conversion units are arranged in a second direction different from the first direction in the pixel unit. ,
The signal processing means includes a signal of a first photoelectric conversion unit, a signal of the second photoelectric conversion unit, and a signal of the second shooting frame included in the first pixel unit for the first shooting. The image sensor according to claim 5, wherein signal processing for expanding a dynamic range of an image signal is performed based on a signal from the second photoelectric conversion unit. Determination means for determining whether to perform signal processing for dynamic range expansion of the image signal;
The image sensor according to claim 6, wherein the signal processing unit performs signal processing for expanding the dynamic range of the image signal when it is determined to perform signal processing for expanding the dynamic range of the image signal. . The first and second pixel units each include a microlens and the first and second photoelectric conversion units corresponding to the microlens,
The image sensor according to any one of claims 1 to 7, wherein the first and second photoelectric conversion units photoelectrically convert subject light that has passed through different exit pupils of an image pickup optical system.   9. The output condition of the pixel signal is one or more of a sensitivity, an exposure time, a signal amplification degree, and an aperture value of the photoelectric conversion unit. The imaging device described in 1. An imaging device having a plurality of pixel units, and an imaging device that acquires signals from the pixel units and performs signal processing for focus detection and image generation,
Setting means for setting signal output conditions for the first and second photoelectric conversion units respectively included in the first and second pixel units;
Signal processing means for performing signal processing of the image generation based on pixel signals of the first to third output conditions obtained from the first and second pixel units for the first and second imaging. An imaging apparatus characterized by that.   The signal processing means further includes a plurality of signal processing units obtained from photoelectric conversion units having the same pixel signal output condition setting among the first and second photoelectric conversion units included in the first and second pixel units, respectively. The image pickup apparatus according to claim 10, further comprising: a signal processing unit that performs signal processing for the focus detection using the pixel signal. An imaging signal processing method for acquiring signals from a plurality of pixel units constituting an imaging device and performing signal processing for focus detection and image generation,
A step of setting signal output conditions for the first and second photoelectric conversion units respectively included in the first and second pixel units;
A step of performing signal processing of the image generation based on pixel signals of the first to third output conditions obtained from the first and second pixel units for the first and second imaging. An imaging signal processing method characterized by the above.

Images