JP2017147721A - Imaging device and driving method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device capable of facilitating the reading of multiple videos based on the signal of different storage period without sacrificing the frame rate, and to provide a driving method therefor.SOLUTION: An imaging device has a reading circuit for reading a first video signal based on multiple pixels each including a first pixel element including a first photoelectric conversion part, and a second pixel element including a second photoelectric conversion part of lower sensitivity than the first photoelectric conversion part, respectively, and signal charges generated during a first storage period in the first pixel element, and a second video signal based on the signal charges generated during a second storage period, longer than the first storage period, in the second pixel element, and synchronized with the first video signal, and a control section for controlling the reading circuit so that the first reading period for reading the first video signal will be during the second storage period.SELECTED DRAWING: Figure 18

Description

  The present invention relates to an imaging apparatus and a driving method thereof.

  If you can shoot movies and still images simultaneously with a single camera, you can watch the shooting scene as a movie and enjoy the decisive scene in the movie as a still image, greatly increasing the value of the shot image. Can be increased. Also, if you can shoot a normal frame rate video and a high frame rate video at the same time with a single camera, you can enjoy a high-quality work while switching a specific scene to a slow motion video. It can convey a dynamic feeling.

  In addition, as a technique for making movies and home television images more realistic, there is an HDR (high dynamic range) technique for moving images. This combines the output from two pixel groups with different sensitivities to expand the brightness reproduction range of the display screen, and provides a more realistic sensation than before, mainly by momentary or partial brightness increase. To do.

  As an imaging apparatus for capturing such an image, an imaging apparatus using an imaging element having pixels including a plurality of photoelectric conversion units having different light receiving efficiency has been proposed. Patent Document 1 discloses an imaging device including a pair of photodiodes having an asymmetric pupil shape in each pixel. In the imaging device described in Patent Document 1, one of the pair of photodiodes has high light reception efficiency, and the other photodiode has low light reception efficiency. By using the two signals from the pair of photodiodes as separate image data, two images can be simultaneously captured.

JP 2014-048459 A JP 2013-172210 A

  However, in an imaging device that simultaneously captures two images, it is necessary to read a plurality of images based on signals having different accumulation times within one frame period. If it takes time to read out these signals, the time required for one frame period becomes longer, and as a result, the frame rate decreases.

  An object of the present invention is to provide an imaging apparatus that can easily read out a plurality of videos based on signals having different accumulation periods without sacrificing the frame rate, and a driving method thereof.

  According to one aspect of the present invention, a first pixel element including a first photoelectric conversion unit, and a second pixel element including a second photoelectric conversion unit having a lower sensitivity than the first photoelectric conversion unit, A first video signal based on a signal charge generated during a first accumulation period in the first pixel element of the plurality of pixels from the plurality of pixels, A video signal based on a signal charge generated during a second accumulation period longer than the first accumulation period in the second pixel element of a plurality of pixels, and synchronized with the first video signal Imaging having a readout circuit for reading out the second video signal and a control unit for controlling the readout circuit so that a first readout period for reading out the first video signal is performed during the second accumulation period. An apparatus is provided.

  According to another aspect of the present invention, a plurality of pixels each including a photoelectric conversion unit and a first signal holding unit and a second signal holding unit, and the photoelectric conversion unit from the plurality of pixels in one imaging cycle. The first video signal generated by transferring the signal charge generated in the conversion unit to the first signal holding unit at least once, and the signal charge generated in the photoelectric conversion unit during the one photographing period A readout circuit that reads out a second video signal that is generated by transferring and adding to the second signal holding unit at least twice or more and that is synchronized with the first video signal; There is provided an imaging apparatus including a control unit that controls the readout circuit such that a readout period for reading out the first video signal is performed during an accumulation period of the second video signal.

  According to still another aspect of the present invention, a first pixel element including a first photoelectric conversion unit, and a second photoelectric conversion unit including a second photoelectric conversion unit having a lower sensitivity than the first photoelectric conversion unit. An image pickup apparatus driving method including an image pickup device having a plurality of pixels each including two pixel elements, wherein the first pixel elements of the plurality of pixels are generated during a first accumulation period. Reading the first video signal based on the signal charge, and based on the signal charge generated in the second pixel element of the plurality of pixels during a second accumulation period longer than the first accumulation period And reading a second video signal that is synchronized with the first video signal, and the step of reading the first video signal is performed during the second accumulation period. A method of driving the apparatus is provided.

  According to still another aspect of the present invention, there is provided a driving method of an imaging apparatus including an imaging device having a plurality of pixels each including a photoelectric conversion unit and a first and a second signal holding unit, Reading out the first video signal generated by transferring the signal charges generated in the photoelectric conversion unit during one imaging period to the first signal holding unit from the plurality of pixels at least once; A video signal generated by transferring and adding signal charges generated in the photoelectric conversion unit from the plurality of pixels to the second signal holding unit at least twice or more during the one imaging cycle. And a step of reading out a second video signal synchronized with the first video signal, and the step of reading out the first video signal comprises driving the image pickup apparatus during the accumulation period of the second video signal. A method is provided.

  According to the present invention, when a plurality of videos based on signals having different accumulation periods are simultaneously photographed using one image sensor, the plurality of videos can be read without sacrificing the frame rate.

1 is an external view showing an imaging apparatus according to a first embodiment of the present invention. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. It is a block diagram which shows the structural example of the image pick-up element of the imaging device by 1st Embodiment of this invention. It is sectional drawing which shows the internal structure of the image pick-up element of the imaging device by 1st Embodiment of this invention. It is a graph which shows the relationship between the angle of the light ray which injects into a pixel, and the output from a photodiode. It is a figure which shows the relationship between the imaging optical system and imaging device in the imaging device by 1st Embodiment of this invention. It is the schematic explaining the video signal output from an image sensor. It is a circuit diagram showing an example of composition of a pixel of an image sensor of an imaging device by a 1st embodiment of the present invention. It is a plane layout figure (the 1) which shows the principal part of the pixel of the image pick-up element of the imaging device by 1st Embodiment of this invention. FIG. 6 is a plan layout diagram (No. 2) illustrating a main part of a pixel of the imaging device of the imaging apparatus according to the first embodiment of the present invention. It is a circuit diagram showing an example of composition of a readout circuit of an image sensor of an imaging device by a 1st embodiment of the present invention. 3 is a timing chart showing a driving sequence of the image sensor in the imaging apparatus according to the first embodiment of the present invention. FIG. 10 is a potential diagram of a pixel along the line AB in FIG. 9. It is sectional drawing which shows the behavior of the signal charge which generate | occur | produced by propagation of the light in an inside of an image pick-up element, and photoelectric conversion. It is a figure explaining the setting screen of the imaging conditions of "picture A" and "picture B". It is a figure which shows the relationship of the ISO sensitivity range of "picture A" and "picture B". It is a program AE diagram in the dual video mode of the imaging device according to the first embodiment of the present invention. It is a flowchart figure which shows the accumulation | storage control method in the imaging device by 1st Embodiment of this invention. It is a figure explaining the difference of the shutter speed of "picture A" and "picture B" on an imaging sequence. It is a figure showing the mode of the display part in the live view display after powering on an imaging device. It is a figure which shows 1 frame of the images | videos acquired by operating switch ST and switch MV. It is a flowchart which shows a series of processing procedures including crosstalk correction. It is a figure explaining the crosstalk correction process performed in a digital signal processing part. It is a figure which shows the specific example of a crosstalk correction function. It is a figure which shows an example of the image | video after performing crosstalk correction | amendment. It is a figure which shows a mode that "picture A" and "picture B" were displayed side by side on the display part. It is a figure explaining the utilization example of "picture A" and "picture B" stored in the storage. It is a circuit diagram which shows the structural example of the pixel of the image pick-up element of the imaging device by 2nd Embodiment of this invention. It is a figure for demonstrating the drive method of the image pick-up element in the imaging device by 2nd Embodiment. It is a timing chart which shows the drive sequence of the image sensor in the imaging device by a 2nd embodiment.

[First Embodiment]
An imaging apparatus according to a first embodiment of the present invention will be described with reference to FIGS. In the present embodiment, an imaging apparatus including an imaging optical system for imaging, an imaging element, and a video processing apparatus for processing a video signal output from the imaging element is a preferred embodiment of the present invention. This will be described as an example. However, the video processing apparatus is not necessarily configured as a part of the imaging apparatus, and may be configured by hardware different from the imaging element and the imaging optical system. Moreover, you may make it mount all or one part of the function of a video processing apparatus in an image pick-up element.

FIG. 1 is an external view of a digital still motion camera as an example of the imaging apparatus according to the present embodiment. FIG. 1A shows a front view thereof, and FIG. 1B shows a rear view thereof.
The imaging apparatus 100 according to the present embodiment includes a housing 151, a photographing optical system 152 provided on the front portion of the housing 151, and a switch ST <b> 154 and a propeller 162 provided on the upper surface of the housing 151. Yes. In addition, the imaging apparatus 100 includes a display unit 153, a switch MV155, a shooting mode selection lever 156, a menu button 157, up / down switches 158 and 159, a dial 160, and a playback button on the rear surface of the casing 151. 161.

  The casing 151 is a container that houses various functional components that constitute the imaging device 100 such as an imaging element and a shutter device. The photographing optical system 152 is an optical system for forming an optical image of a subject. The display unit 153 includes a display device for displaying shooting information and video. The display unit 153 may be provided with a movable mechanism for changing the orientation of the screen as necessary. The display unit 153 has a display luminance range that can display an image with a wide dynamic range without suppressing the luminance range. The switch ST154 is a shutter button used mainly for taking a still image. The switch MV155 is a button for starting and stopping moving image shooting. The shooting mode selection lever 156 is a changeover switch for selecting a shooting mode. The menu button 157 is a button for shifting to a function setting mode for setting a function of the imaging apparatus 100. Up / down switches 158 and 159 are buttons used when changing various setting values. The dial 160 is a dial for changing various setting values. The playback button 161 is a button for shifting to a playback mode in which video recorded on a recording medium stored in the imaging apparatus 100 is played on the display unit 153. The propeller 162 is for floating the imaging device 100 in the air in order to perform shooting from the air.

  FIG. 2 is a block diagram illustrating a schematic configuration of the imaging apparatus according to the present embodiment. As illustrated in FIG. 2, the imaging apparatus 100 includes a diaphragm 181, a diaphragm controller 182, an optical filter 183, an image sensor 184, a digital signal processor 187, and a timing generator 189. Further, the imaging apparatus 100 includes a system control CPU 178, a switch input unit 179, a video memory 190, and a flight control apparatus 200. Further, the imaging apparatus 100 includes a display interface unit 191, a recording interface unit 192, a recording medium 193, a print interface unit 194, an external interface unit 196, and a wireless interface unit 198.

  The image sensor 184 is for converting an optical image of a subject formed through the photographing optical system 152 into an electrical video signal. The image sensor 184 is not particularly limited, but has, for example, a sufficient number of pixels, a signal readout speed, a color gamut, and a dynamic range to satisfy the UHDTV (Ultra High Definition Television) standard. The diaphragm 181 is for adjusting the amount of light passing through the photographing optical system 152. The aperture control unit 182 is for controlling the aperture 181. The optical filter 183 is for limiting the wavelength of light incident on the image sensor 184 and the spatial frequency transmitted to the image sensor 184.

  The digital signal processing unit 187 is for compressing the video data after performing various corrections on the digital video data output from the image sensor 184. The correction performed by the digital signal processing unit 187 includes crosstalk correction described later. The timing generator 189 is for outputting various timing signals to the image sensor 184 and the digital signal processor 187. The system control CPU 178 is a control unit that performs various calculations and controls the entire imaging apparatus 100 together with the timing generation unit 189.

  The video memory 190 is for temporarily storing video data. A display interface unit (display I / F) 191 is an interface between the system control CPU 178 and the display unit 153 for displaying a captured image on the display unit 153 such as a liquid crystal display. The recording medium 193 is a recording medium such as a semiconductor memory for recording video data, additional data, and the like, and may be provided in the imaging apparatus 100 or detachable. The recording interface unit (recording I / F) 192 is an interface between the system control CPU 178 and the recording medium 193 for recording on the recording medium 193 or reading from the recording medium. An external interface unit (external I / F) 196 is an interface between the system control CPU 178 and an external device for communicating with an external device such as the external computer 197. A print interface unit (print I / F) 194 is an interface between the system control CPU 178 and the printer 195 for outputting a photographed video to a printer 195 such as a small inkjet printer and printing it. A wireless interface unit (wireless I / F) 198 is an interface between the system control CPU 178 and the network 199 for communicating with a network 199 such as the Internet. The switch input unit 179 includes a switch ST154, a switch MV155, and a plurality of switches for switching various modes. The flight control device 200 is a control device for controlling the propeller 162 to fly the imaging device 100 in order to perform shooting from the air.

  FIG. 3 is a block diagram illustrating a configuration example of the image sensor 184. As shown in FIG. 3, the image sensor 184 includes a pixel array 302, a vertical scanning circuit 307, a reading circuit 308, and a timing control circuit 309.

  In the pixel array 302, a plurality of pixels 303 are arranged in a matrix. Note that the actual number of pixels 303 belonging to the pixel array 302 is generally large, but here, in order to simplify the drawing, 16 pixels 303 arranged in a matrix of 4 rows × 4 columns. Only shows. Each of the plurality of pixels 303 includes a set of a pixel element 303A and a pixel element 303B. In FIG. 3, the upper half area of the pixel 303 is a pixel element 303A, and the lower half area of the pixel 303 is a pixel element 303B. The pixel element 303A and the pixel element 303B each generate a signal by photoelectric conversion.

  Each column of the pixel array 302 is provided with a signal output line 304 extending in the column direction. The signal output line 304 of each column is connected to the pixel element 303A and the pixel element 303B belonging to the column. The signal output line 304 outputs a signal from the pixel element 303A and a signal from the pixel element 303B. Each column of the pixel array 302 is also provided with a power supply line 305 and a ground line 306 extending in the column direction. The power supply line 305 and the ground line 306 in each column are connected to the pixels 303 belonging to the column. The power supply line 305 and the ground line 306 may be signal lines extending in the row direction.

  The vertical scanning circuit 307 is disposed adjacent to the pixel array 302 in the row direction. The vertical scanning circuit 307 controls a readout circuit in the pixel 303 via a control line (not shown) extending in the row direction with respect to the plurality of pixels 303 of the pixel array 302 in the row direction. A predetermined control signal is output. In the figure, reset pulses φRESn and transfer pulses φTXnA, φTXnB are shown as control signals (n is an integer corresponding to a row number).

  The readout circuit 308 is disposed adjacent to the pixel array 302 in the column direction. The read circuit 308 is connected to the signal output line 304 of each column. The read circuit 308 sequentially activates the signal output lines 304 of each column sequentially, thereby sequentially reading the signals of the signal output lines 304 of each column and performing predetermined signal processing. The readout circuit 308 can include a noise removal circuit, an amplification circuit, an analog-digital conversion circuit, a horizontal scanning circuit, and the like, and sequentially outputs signals that have undergone predetermined signal processing.

  The timing control circuit 309 is connected to the vertical scanning circuit 307 and the readout circuit 308. The timing control circuit 309 outputs a control signal for controlling the driving timing of the vertical scanning circuit 307 and the readout circuit 308.

  FIG. 4 is a cross-sectional view showing the internal structure of the pixel 303 of the image sensor 184. Each pixel 303 includes two photodiodes 310A and 310B, a light guide 255, and a color filter 256, as shown in FIG. The photodiode 310A constitutes a part of the pixel element 303A, and the photodiode 310B constitutes a part of the pixel element 303B. The photodiodes 310A and 310B are provided in the silicon substrate 251. The light guide 255 is provided in an insulating layer 254 provided on the silicon substrate 251. The insulating layer 254 is made of, for example, silicon oxide, and the light guide 255 is made of a material having a higher refractive index than the insulating layer 254, for example, silicon nitride. A wiring layer 252 is provided on the insulating layer 254 between the light guides 255. On the light guide 255, a color filter 256 having a predetermined spectral transmittance characteristic is provided. FIG. 4 shows an example in which the color filters of two adjacent pixels 303 are configured by color filters 256 and 257 having different spectral transmittance characteristics.

  The light guide 255 has a property of confining light inside due to a difference in refractive index with the insulating layer 254. Thereby, the light incident through the color filter 256 can be guided to the photodiodes 310 </ b> A and 310 </ b> B by the light guide 255. The photodiodes 310A and 310B are arranged asymmetrically with respect to the light guide 255, and the light beam propagated through the light guide 255 is incident on the photodiode 310A with high efficiency and is incident on the photodiode 310B with low efficiency. Furthermore, the light guide 255 is configured such that the incident angle characteristic is not biased with respect to the incident light beam that can be effectively photoelectrically converted by the photodiodes 310A and 310B by adjusting the depth and the inclination angle.

  FIG. 5 is a graph showing the relationship between the angle of light rays incident on the pixel and the output from the photodiode. In FIG. 5, the horizontal axis represents the angle of light rays incident on the pixel, and the vertical axis represents the output from the photodiode. FIG. 5 shows output characteristics 261 from the photodiode 310A and output characteristics from the photodiode 310B.

  As shown in FIG. 5, the characteristic 261 and the characteristic 262 are both symmetrically mountain-shaped with a peak when the incident angle of the light beam is zero. Further, the peak intensity PB of the characteristic 262 is about 1/8 of the peak intensity PA of the characteristic 261. This means that the incident angle dependence of the photodiodes 310A and 310B is small, and the light receiving efficiency of the photodiode 310B is 1/8 of that of the photodiode 310A. That is, if the photodiode 310B is replaced with the ISO sensitivity setting value, the sensitivity is lower by three stages than the photodiode 310A.

  Next, the relationship between the photographing optical system 152 and the image sensor 184 will be described in more detail with reference to FIG. FIG. 6 is a diagram for explaining the relationship between the photographing optical system 152 and the image sensor 184. FIG. 6A is a view of the photographing optical system 152 as seen from the direction of the optical axis 180. FIG. 6B is a diagram showing in more detail the part from the photographing optical system 152 to the image sensor 184 in FIG.

  As shown in FIG. 6B, the imaging element 184 includes a pixel 276 located at the center of the imaging area and a pixel 277 located near the outer edge of the imaging area. In this case, the pixel 276 can receive a light flux from a region surrounded by the light rays 272 and 273. Further, the pixel 277 can receive a light beam from a region surrounded by the light beam 274 and the light beam 275. At this time, since the field lens 270 is disposed between the optical filter 183 and the photographing optical system 152, the light flux received by the pixel 276 and the light flux received by the pixel 277 in the vicinity of the photographing optical system 152 are illustrated in FIG. 6 (a) is overlapped as indicated by a region 271. As a result, the light beam emitted from the photographing optical system 152 can be received with high efficiency in any pixel.

  FIG. 7 is a schematic diagram illustrating a video signal output from the image sensor. Here, it is assumed that a color filter having a predetermined light transmittance characteristic is arranged in the pixel array 302 in the color filter array 281 shown in FIG. FIG. 7A schematically shows a pixel array 302 in which pixels 303 are arranged in a 6-row × 8-column matrix, and the color of the color filter arranged in each pixel. In the figure, R represents a red color filter, G1 and G2 represent green color filters, and B represents a blue color filter. The color filter array 281 shown in the figure is a color filter array called a so-called Bayer array, and color filters for each color are arranged for each row by repeating G1BG1B... RG2RG2... G1BG1B.

  From the pixel array 302 having such a color filter array 281, output data 282 and 283 shown in FIGS. 7B and 7C are obtained. In FIG. 7B, g1A and g2A represent outputs from the pixel element 303A of the pixel 303 in which the green color filter is arranged. bA represents the output from the pixel element 303A of the pixel 303 in which the blue color filter is arranged. rA represents the output from the pixel element 303A of the pixel 303 in which the red color filter is arranged. In FIG. 7C, g1B and g2B represent outputs from the pixel element 303B of the pixel 303 in which the green color filter is arranged. bB represents the output from the pixel element 303B of the pixel 303 in which the blue color filter is arranged. rB represents the output from the pixel element 303B of the pixel 303 in which the red color filter is arranged.

  As described with reference to FIG. 3, the image sensor 184 obtains two systems of outputs, that is, an output from the pixel element 303A and an output from the pixel element 303B, one of which is shown in FIG. 7B. The output data 282 and the other is the output data 283 shown in FIG. The output data 282 becomes the first video signal “picture A” after predetermined signal processing. The output data 283 becomes a second video signal “picture B” after predetermined signal processing. In the following description, the video signal based on the output data 282 is referred to as “picture A”, and the video signal based on the output data 283 is referred to as “picture B”. Strictly speaking, “picture A” and “picture B” are video signals after processing such as predetermined correction, but for convenience of explanation, “picture A” is also used for video signals before or during correction. ”And“ picture B ”.

  FIG. 8 is a circuit diagram illustrating a configuration example of the pixel 303. As described above, the pixel 303 includes the pixel element 303A and the pixel element 303B. The pixel element 303A includes a photodiode 310A, a transfer transistor 311A, a floating diffusion region 313, a reset transistor 314, an amplification transistor 315, and a selection transistor 317. The pixel element 303B includes a photodiode 310B, a transfer transistor 311B, a floating diffusion region 313, a reset transistor 314, an amplification transistor 315, and a selection transistor 317. The floating diffusion region 313, the reset transistor 314, the amplification transistor 315, and the selection transistor 317 are shared by the pixel element 303A and the pixel element 303B. This makes it possible to increase the area of the photodiode region. The photodiode 310A corresponds to the photodiode 310A illustrated in FIG. 4, and the photodiode 310B corresponds to the photodiode 310B illustrated in FIG.

  The anode of the photodiode 310A is connected to the ground line 306, and the cathode of the photodiode 310A is connected to the source of the transfer transistor 311A. The anode of the photodiode 310B is connected to the ground line 306, and the cathode of the photodiode 310B is connected to the source of the transfer transistor 311B. The drains of the transfer transistors 311A and 311B are connected to the source of the reset transistor 314 and the gate of the amplification transistor 315. A connection node between the drains of the transfer transistors 311A and 311B, the source of the reset transistor 314, and the gate of the amplification transistor 315 forms a floating diffusion region 313. The drain of the reset transistor 314 and the drain of the amplification transistor 315 are connected to the power supply line 305 to which the voltage VRES is supplied. The source of the amplification transistor 315 is connected to the drain of the selection transistor 317. The source of the selection transistor 317 constituting the pixel signal output unit 316 is connected to the signal output line 304.

  The pixels 303 in each column are connected to the selection signal line 318, the reset control line 319, and the transfer control lines 320A and 320B arranged in the row direction from the vertical scanning circuit 307 for each row. The selection signal line 318 is connected to the gate of the selection transistor 317. The reset control line 319 is connected to the gate of the reset transistor 314. The transfer control line 320A is connected to the gate of the transfer transistor 311A via the contact portion 312A. The transfer control line 320B is connected to the gate of the transfer transistor 311B via the contact portion 312B. The reset control line 319 supplies a reset pulse φRESn output from the vertical scanning circuit 307 to the gate of the reset transistor 314. The transfer control line 320A supplies a transfer pulse φTXnA output from the vertical scanning circuit 307 to the gate of the transfer transistor 311A. The transfer control line 320B supplies the transfer pulse φTX1B output from the vertical scanning circuit 307 to the gate of the transfer transistor 311B. Note that n added to the signs of the reset pulse φRESn, the transfer pulse φTXnA, and the transfer pulse φTXnB is an integer corresponding to the row number. In the drawing, n is represented by a code in which n is replaced with an integer corresponding to the row number.

  The photodiode 310A is a first photoelectric conversion unit that generates charges by photoelectric conversion, and the photodiode 310B is a second photoelectric conversion unit that generates charges by photoelectric conversion. The floating diffusion region 313 is a region for accumulating charges. The transfer transistor 311A is for transferring the charge generated by the photodiode 310A to the floating diffusion region 313. The transfer transistor 311B is for transferring the charge generated by the photodiode 310B to the floating diffusion region 313.

  When the high-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned on, and the photodiode 310A and the floating diffusion region 313 are connected. Similarly, when the high-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned on, and the photodiode 310B and the floating diffusion region 313 are connected. When the high level reset pulse φRESn is output from the vertical scanning circuit 307, the reset transistor 314 is turned on, and the photodiodes 310A and 310B and the floating diffusion region 313 are reset.

  When the low-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned off, and the photodiode 310A starts accumulating signal charges generated by photoelectric conversion. Next, when the high-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned on, and the signal charge accumulated in the photodiode 310A is transferred to the floating diffusion region 313. Then, the amplification transistor 315 amplifies the voltage of the floating diffusion region 313 according to the amount of signal charge transferred from the photodiode 310A. When the high-level selection pulse φSELn is output from the vertical scanning circuit 307, the selection transistor 317 is turned on, and the voltage amplified by the amplification transistor 315 is output to the signal output line 304 via the selection transistor 317.

  Similarly, when the low-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned off, and the photodiode 310B starts accumulating signal charges generated by photoelectric conversion. Next, when the high-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned on, and the signal charge accumulated in the photodiode 310B is transferred to the floating diffusion region 313. Then, the amplification transistor 315 amplifies the voltage of the floating diffusion region 313 according to the amount of signal charge transferred from the photodiode 310B. When the high-level selection pulse φSELn is output from the vertical scanning circuit 307, the selection transistor 317 is turned on, and the voltage amplified by the amplification transistor 315 is output to the signal output line 304 via the selection transistor 317.

  9 and 10 are plan layout diagrams showing the main part of the pixel 303. FIG. 9 shows photodiodes 310A and 310B, transfer transistors 311A and 311B, and a floating diffusion region 313 among the components of the pixel 303. Other circuit elements including the reset transistor 314, the amplification transistor 315, and the selection transistor 317 are represented as a readout circuit 321 in the drawing, and detailed illustration is omitted. Further, the signal output line 304 and the power supply line 305 arranged in the vertical direction of the pixel 303 are omitted, and the contact portions of the reset control line 319, the power supply line 305, and the ground line 306 are omitted. FIG. 10 shows the light guide 255 described in FIG. 4 in addition to the components shown in FIG. In the light guide 255, a hatched portion indicates a low refractive index region, and a white portion indicates a high refractive index region, that is, a light guide portion.

  9 and 10, the contact portion 312A is a contact portion that connects the transfer control line 320A and the gate of the transfer transistor 311A. The contact portion 312B is a contact portion that connects the transfer control line 320B and the gate of the transfer transistor 311B. The photodiodes 310 </ b> A and 310 </ b> B are photoelectric conversion units that perform photoelectric conversion. The photodiodes 310 </ b> A and 310 </ b> B are a first conductivity type (for example, P type) semiconductor region and a second conductivity type (for example, a PN junction with the first conductivity type semiconductor region). N-type) semiconductor region (N-type electron storage region). The second conductivity type semiconductor region of the photodiode 310A and the second conductivity type semiconductor region of the photodiode 310B are separated by a separation unit 322.

  The transfer transistors 311A and 311B, the contact portions 312A and 312B, and the transfer control lines 320A and 320B are arranged in line symmetry or substantially line symmetry with respect to the separation portion 322 between the photodiodes 310A and 310B, respectively. On the other hand, the light guide 255 is disposed at a position biased with respect to the separation portion 322 as shown in FIG. That is, the photodiode 310A occupies a large area of the bottom portion of the light guide 255, whereas the photodiode 310B is slightly hooked on the bottom portion of the light guide 255. As a result, the light receiving efficiency of the photodiode 310A is high, and the light receiving efficiency of the photodiode 310B is low.

  In the image sensor 184 according to the present embodiment, the ratio of the light receiving efficiencies of the photodiodes 310A and 310B is set to about 8: 1, that is, the sensitivity difference is set to about three stages. And while shooting two images with different storage time settings, the two pixel elements get the same signal charge, both have good SN ratio and no noise, or This makes it possible to synthesize high HDR video. Details will be described later.

FIG. 11 is a circuit diagram illustrating a configuration example of the readout circuit 308 of the image sensor 184.
As shown in FIG. 11, the read circuit 308 includes a clamp capacitor C0, a feedback capacitor Cf, an operational amplifier 406, a reference voltage source 407, and a switch 423. One input terminal of the operational amplifier 406 is connected to the signal output line 304 via the clamp capacitor C0. A feedback capacitor Cf and a switch 423 are connected in parallel between the one input terminal and the output terminal of the operational amplifier 406. The other input terminal of the operational amplifier is connected to the reference voltage source 407. The reference voltage source 407 is for supplying the reference voltage Vref to the operational amplifier 406. The switch 423 is a switch controlled by the signal PC0R, and is turned on when the signal PC0R is at a high level, and short-circuits both ends of the feedback capacitor Cf.

  The readout circuit 308 also includes switches 414A, 415A, 414B, 415B, 418A, 418B, 419A, 419B, capacitors CTSA, CTNA, CTSB, CTNB, horizontal output lines 424, 425, and an output amplifier 421. The switches 414A, 415A, 414B, and 415B are switches that control writing of pixel signals to the capacitors CTSA, CTNA, CTSB, and CTNB, respectively. The switch 414A is a switch controlled by the signal PTSA and is turned on when the signal PTSA is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTSA. The switch 415A is a switch controlled by the signal PTNA and is turned on when the signal PTNA is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTNA. The switch 414B is a switch controlled by the signal PTSB and is turned on when the signal PTSB is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTSB. The switch 415B is a switch controlled by the signal PTNB and is turned on when the signal PTNB is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTNB.

  The switches 418A, 419A, 418B, and 419B are switches for controlling the output of the pixel signals held in the capacitors CTSA, CTNA, CTSB, and CTNB to the output amplifier 421. Switches 418A, 419A, 418B, and 419B are turned on in response to a control signal from the horizontal shift register. As a result, the signal written in the capacitor CTSA is output to the output amplifier 421 via the switch 418A and the horizontal output line 424. The signal written in the capacitor CTNA is output to the output amplifier 421 through the switch 419A and the horizontal output line 425. Similarly, the signal written in the capacitor CTSB is output to the output amplifier 421 via the switch 418B and the horizontal output line 424. The signal written in the capacitor CTNB is output to the output amplifier 421 through the switch 419B and the horizontal output line 425.

  The signal PC0R, the signal PTNA, the signal PTSA, the signal PTNB, and the signal PTSB are signals supplied from the timing generation unit 189 under the control of the system control CPU 178 that is a control unit.

  Next, the driving method of the imaging apparatus according to the present embodiment will be described with reference to the timing chart of FIG. Here, it is assumed that the image sensor 184 has an m-row pixel array 302. FIG. 12 is an example in the case where control is performed so that the accumulation start timings of the photodiode 310A constituting the pixel element 303A and the photodiode 310B constituting the pixel element 303B of the pixel 303 arranged on the same line are the same. is there.

  First, at time t1, the vertical scanning circuit 307 causes the transfer pulses φTX1A and φTX1B supplied to the transfer control lines 320A and 320B to transition from the low level to the high level. As a result, the transfer transistors 311A and 311B are turned on. At this time, the high level reset pulse φRES1 is supplied from the vertical scanning circuit 307 to the reset control line 319, and the reset transistor 314 is also in the ON state. As a result, the photodiodes 310A and 310B of the pixels 303 in the first row are connected to the power supply line 305 via the transfer transistors 311A and 311B and the reset transistor 314 to be in a reset state. At this time, the floating diffusion region 313 is also reset.

Next, at time t2, the vertical scanning circuit 307 changes the transfer pulses φTX1A and φTX1B from the high level to the low level. As a result, the transfer transistors 311A and 311B are turned off, and accumulation of signal charges by photoelectric conversion starts in the photodiodes 310A and 310B of the pixels 303 in the first row. That is, time t2 is the start of the accumulation period of the photodiodes 310A and 310B of the pixels 303 in the first row.
Next, the same procedure is repeated to sequentially start accumulation of signal charges in the photodiodes 310A and 310B of the pixels 303 in the second to (m-1) th rows.

  Next, at time t3, the vertical scanning circuit 307 causes the transfer pulses φTXmA and φTXmB supplied to the transfer control lines 320A and 320B to transition from the low level to the high level. Thereby, the photodiodes 310A and 310B of the pixels 303 in the m-th row are reset as in the first row.

  Next, at time t4, the vertical scanning circuit 307 changes the transfer pulses φTXmA and φTXmB from the high level to the low level. Thereby, accumulation of signal charges in the photodiodes 310A and 310B of the pixels 303 in the m-th row is started. That is, time t4 is the beginning of the accumulation period of the photodiodes 310A and 310B of the pixels 303 in the m-th row.

  Next, at time t5, the vertical scanning circuit 307 changes the selection pulse φSEL1 supplied to the selection signal line from the low level to the high level, and turns on the selection transistors 317 of the pixels 303 in the first row. Thereby, the pixels 303 in the first row are selected.

  Next, at time t6, the vertical scanning circuit 307 makes the reset pulse φRES1 transition from the high level to the low level. As a result, the reset transistor 314 of the pixel 303 in the first row is turned off, and the reset of the floating diffusion region 313 is released. As a result, the potential of the floating diffusion region 313 is read as a reset signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308.

  At time t6, the high-level signal PC0R is supplied from the timing generator 189 to the reading circuit 308, and the switch 423 is in the on state. Therefore, the pixel signal at the reset signal level is input to the readout circuit 308 in a state where the operational amplifier 406 buffers the output of the reference voltage Vref.

  Next, at time t <b> 7, the signal PC <b> 0 </ b> R supplied from the timing generation unit 189 to the reading circuit 308 is changed from the high level to the low level, and the switch 423 is turned off. As a result, in the readout circuit 308, a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf with respect to the pixel signal at the reset signal level is output from the operational amplifier 406.

  Next, at time t8, the signal PTNA supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 415A, and the output of the operational amplifier 406 at that time is written into the capacitor CTNA.

  Next, at time t9, the signal PTNA supplied from the timing generation unit 189 to the reading circuit 308 is changed from a high level to a low level, the switch 415A is turned off, and writing to the capacitor CTNA is completed.

  Next, at time t10, the vertical scanning circuit 307 changes the transfer pulse φTX1A from the low level to the high level, and turns on the transfer transistor 311A. As a result, the signal charge accumulated in the photodiode 310A is transferred to the floating diffusion region 313.

  Next, at time t11, the vertical scanning circuit 307 changes the transfer pulse φTX1A from the high level to the low level, and turns off the transfer transistor 311A. Thereby, the reading of the signal charges accumulated in the photodiode 310A to the floating diffusion region 313 is completed. That is, time t11 is the end of the accumulation period of the photodiode 310A of the pixel 303 in the first row. The accumulation period of the photodiode 310A of the pixel 303 in the first row is from time t2 to time t11.

  As a result, the potential of the floating diffusion region 313 changed by the signal charge is read as an optical signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308. In the readout circuit 308, the operational amplifier 406 outputs a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C 0 and the feedback capacitor Cf with respect to the pixel signal at the optical signal level.

  Next, at time t12, the signal PTSA supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 414A, and the output of the operational amplifier 406 at that time is written into the capacitor CTSA.

  Next, at time t13, the signal PTSA supplied from the timing generation unit 189 to the reading circuit 308 is changed from the high level to the low level, the switch 414A is turned off, and writing to the capacitor CTSA is completed.

  Next, at time t <b> 14, the vertical scanning circuit 307 changes the selection pulse φSEL <b> 1 from the high level to the low level, turns off the selection transistor 317 of the pixel 303 in the first row, and sets the pixel 303 in the first row. Cancel the selection.

  Thereafter, in response to the signal from the horizontal shift register, the switches 418A and 419A are sequentially turned on for each column. As a result, the signal written to the capacitor CTSA is input to the output amplifier 421 via the horizontal output line 424, and the signal written to the capacitor CTNA is input to the output amplifier 421 via the horizontal output line 425, respectively.

  In this manner, reading from the pixel element 303A of the pixel 303 in each column of the first row is performed.

  Next, in the same manner as the reading from the pixel element 303A of the pixel 303 in each column of the first row described above, the reading from the pixel element 303A of the pixel 303 in each column of the (m-1) th row is performed. I do.

  Next, at time t15, the vertical scanning circuit 307 changes the reset pulse φRES1 from the low level to the high level to turn on the reset transistor 314. As a result, the floating diffusion region 313 is connected to the power supply line 305 via the reset transistor 314 and is in a reset state.

  Next, at time t <b> 16, the timing generation unit 189 changes the signal PC <b> 0 </ b> R from the low level to the high level and turns on the switch 423. As a result, the operational amplifier 406 enters a state of buffering the output of the reference voltage Vref.

  Next, at time t17, the vertical scanning circuit 307 changes the selection pulse φSELm from the low level to the high level, and turns on the selection transistor 317 of the pixel 303 in the m-th row. Thereby, the pixel 303 in the m-th row is selected.

  Next, at time t18, the vertical scanning circuit 307 causes the reset pulse φRESm to transition from the high level to the low level. Thereby, the reset transistor 314 of the pixel 303 in the m-th row is turned off, and the reset of the floating diffusion region 313 is released. As a result, the potential of the floating diffusion region 313 is read as a reset signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308.

  At time t18, the high-level signal PC0R is supplied from the timing generation unit 189 to the reading circuit 308, and the switch 423 is on. Therefore, the pixel signal at the reset signal level is input to the readout circuit 308 in a state where the operational amplifier 406 buffers the output of the reference voltage Vref.

  Next, at time t <b> 19, the signal PC <b> 0 </ b> R supplied from the timing generation unit 189 to the reading circuit 308 is changed from the high level to the low level, and the switch 423 is turned off. As a result, in the readout circuit 308, a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf with respect to the pixel signal at the reset signal level is output from the operational amplifier 406.

  Next, at time t20, the signal PTNA supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 415A, and the output of the operational amplifier 406 at that time is written to the capacitor CTNA.

  Next, at time t21, the signal PTNA supplied from the timing generation unit 189 to the reading circuit 308 is changed from the high level to the low level, the switch 415A is turned off, and writing to the capacitor CTNA is completed.

  Next, at time t22, the vertical scanning circuit 307 changes the transfer pulse φTXmA from the low level to the high level, and turns on the transfer transistor 311A. As a result, the signal charge accumulated in the photodiode 310A is transferred to the floating diffusion region 313.

  Next, at time t23, the vertical scanning circuit 307 changes the transfer pulse φTXmA from the high level to the low level, and turns off the transfer transistor 311A. Thereby, the reading of the signal charges accumulated in the photodiode 310A to the floating diffusion region 313 is completed. That is, time t23 is the end of the accumulation period of the photodiode 310A of the pixel 303 in the m-th row. The accumulation period of the photodiode 310A of the pixel 303 in the m-th row is from time t4 to time t23.

  As a result, the potential of the floating diffusion region 313 changed by the signal charge is read as an optical signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308. In the readout circuit 308, a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C 0 and the feedback capacitor Cf with respect to the pixel signal at the optical signal level is output from the operational amplifier 406.

  Next, at time t24, the signal PTSA supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 414A, and the output of the operational amplifier 406 at that time is written into the capacitor CTSA.

  Next, at time t25, the signal PTSA supplied from the timing generation unit 189 to the reading circuit 308 is changed from a high level to a low level, the switch 414A is turned off, and writing to the capacitor CTSA is completed.

  Next, at time t <b> 26, the vertical scanning circuit 307 changes the selection pulse φSELm from the high level to the low level, turns off the selection transistor 317 of the m-th row pixel 303, and sets the m-th row pixel 303. Cancel the selection.

  Thereafter, in response to the signal from the horizontal shift register, the switches 418A and 419A are sequentially turned on for each column. As a result, the signal written to the capacitor CTSA is input to the output amplifier 421 via the horizontal output line 424, and the signal written to the capacitor CTNA is input to the output amplifier 421 via the horizontal output line 425, respectively.

  In this manner, reading from the pixel element 303A of the pixel 303 in each column of the m-th row is performed.

  The above-described readout from the pixel element 303A of the pixels 303 from the first row to the m-th row is performed during the accumulation period of the pixel element 303B. That is, reading from the pixel element 303A is completed before reading from the pixel element 303B to be performed thereafter. Therefore, the reading operation of the pixel element 303A does not affect the reading frame rate of the pixel element 303B.

  Next, at time t27, the vertical scanning circuit 307 changes the reset pulse φRESm from the low level to the high level, and turns on the reset transistor 314. As a result, the floating diffusion region 313 is connected to the power supply line 305 via the reset transistor 314 and is in a reset state.

  Next, at time t <b> 28, the timing generation unit 189 changes the signal PC <b> 0 </ b> R from the low level to the high level and turns on the switch 423. As a result, the operational amplifier 406 enters a state of buffering the output of the reference voltage Vref.

  Next, reading from the pixel element 303A ends, and after a predetermined time corresponding to the accumulation time of the pixel element 303B has elapsed, reading from the pixel element 303B is started.

  At time t29 after a predetermined time has elapsed, the vertical scanning circuit 307 changes the selection pulse φSEL1 from the low level to the high level, and turns on the selection transistors 317 of the pixels 303 in the first row. Thereby, the pixels 303 in the first row are selected.

  Next, at time t30, the vertical scanning circuit 307 changes the reset pulse φRES1 from the high level to the low level. As a result, the reset transistor 314 of the pixel 303 in the first row is turned off, and the reset of the floating diffusion region 313 is released. As a result, the potential of the floating diffusion region 313 is read as a reset signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308.

  At time t30, the high-level signal PC0R is supplied from the timing generation unit 189 to the reading circuit 308, and the switch 423 is on. Therefore, the pixel signal at the reset signal level is input to the readout circuit 308 in a state where the operational amplifier 406 buffers the output of the reference voltage Vref.

  Next, at time t <b> 31, the signal PC <b> 0 </ b> R supplied from the timing generation unit 189 to the reading circuit 308 transitions from a high level to a low level, and the switch 423 is turned off. As a result, in the readout circuit 308, a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf with respect to the pixel signal at the reset signal level is output from the operational amplifier 406.

  Next, at time t32, the signal PTNB supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 415B, and the output of the operational amplifier 406 at that time is written to the capacitor CTNB.

  Next, at time t33, the signal PTNB supplied from the timing generation unit 189 to the reading circuit 308 is changed from a high level to a low level, the switch 415B is turned off, and writing to the capacitor CTNB is completed.

  Next, at time t34, the vertical scanning circuit 307 changes the transfer pulse φTX1B from the low level to the high level, and turns on the transfer transistor 311B. As a result, the signal charge accumulated in the photodiode 310B is transferred to the floating diffusion region 313.

  Next, at time t35, the vertical scanning circuit 307 changes the transfer pulse φTX1B from the high level to the low level, and turns off the transfer transistor 311B. Thereby, the reading of the signal charge accumulated in the photodiode 310B to the floating diffusion region 313 is completed. That is, time t35 is the end of the accumulation period of the photodiode 310B of the pixel 303 in the first row. The accumulation period of the photodiode 310B of the pixel 303 in the first row is from time t2 to time t35.

  As a result, the potential of the floating diffusion region 313 changed by the signal charge is read as an optical signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308. In the readout circuit 308, the operational amplifier 406 outputs a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf with respect to the pixel signal at the optical signal level.

  In the driving method of the imaging apparatus according to the present embodiment, the accumulation time of the photodiode 310B of the pixel element 303B is constant. As will be described later, the accumulation time of the photodiode 310B is appropriately set so as to obtain a high-quality moving image in which so-called jerkiness is suppressed, such as frame advance.

  The second set time T2 that is a parameter for controlling the accumulation timing of the photodiode 310A of the pixel element 303A is, for example, half the accumulation time of the photodiode 310B of the pixel element 303B, that is, T2 = (t35−t2) / 2 is set. A method for controlling the accumulation timing of the photodiode 310A of the pixel element 303A using the second set time T2 will be described later.

  Next, at time t36, the signal PTSB supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 414B, and the output of the operational amplifier 406 at that time is written to the capacitor CTSB.

  Next, at time t37, the signal PTSB supplied from the timing generation unit 189 to the reading circuit 308 is changed from the high level to the low level, the switch 414B is turned off, and the writing to the capacitor CTSB is completed.

  Next, at time t <b> 38, the vertical scanning circuit 307 changes the selection pulse φSEL <b> 1 from the high level to the low level, turns off the selection transistor 317 of the pixel 303 in the first row, and sets the pixel 303 in the first row. Cancel the selection.

  Thereafter, in response to a signal from the horizontal shift register, the switches 418B and 419B are sequentially turned on for each column. As a result, the signal written to the capacitor CTSB is input to the output amplifier 421 via the horizontal output line 424, and the signal written to the capacitor CTNB is input to the output amplifier 421.

  In this manner, reading from the pixel element 303B of the pixel 303 in each column of the first row is performed.

  Next, at time t39, the vertical scanning circuit 307 changes the reset pulse φRES1 from the low level to the high level, and turns on the reset transistor 314. As a result, the floating diffusion region 313 is connected to the power supply line 305 via the reset transistor 314 and is in a reset state.

  Next, at time t40, the timing generation unit 189 changes the signal PC0R from the low level to the high level, and turns on the switch 423. As a result, the operational amplifier 406 enters a state of buffering the output of the reference voltage Vref.

  Next, at time t41, the vertical scanning circuit 307 causes the transfer pulses φTX1A and φTX1B supplied to the transfer control lines 320A and 320B to transition from the low level to the high level. As a result, the transfer transistors 311A and 311B are turned on. At this time, the high level reset pulse φRES1 is supplied from the vertical scanning circuit 307 to the reset control line 319, and the reset transistor 314 is also in the ON state. As a result, the photodiodes 310A and 310B of the pixels 303 in the first row are connected to the power supply line 305 via the transfer transistors 311A and 311B and the reset transistor 314 to be in a reset state. At this time, the floating diffusion region 313 is also reset.

  Next, at time t42, the vertical scanning circuit 307 causes the transfer pulses φTX1A and φTX1B to transition from the high level to the low level. As a result, the transfer transistors 311A and 311B are turned off, and accumulation of signal charges by photoelectric conversion starts in the photodiodes 310A and 310B of the pixels 303 in the first row. Further, accumulation of the photodiodes 310A and 310B of the pixels 303 in each row is started sequentially. That is, time t42 is the beginning of the accumulation period of the next frame of the photodiodes 310A and 310B of the pixels 303 in the first row.

  Next, in the same manner as the readout from the pixel element 303B of the pixel 303 in each column of the first row described above, the readout from the pixel element 303B of the pixel 303 in each column of the (m-1) th row is performed. I do.

  Next, at time t43, the vertical scanning circuit 307 changes the selection pulse φSELm from the low level to the high level, and turns on the selection transistor 317 of the pixel 303 in the m-th row. Thereby, the pixels 303 in the first row are selected.

  Next, at time t44, the vertical scanning circuit 307 causes the reset pulse φRESm to transition from the high level to the low level. Thereby, the reset transistor 314 of the pixel 303 in the m-th row is turned off, and the reset of the floating diffusion region 313 is released. As a result, the potential of the floating diffusion region 313 is read as a reset signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308.

  At time t44, the high-level signal PC0R is supplied from the timing generation unit 189 to the reading circuit 308, and the switch 423 is on. Therefore, the pixel signal at the reset signal level is input to the readout circuit 308 in a state where the operational amplifier 406 buffers the output of the reference voltage Vref.

  Next, at time t <b> 45, the signal PC <b> 0 </ b> R supplied from the timing generation unit 189 to the reading circuit 308 transitions from a high level to a low level, and the switch 423 is turned off. As a result, in the readout circuit 308, a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf with respect to the pixel signal at the reset signal level is output from the operational amplifier 406.

  Next, at time t46, the signal PTNB supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 415B, and the output of the operational amplifier 406 at that time is written into the capacitor CTNB.

  Next, at time t47, the signal PTNB supplied from the timing generation unit 189 to the reading circuit 308 is changed from the high level to the low level, the switch 415B is turned off, and writing to the capacitor CTNB is completed.

  At time t48, the vertical scanning circuit 307 changes the transfer pulse φTXmB from the low level to the high level, and turns on the transfer transistor 311B. As a result, the signal charge accumulated in the photodiode 310B is transferred to the floating diffusion region 313.

  Next, at time t49, the vertical scanning circuit 307 changes the transfer pulse φTXmB from the high level to the low level, and turns off the transfer transistor 311B. Thereby, the reading of the signal charge accumulated in the photodiode 310B to the floating diffusion region 313 is completed. That is, time t49 is the end of the accumulation period of the photodiode 310B of the pixel 303 in the m-th row. The accumulation period of the photodiode 310B of the pixel 303 in the m-th row is from time t4 to time t49.

  As a result, the potential of the floating diffusion region 313 changed by the signal charge is read as an optical signal level to the signal output line 304 via the amplification transistor 315 and the selection transistor 317 and input to the reading circuit 308. In the readout circuit 308, the operational amplifier 406 outputs a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf with respect to the pixel signal at the optical signal level.

  Next, at time t50, the signal PTSB supplied from the timing generation unit 189 to the reading circuit 308 is changed from the low level to the high level to turn on the switch 414B, and the output of the operational amplifier 406 at that time is written to the capacitor CTSB.

  Next, at time t51, the signal PTSB supplied from the timing generation unit 189 to the reading circuit 308 is changed from the high level to the low level, the switch 414B is turned off, and the writing to the capacitor CTSB is completed.

  Next, at time t <b> 52, the vertical scanning circuit 307 changes the selection pulse φSELm from the high level to the low level, turns off the selection transistor 317 of the m-th row pixel 303, and sets the m-th row pixel 303. Cancel the selection.

  Thereafter, in response to a signal from the horizontal shift register, the switches 418B and 419B are sequentially turned on for each column. As a result, the signal written to the capacitor CTSB is input to the output amplifier 421 via the horizontal output line 424, and the signal written to the capacitor CTNB is input to the output amplifier 421.

  In this manner, reading from the pixel element 303B of the pixel 303 in each column of the m-th row is performed.

  Next, at time t53, the vertical scanning circuit 307 changes the reset pulse φRESm from the low level to the high level, and turns on the reset transistor 314. As a result, the floating diffusion region 313 is connected to the power supply line 305 via the reset transistor 314 and is in a reset state.

  Next, at time t54, the timing generation unit 189 changes the signal PC0R from the low level to the high level, and turns on the switch 423. As a result, the operational amplifier 406 enters a state of buffering the output of the reference voltage Vref.

  Next, at time t55, the vertical scanning circuit 307 transitions the transfer pulses φTXmA and φTXmB supplied to the transfer control lines 320A and 320B from the low level to the high level. As a result, the transfer transistors 311A and 311B are turned on. At this time, a high level reset pulse φRESm is output from the vertical scanning circuit 307 to the reset control line 319, and the reset transistor 314 is also in the ON state. As a result, the photodiodes 310A and 310B of the pixel 303 in the m-th row are connected to the power supply line 305 via the transfer transistors 311A and 311B and the reset transistor 314, and are in a reset state. At this time, the floating diffusion region 313 is also reset.

  Next, at time t56, the vertical scanning circuit 307 changes the transfer pulses φTXmA and φTXmB from the high level to the low level. Accordingly, the transfer transistors 311A and 311B are turned off, and accumulation of signal charges by photoelectric conversion is started in the photodiodes 310A and 310B of the pixels 303 in the m-th row. That is, time t56 is the start of the accumulation period of the next frame of the photodiodes 310A and 310B of the pixel 303 in the m-th row.

  The first set time T1, which is a parameter for controlling the accumulation timing of the photodiode 310A of the pixel element 303A, is, for example, a time corresponding to a period for reading all the pixel elements 303B, that is, T1 = t56-t35. Is set. A method for controlling the accumulation timing of the photodiode 310A of the pixel element 303A using the first set time T1 will be described later.

  FIG. 13 is a potential diagram of the pixel 303 along the line AB in FIG. 13A is a potential diagram at time ta in FIG. 12, FIG. 13B is a potential diagram at time tb in FIG. 12, and FIG. 13C is a potential diagram at time tc in FIG.

  At time ta, as shown in FIG. 13A, the transfer transistors 311A and 311B are in an off state, and signal charges of signal accumulation levels 323A and 323B are accumulated in the photodiodes 310A and 310B, respectively. As described above, since the light receiving efficiency is different between the photodiode 310A and the photodiode 310B, the signal accumulation level 323A at the same time is higher than the signal accumulation level 323B.

  At time tb, as shown in FIG. 13B, the transfer transistor 311A is on. In this case, the potential barrier of the transfer transistor 311A is lowered, and the signal charge accumulated in the photodiode 310A is transferred to the floating diffusion region 313. At this time, the potential barrier of the separation unit 322 is also lowered due to the reduction of the potential barrier of the transfer transistor 311A. However, since the potential barrier of the transfer transistor 311A is sufficiently small, the phenomenon that the signal charge accumulated in the photodiode 310A leaks to the adjacent photodiode 310B via the separation portion 322 at this timing hardly occurs.

  At time tc, as shown in FIG. 13C, the transfer transistor 311B is in the on state. In this case, the potential barrier of the transfer transistor 311B is lowered, and the signal charge accumulated in the photodiode 310B is transferred to the floating diffusion region 313. At this time, the signal charge accumulated in the photodiode 310 </ b> A is read out to the reading circuit 308. Further, the potential barrier of the separation unit 322 is also lowered due to the lowering of the potential barrier of the transfer transistor 311B. However, since the potential barrier of the transfer transistor 311B is sufficiently small, the signal charge accumulated in the photodiode 310B at this timing hardly leaks to the adjacent photodiode 310A via the separation unit 322.

  FIG. 14 is a cross-sectional view showing the behavior of signal charges generated by light propagation and photoelectric conversion inside the image sensor 184. The light beam 451 incident on the pixel 303 first enters the color filter 256 and a predetermined wavelength component is absorbed here, passes through an interface deactivation film (not shown) corresponding to the uppermost portion of the insulating layer 254, and the light guide. Incident to 255. In the light guide 255, as described with reference to FIG. 5, the light beam azimuth information, that is, pupil information, disappears due to the wave behavior of light. The light beam 451 proceeds to the silicon substrate 251 side while being confined inside the light guide 255 due to the difference in refractive index between the light guide 255 and the insulating layer 254, and reaches the bottom portion of the light guide 255. The bottom portion of the light guide 255 is adjacent to the silicon substrate 251, and the light beam emitted from the light guide 255 is incident on the silicon substrate 251. The photodiode 310A and the photodiode 310B provided adjacent to each other in the silicon substrate 251 are arranged so as to be greatly decentered with respect to the light guide 255. Therefore, most of the light beam 452 emitted from the light guide 255 is incident on the photodiode 310A, and the remaining part of the light beam 453 emitted from the light guide 255 is incident on the photodiode 310B. In the photodiodes 310A and 310B, incident photons are converted into signal charges.

  At this time, signal charges generated inside the silicon substrate 251 of the image sensor 184 leak into adjacent pixel elements by diffusion. For example, the signal charge 454 generated in the photodiode 310A leaks into the photodiode 310B by diffusion. Also. The signal charge 455 generated in the photodiode 310B leaks into the photodiode 310A by diffusion. This phenomenon has an adverse effect on the video and appears as blurring of the image.

  FIG. 15 is a diagram for explaining a screen for setting the imaging conditions of “picture A” and “picture B”. Suppose that the shooting mode selection lever 156 is rotated 90 degrees clockwise from the position of FIG. 1B, for example, to enter a dual video mode in which two videos can be shot simultaneously. The display unit 153 displays the Bv value 521 corresponding to the luminance of the subject at that time, the F number 522, the ISO sensitivities 523 and 524 of “picture A” and “picture B”, and the shutter speeds 525 and 526, respectively. . In addition, currently set picture modes 527 and 528 are displayed for each of “picture A” and “picture B”. In the picture mode, the up / down switches 158 and 159 and the dial 160 can be used to select one that meets the purpose of shooting from a plurality of options.

  As described above, the difference in light receiving efficiency between the photodiode 310A and the photodiode 310B is set to three stages. Therefore, there is a three-stage difference in the ISO sensitivity range between “picture A” and “picture B”. As shown in FIG. 16, “picture A” is ISO 100 to ISO 102400, and “picture B” is ISO 12 to ISO 12800.

  FIG. 17 is a program AE (Automatic Exposure) diagram in the dual video mode. The horizontal axis represents the Tv value and the corresponding shutter speed, and the vertical axis represents the AV value and the corresponding aperture value. Further, the oblique direction is an equal Bv line. The relationship between the Bv value of “picture A” and the ISO sensitivity is represented in the gain notation area 556, and the relationship between the Bv value of “picture B” and the ISO sensitivity is represented in the gain notation area 557. In FIG. 17, each Bv value is represented by a numerical value surrounded by a square in order to distinguish it from other parameters.

  How the shutter speed, aperture value, and ISO sensitivity change as the brightness changes from high to low will be described with reference to FIG.

  First, in the case of Bv13, the ISO sensitivity is set to ISO100 in “picture A”. The equal Bv line of “picture A” intersects the program diagram 558 of “picture A” at a point 551, and the shutter speed 1/4000 and the aperture value F11 are determined from the point 551. On the other hand, in “picture B”, the ISO sensitivity is set to ISO12. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 552, and the shutter speed 1/500 and the aperture value F11 are determined from the point 552.

  In the case of Bv10, in “picture A”, the ISO sensitivity is increased by one step and set to ISO200. The equal Bv line of “picture A” intersects the program diagram 558 of “picture A” at a point 553, and the shutter speed 1/1000 and the aperture value F11 are determined from the point 553. On the other hand, in “picture B”, the ISO sensitivity is set to ISO12. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 560, and the shutter speed 1/60 and the aperture value F11 are determined from the point 560.

  In the case of Bv6, the ISO sensitivity is set to ISO200 in “picture A”. The equal Bv line of “picture A” intersects with the program diagram 558 of “picture A” at a point 554, and the shutter speed is 1/1000 and the aperture value F2.8 is determined from the point 554. On the other hand, in “picture B”, the ISO sensitivity is set to ISO12. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 555, and the shutter speed 1/60 and the aperture value F2.8 are determined from the point 555.

  In the case of Bv5, in “picture A”, the ISO sensitivity is increased by one step and set to ISO400. The equal Bv line of “picture A” intersects with the program diagram 558 of “picture A” at a point 554, and the shutter speed is 1/1000 and the aperture value F2.8 is determined from the point 554. On the other hand, in “picture B”, the ISO sensitivity is set to ISO25. The equal Bv line of “picture B” intersects the program diagram 559 of “picture B” at a point 555, and the shutter speed 1/60 and the aperture value F2.8 are determined from the point 555.

  Thereafter, as the brightness decreases, both the “picture A” and “picture B” increase the gain without changing the shutter speed and the aperture value, and the ISO sensitivity increases.

  By performing the exposure operation shown in the program AE diagram, “picture A” maintains a shutter speed of 1/1000 or more in the entire luminance range described, and “picture B” is 1/60 in many luminance ranges. The shutter speed is maintained. As a result, while “picture A” obtains a stop motion effect, “picture B” makes it possible to obtain a high-quality moving image in which so-called jerkiness is suppressed, such as frame advance.

  In general, when the shutter speed at the time of moving image shooting is high, so-called jerkiness such as frame advance appears at the time of reproduction, and the smoothness of the image is lost. In order to obtain such a smooth image with reduced jerkiness, it is necessary to set an accumulation time close to one frame period in a series of photographing. That is, if the frame rate is 30 fps, a relatively long accumulation time such as 1/30 seconds or 1/60 seconds is appropriate. This setting is particularly important in situations where the camera posture is unstable, such as aerial photography.

  On the other hand, in still images, it is required to shoot a video with a so-called stop motion effect that suppresses blur and stops a moment. For this reason, it is necessary to set a short accumulation time of, for example, about 1/1000 second. In addition, since one frame period is short in a high frame rate moving image, for example, if the frame rate is 120 fps, a short accumulation time such as 1/125 seconds or 1/250 seconds is inevitably set.

  Shooting two images of a moving image and a still image or a normal frame rate moving image and a high frame rate moving image simultaneously through a single shooting lens means that the aperture used for the shooting is the same. . At this time, while the two images are shot with different storage time settings, the image sensor obtains the same level of signal charge, and both of them have a good S / N ratio and no noise. Is desirable.

  FIG. 18 is a flowchart illustrating the driving method of the imaging apparatus according to the present embodiment. In the driving method of the imaging apparatus according to the present embodiment, the presence or absence of reading of the pixel element 303A, the accumulation start timing or the accumulation end timing of the pixel element 303A is changed according to the accumulation time of the pixel element 303A. Specifically, accumulation and readout of the pixel elements 303A and 303B are performed according to the procedure from step S101 to step S108 shown in FIG.

  First, when the luminance of the subject is detected, in step S101, the accumulation time Tk of the pixel element 303A is set according to the detected luminance of the subject.

  Next, in step S102, it is determined whether or not the set accumulation time Tk of the pixel element 303A is longer than the second set time T2. Here, the second set time T2 is set to a half of the accumulation time of the pixel element 303B. The second set time T2 does not necessarily have to be half of the accumulation time of the pixel element 303B, and can be defined as the longest accumulation time in which a stop motion effect can be expected.

  When the accumulation time Tk of the pixel element 303A is longer than the second set time T2 (“yes” in FIG. 18), the process proceeds to step S103. In this case, since “stop motion effect” cannot be expected from “picture A” obtained from the output of the pixel element A, reading from the pixel element 303A is stopped (step S103), and the process proceeds to step S108.

  On the other hand, when the accumulation time Tk of the pixel element 303A is equal to or shorter than the second set time T2 (“no” in FIG. 18), the process proceeds to step S104.

  In step S104, it is determined whether the accumulation time Tk of the pixel element 303A is equal to or shorter than the first set time T1. Here, the first set time T1 is set to a time corresponding to a period required to read all the pixel elements 303B of the image sensor 184 (for example, a period from time t35 to time t56).

  When the accumulation time TK of the pixel element 303A is equal to or shorter than the first set time T1 (“yes” in FIG. 18), the process proceeds to step S105. In step S105, the control unit sets the accumulation period of the pixel element 303A so that the accumulation period of the pixel element 303A ends after the readout period of the pixel element 303B of the previous frame. In one example, the accumulation period of the pixel element 303A is set so that the timing at which the accumulation period of the pixel element 303A in the first row ends coincides with the timing at which the accumulation period of the pixel element 303B in the m-th row coincides. . That is, the accumulation period of the pixel element 303A is set so that the end timing of the accumulation period of the pixel element 303A is fixed at a predetermined time corresponding to the accumulation period of the pixel element 303B.

  On the other hand, when the accumulation time TK of the pixel element 303A is longer than the first set time T1 (“no” in FIG. 18), the process proceeds to step S106. In step S106, the control unit sets the accumulation period of the pixel element 303A so that the accumulation period of the pixel element 303A does not end during the readout period of the pixel element 303B of the previous frame. In one example, the accumulation period of the pixel element 303A is set so that the timing of starting the accumulation period of the pixel element 303A in the first row coincides with the timing of starting the accumulation period of the pixel element 303B in the first row. . That is, the accumulation period of the pixel element 303A is set so that the start timing of the accumulation period of the pixel element 303A is fixed at a predetermined time according to the accumulation period of the pixel element 303B.

  In this way, after setting the accumulation period of the pixel element 303A in step S105 or step S106, the process proceeds to step S107.

  When the accumulation timing of the pixel element 303A is set, the control unit executes accumulation and readout of the pixel element 303A in step S107. After the accumulation and readout of the pixel element 303A are completed, the process proceeds to step S108.

  Next, in step S108, the control unit executes accumulation and readout of the pixel element 303B.

  The method for setting the accumulation period of the pixel element 303A in step S105 and step S106 will be described in more detail with reference to FIG. FIG. 19 is a diagram illustrating a difference in shutter speed between “picture A” that is the output of the pixel element 303A and “picture B” that is the output of the pixel element 303B on the imaging sequence. In the figure, the horizontal axis indicates time, and the V synchronization signal 481, “picture A” accumulation periods 482 and 483, and “picture B” accumulation periods 484 and 485 are shown. n is a frame number.

  FIG. 19A is an example of an imaging sequence when the accumulation time Tk of the pixel element 303A is equal to or shorter than the first set time T1 (Tk ≦ T1). In FIG. 19A, the accumulation period 482 is the accumulation period of the screen top line (first line) of “picture A”, and the accumulation period 483 is the bottom line (m-th line) of “picture A”. It is an accumulation period. In order to perform an exposure operation with the rolling electronic shutter function, the image sensor 184 sequentially starts accumulation at a predetermined time interval from the line at the upper end of the screen toward the line at the lower end of the screen, and ends the accumulation at the time interval. When the accumulation is completed, the signal charges are sequentially read out by the reading circuit 308. The accumulation period 482 is from time t60 to time t4, and the accumulation period 483 is from time t61 to time t23.

  Further, the accumulation period 484 is an accumulation period of the “picture B” screen upper end line (first line), and the accumulation period 485 is an accumulation period of the “picture B” screen lower end line (m-th line). Similarly to “picture A” in “picture B”, accumulation starts sequentially from a line at the upper end of the screen toward a line at the lower end of the screen at a predetermined time interval, and the accumulation ends at the time interval. When the accumulation is completed, the signal charges are sequentially read out by the reading circuit 308. The accumulation period 484 is from time t2 to time t35, and the accumulation period 485 is from time t4 to time t49. In the driving method of the imaging apparatus according to the present embodiment, the accumulation time of “picture B” is fixed.

  Here, in the case of Tk ≦ T1, for example, if the accumulation timing of the pixel element 303A and the accumulation period of the pixel element 303B are made the same, the pixel element 303A is read before the reading of the pixel element 303B of the previous frame is completed. May start. For example, when the accumulation period 482 starts at time t2, the accumulation period 482 ends before time t4. In this case, the readout period of the pixel element 303A overlaps with the readout period of the pixel element 303B of the previous frame, which affects the readout frame rate of the pixel element 303B.

  Therefore, in the driving method of the imaging apparatus according to the present embodiment, in step S106, the end timing of the accumulation period of the pixel element 303A is set according to the start timing of the accumulation period of the pixel element 303B. In one example, as shown in FIG. 19A, the end timing of the accumulation period of the pixel element 303A in the first row is the same as the start timing of the accumulation period of the pixel element 303B in the m-th row. The accumulation period of the pixel element 303A is set. Note that the start timing of the accumulation period of the pixel element 303B in the m-th row is also the end timing of the readout period of the previous frame of the pixel element 303B. By doing so, the accumulation period of the pixel element 303A ends after the readout period of the pixel element 303B of the previous frame, and it is possible to prevent the frame rate of readout of the pixel element 303B from being affected.

  Note that the accumulation timing of the pixel element 303A in the first row is the timing at which the readout of the pixel element 303A in the m-th row ends by time t35, and the accumulation of the pixel element 303B in the m-th row is performed. It may be after the start timing of the period. For example, it is also effective to set the end timing of the accumulation period of the pixel element 303A so that the accumulation period of the pixel element 303A ends near the middle of the accumulation period of the pixel element 303B. This has the effect of bringing the centroid position of the accumulation time of the pixel element 303A closer to the centroid position of the accumulation time of the pixel element 303B.

  FIG. 19B is an example of an imaging sequence when the accumulation time Tk of the pixel element 303A is longer than the first set time T1 (Tk> T1). In FIG. 19B, the accumulation period 482 is the accumulation period of the screen upper end line (first line) of “picture A”, and the accumulation period 483 is the lower end line (m-th line) of “picture A”. It is an accumulation period. In order to perform an exposure operation with the rolling electronic shutter function, the image sensor 184 sequentially starts accumulation at a predetermined time interval from the line at the upper end of the screen toward the line at the lower end of the screen, and ends the accumulation at the time interval. When the accumulation is completed, the signal charges are sequentially read out by the reading circuit 308. The accumulation period 482 is from time t2 to time t11, and the accumulation period 483 is from time t4 to time t23.

  Further, the accumulation period 484 is an accumulation period of the “picture B” screen upper end line (first line), and the accumulation period 485 is an accumulation period of the “picture B” screen lower end line (m-th line). Similarly to “picture A” in “picture B”, accumulation starts sequentially from a line at the upper end of the screen toward a line at the lower end of the screen at a predetermined time interval, and the accumulation ends at the time interval. When the accumulation is completed, the signal charges are sequentially read out by the reading circuit 308. The accumulation period 484 is from time t2 to time t35, and the accumulation period 485 is from time t4 to time t49. In the driving method of the imaging apparatus according to the present embodiment, the accumulation time of “picture B” is fixed.

  Here, when Tk> T1, if the accumulation period of the pixel element 303A is started before the accumulation period of the pixel element 303B, the reading of the pixel element 303A starts before the reading of the pixel element 303B of the previous frame is completed. There is a risk. For example, if the accumulation period 482 is started before the time t2, the time t11 at which the accumulation period 482 ends before the time t4 may arrive. In this case, the readout period of the pixel element 303A overlaps with the readout period of the pixel element 303B of the previous frame, which affects the readout frame rate of the pixel element 303B.

  Therefore, in the driving method of the imaging apparatus according to the present embodiment, in step S105, the start timing of the accumulation period of the pixel element 303A is set according to the start timing of the accumulation period of the pixel element 303B. In one example, as shown in FIG. 19B, the accumulation period of the pixel element 303A is set so that the start timing of the accumulation period of the pixel element 303A is the same as the start timing of the accumulation period of the pixel element 303B. By doing so, the accumulation period of the pixel element 303A ends after the readout period of the pixel element 303B of the previous frame, and it is possible to prevent the frame rate of readout of the pixel element 303B from being affected.

  It should be noted that the accumulation timing of the pixel element 303A in the first row is the timing at which the readout of the pixel element 303A in the m-th row ends by time t35 when the accumulation period of the pixel element 303A in the first row ends. It may be after the start timing of the period. For example, it is also effective to set the start timing of the accumulation period of the pixel element 303A so that the accumulation period of the pixel element 303A ends near the middle of the accumulation period of the pixel element 303B. This has the effect of bringing the centroid position of the accumulation time of the pixel element 303A closer to the centroid position of the accumulation time of the pixel element 303B.

  As described above, in the driving method of the imaging apparatus according to the present embodiment, the accumulation period of the pixel element 303A is controlled so that the readout period of the pixel element 303A and the readout period of the pixel element 303B do not overlap. More specifically, the reading of the pixel element 303A ends during the accumulation time of the pixel element 303B and before the reading of the pixel element 303B. Thereby, the readout of the pixel element 303A does not affect the readout frame rate of the pixel element 303B.

  FIG. 20 is a diagram illustrating a state of the display unit 153 during live view display after the imaging apparatus 100 is powered on. On the display unit 153, the sports scene of the person 163 captured through the photographing optical system 152 is displayed. Also, since the shooting mode selection lever 156 is at a position rotated 90 degrees clockwise from the state of FIG. 1B, the shutter speeds 491 and 492 of “picture A” and “picture B” in the dual video mode and An F number 493 is displayed.

  FIG. 21 shows one frame of video obtained by operating the switch ST154 and the switch MV155. FIG. 21A is an image of “picture A” captured at a shutter speed of 1/1000 and an aperture value of F4.0. FIG. 21B is an image of “picture B” taken at a shutter speed of 1/60 and an aperture value of F4.0. The video shown in FIG. 21B is blurred without stopping the movement of the subject because the shutter speed is slow. However, when this is played back as a moving image having a frame rate of about 60 fps, the blur works in a rather good direction, resulting in a smooth high-quality image in which jerkiness is suppressed. On the other hand, the image shown in FIG. 21A has a high shutter speed, and a stop motion effect should appear. However, as described above with reference to FIG. 14, the signal charges generated inside the silicon substrate leak into adjacent pixel elements due to diffusion, and the image shown in FIG. 21B is blurred as if it were added together. It has become a video. This crosstalk phenomenon also occurs in the image shown in FIG. 21B, but is hardly noticeable because the image is originally blurred.

  Therefore, in order to obtain the original stop motion effect due to the fast shutter speed, the imaging apparatus according to the present embodiment performs crosstalk correction on the video signal output from the imaging element 184.

  FIG. 22 is a flowchart showing a series of processing procedures including crosstalk correction. Processing from imaging to recording in the imaging apparatus 100 according to the present embodiment is executed by, for example, steps S151 to S155 shown in FIG.

  In step S151, accumulation of signal charges in the photodiodes 310A and 310B and readout of the signal charges are performed according to the operation of the switch MV155 in accordance with the sequence described with reference to FIG.

  In step S153, correction (crosstalk correction) is performed to reduce crosstalk caused by signal charges generated inside the silicon substrate leaking into adjacent pixel elements. Crosstalk correction is performed in the digital signal processing unit 187. That is, the digital signal processing unit 187 functions as a crosstalk correction unit.

  In step S154, development and compression processing are performed as necessary. In the development process, gamma correction is performed as one of a series of processes. Gamma correction is a process for applying a gamma function to an input light amount distribution. As a result, the linearity of the output is not maintained with respect to the input light quantity distribution, and the crosstalk ratio also changes depending on the light quantity at that time. For this reason, it is desirable to perform the crosstalk correction at a stage prior to step S154 as shown in FIG. When crosstalk correction is performed after development, the crosstalk processing may be changed depending on the amount of light, or the video signal itself may be subjected to inverse gamma correction before crosstalk correction is performed.

  In step S155, video is recorded on the recording medium 193. Instead of recording on the recording medium 193 or together with recording on the recording medium 193, the data may be stored in a recording device or the like on the network via the wireless interface unit 198.

  FIG. 23 is a diagram for explaining the crosstalk correction processing performed in the digital signal processing unit 187 in step S153. Actual processing is executed as digital signal processing.

  In the digital signal processing unit 187, the signal 471A after A / D conversion processing is input to the crosstalk amount correction unit 473A, and also to the crosstalk amount correction unit 473B via the crosstalk amount calculation unit 472A. Entered. Similarly, in the digital signal processing unit 188, the signal 471B after A / D conversion processing is input to the crosstalk amount correction unit 473B, and the crosstalk amount correction is performed via the crosstalk amount calculation unit 472B. This is input to the unit 473A.

  In the crosstalk amount correcting unit 473A, the crosstalk correction of the signal 471A is performed based on the signal 471A and the signal 471B after the predetermined calculation is performed by the crosstalk correction function gij (n) in the crosstalk amount calculating unit 472B. To obtain an output signal 474A. The output signal 474A is subjected to development and compression processing, which are subsequent processing in the digital signal processing unit 187.

  The crosstalk correction unit 473B corrects the crosstalk of the signal 471B based on the signal 471B and the signal 471A after a predetermined calculation is performed by the crosstalk correction function fij (n) in the crosstalk calculation unit 472A. To obtain an output signal 474B. The output signal 474B is subjected to development and compression processing, which are subsequent processing in the digital signal processing unit 188.

  Since the crosstalk depends on the amount of signal charge to be generated, the crosstalk amount correction units 473A and 473B output the other pixel element by the amount of crosstalk corresponding to the amount of signal charge generated in one pixel element. Crosstalk correction can be performed by correcting the signal. Thereby, the crosstalk component from one pixel element superimposed on this can be removed from the output signal of the other pixel.

Here, the data at the pixel address ij of “picture A” in the nth frame is DATA_Aij (n), the data at the pixel address ij of “picture B” in the nth frame is DATA_Bij (n), and the correction coefficient is α. Since the crosstalk depends on the input light amount, the corrected data of the pixel address ij of “picture A” in the nth frame can be expressed by the equation (4) as C_DATA_Aij (n).
C_DATA_Aij (n)
= DATA_Aij (n) −α × DATA_Bij (n) (4)
The crosstalk correction function fij (n) is
fij (n) = − α × DATA_Bij (n)
Then, equation (4) becomes
C_DATA_Aij (n) = DATA_Aij (n) + fij (n)
It can be expressed as.

Similarly, the corrected data C_DATA_Bij (n) at the pixel address ij of “picture B” in the nth frame can be expressed as Equation (5), where β is the correction coefficient.
C_DATA_Bij (n)
= DATA_Bij (n) −β × DATA_Aij (n) (5)

The crosstalk correction function gij (n) is
gij (n) = − β × DATA_Aij (n)
Then, equation (5) becomes
C_DATA_Bij (n) = DATA_Bij (n) + gij (n) (6)
It can be expressed as.

  As described above, crosstalk occurs even in “picture B”, but it is hardly noticeable because the image is originally blurred. Therefore, the processing shown in equations (5) to (6) can be omitted. Good. If crosstalk correction is performed on a video with a relatively short storage time and crosstalk correction is not performed on a video with a relatively long storage time, the calculation load can be reduced.

  FIG. 24 is a diagram illustrating a specific example of the crosstalk correction functions fij (n) and gij (n). In the figure, the horizontal axis indicates the size of the input data, and the vertical axis indicates the crosstalk correction amount to be corrected. The crosstalk correction functions fij (n) and gij (n) are functions that obtain a crosstalk correction amount proportional to the input data. Strictly speaking, although different depending on the pixel structure, the correction coefficient α and the correction coefficient β are the same numerical values. However, the degree to which signal charges generated inside the silicon substrate leak into adjacent pixel elements due to diffusion differs depending on the incident angle of light to the pixel elements. For this reason, the crosstalk becomes larger and the absolute value of the crosstalk correction amount becomes larger as the F-number becomes larger when the aperture 181 is opened. On the other hand, as the F-number is reduced by reducing the aperture 181, the crosstalk is reduced and the absolute value of the crosstalk correction amount is also reduced. In the figure, characteristic 591 is a crosstalk correction function at F2.8, characteristic 592 is a crosstalk correction function at F5.6, and characteristic 593 is a crosstalk correction function at F11. The inclination decreases in the order of the characteristic 591, the characteristic 592, and the characteristic 593. Since the F number of the photographic optical system 152 can be continuously changed, more accurate crosstalk correction can be realized if the correction coefficient α and the correction coefficient β are functions of the F number.

  The correction coefficient α and the correction coefficient β may be functions of the accumulation time of the “picture A” photodiode set relatively short. Further, more accurate crosstalk correction can be realized by changing the crosstalk correction amount according to the image height. Since the crosstalk increases when the light incident on the light guide 255 becomes oblique, the distance ZK from the optical axis 180 to the pixel is calculated based on the pixel address ij, and the absolute value increases in proportion to the distance ZK. It is sufficient to add crosstalk correction as described above. Furthermore, since the change in the incident angle of the light to the light guide 255 also depends on the distance HK between the exit pupil of the photographing optical system 152 and the image sensor 184, the crosstalk correction function can be used as a function of the distance HK. High-precision correction can be performed.

  FIG. 25 is an image of “picture A” after crosstalk correction is performed on an image of “picture A” (FIG. 21A) taken at a shutter speed of 1/1000 and an aperture value of F4.0. is there. In the image of FIG. 21A, signal charges generated inside the silicon substrate leak into adjacent pixel elements due to diffusion, and the image shown in FIG. 21B is blurred as if they were added together. . On the other hand, in the video of FIG. 25, the stop motion effect due to the original fast shutter speed appears. On the display unit 153 of the digital still motion camera, when the play button 161 is operated, it is desirable that both “picture A” 496 and “picture B” 497 can be displayed side by side as shown in FIG. 26, for example. In this way, the level of the stop motion effect can be confirmed by comparing the images. In this process, the video data may be supplied to a system or apparatus via a network, and a computer of the system or apparatus may read and execute the program.

  FIG. 27 is a diagram for explaining an application example of “picture A” and “picture B” stored in the storage in a tablet terminal, a personal computer, a television monitor, and the like.

  It is assumed that the data files of “picture A” and “picture B” are stored in a storage on the network. In the figure, a frame group 581 is a “picture A” frame group stored in an MP4 file, and a frame group 571 is a “picture B” frame group stored in another MP4 file. These MP4 files are associated with the same CLIP-UMID set at the time of shooting.

  First, when playback of a moving image is started, frames are sequentially played back at a frame rate determined from the first frame 572 of the “picture B” frame group 571. Since “picture B” is shot at a setting that prevents the shutter speed from becoming too fast (in this example, 1/60 seconds), the reproduced video is high-quality with reduced jerkiness.

  Here, it is assumed that the user gives a playback mode switching instruction during the presentation of the “picture B” video. For example, when the user performs a pause operation at the time when playback has progressed to frame 573, a frame 582 of the same time code is automatically searched from the data file of “picture A” corresponding to “picture B”, Is displayed. “Picture A” is a powerful image that is captured at a high shutter speed (in this example, 1/1000 second) at which a stop motion effect is easily obtained, and captures a moment in a sports scene. Two pictures of “picture A” and “picture B” are taken with different accumulation time settings, but the gain of “picture A” is not increased, but a similar signal charge is obtained at the image sensor 184. Yes. For this reason, both “picture A” and “picture B” are images having a good S / N ratio and no noise.

  When printing is instructed, the data of the “picture A” frame 582 is output to the printer 195 via the print interface unit 194. Therefore, the printed matter is also powerful with a stop motion effect reflecting “picture A”.

  When the user cancels the pause, the frame automatically returns to the “picture B” frame group 571 and the reproduction is resumed from the frame 574. At this time, the reproduced video is of high quality with reduced jerkiness.

  As described above, according to the present embodiment, when a plurality of images based on signals having different accumulation periods are simultaneously captured using one image sensor, the plurality of images are read without sacrificing the frame rate. be able to.

[Second Embodiment]
An imaging apparatus according to the second embodiment of the present invention will be described with reference to FIGS. Constituent elements similar to those of the imaging apparatus according to the first embodiment shown in FIGS. 1 to 27 are denoted by the same reference numerals, and description thereof is omitted or simplified.

  In the first embodiment, two photodiodes 310A and 310B having different light receiving efficiencies (sensitivities) are selectively used according to the accumulation time, thereby enabling video shooting suitable for various shooting scenes. In the present embodiment, an example will be described in which the same effect as that of the first embodiment is realized by controlling the accumulation time of one photodiode.

  The imaging apparatus according to the present embodiment is the same as the imaging apparatus according to the first embodiment except that the circuit configuration of the pixel 303 of the imaging element 184 is different.

  FIG. 28 is a circuit diagram illustrating a circuit configuration of the pixel 303 of the image sensor 184 of the image pickup apparatus according to the present embodiment. FIG. 28 shows a first column, first row of pixels 303 and a first column, mth row of pixels 303 among a plurality of pixels 303 constituting the pixel array 302. As shown in FIG. 28, each pixel 303 includes a photodiode 600, transfer transistors 601A, 601B, 602A, 602B, and 603, a reset transistor 604, an amplification transistor 605, and a selection transistor 606.

  The anode of the photodiode 600 is connected to the ground line. The cathode of the photodiode 600 is connected to the source of the transfer transistor 601A, the source of the transfer transistor 601B, and the source of the transfer transistor 603, respectively. The drain of the transfer transistor 601A is connected to the source of the transfer transistor 602A. A connection node between the drain of the transfer transistor 601A and the source of the transfer transistor 602A constitutes a signal holding unit 607A. The drain of the transfer transistor 601B is connected to the source of the transfer transistor 602B. A connection node between the drain of the transfer transistor 601B and the source of the transfer transistor 602B constitutes a signal holding unit 607B.

  The drain of the transfer transistor 602A and the drain of the transfer transistor 602B are connected to the source of the reset transistor 604 and the gate of the amplification transistor 605. A connection node of the drain of the transfer transistor 602A, the drain of the transfer transistor 602B, the source of the reset transistor 604, and the gate of the amplification transistor 605 forms a floating diffusion region 608. The source of the amplification transistor 605 is connected to the drain of the selection transistor 606. The drain of the reset transistor 604 and the drain of the amplification transistor 605 are connected to the power supply line 620. The drain of the transfer transistor 603 is connected to the power supply line 621. The source of the selection transistor 606 is connected to the signal output line 623.

  As described above, the pixel 303 of the image sensor 184 of the image pickup apparatus according to the present embodiment has two signal holding units 607A and 607B for one photodiode 600. Note that the basic structure of a CMOS image sensor having a signal holding unit is disclosed in, for example, Patent Document 2 by the same applicant, and detailed description thereof is omitted here.

  The plurality of pixels 303 of the pixel array 302 are connected to control lines arranged in the row direction from the vertical scanning circuit 307 in units of rows. The control lines for each row include a plurality of control lines connected to the gates of transfer transistors 601A, 602A, 601B, 602B, 603, reset transistor 604, and select transistor 606, respectively. The transfer transistor 601A is controlled by a transfer pulse φTX1A, and the transfer transistor 602A is controlled by a transfer pulse φTX2A. The transfer transistor 601B is controlled by a transfer pulse φTX1B, and the transfer transistor 602B is controlled by a transfer pulse φTX2B. The reset transistor 604 is controlled by a reset pulse φRES, and the selection transistor 606 is controlled by a selection pulse φSEL. The transfer transistor 603 is controlled by a transfer pulse φTX3. Each control pulse is sent from a vertical scanning circuit 307 (not shown) based on a control signal from the system control CPU 178 of the imaging apparatus. Each transistor is turned on when the control pulse is at a high level and turned off when the control pulse is at a low level.

  The image sensor 184 constituting the image pickup apparatus of the present embodiment has two signal holding units 607A and 607B for one photodiode 600. Thereby, it is possible to simultaneously photograph a still image that is the first video signal and a moving image that is the second video signal. Therefore, it is possible to read out two video signals having different accumulation periods without lowering the S / N ratio.

  FIG. 29 is a diagram for explaining the accumulation and readout timing of the image sensor 184 when simultaneously capturing a still image that is the first video signal and a moving image that is the second video signal in the imaging apparatus of the present embodiment. It is. The accumulation here is an operation of transferring the charges generated in the photodiode 600 to the signal holding units 607A and 607B and accumulating them. Reading is an operation of outputting a signal based on the charges held in the signal holding units 607A and 607B to the outside of the image sensor 184 through the floating diffusion region 608.

  In FIG. 29, the horizontal axis represents time, the vertical synchronization signal 650, the horizontal synchronization signal 651, the still image accumulation period 661, the still image transfer period 662, the still image readout period 665, the moving image accumulation period 663, the moving image transfer period 664, A readout period 666 is shown. Here, the still image accumulation period 661 indicates an accumulation period of signal charges for still images in the photodiode 600. The still image transfer period 662 indicates a period during which signal charges for a still image are transferred from the photodiode 600 to the signal holding unit 607. The still image readout period 665 is a still image readout period. The moving image accumulation period 663 indicates the accumulation period of signal charges for moving images in the photodiode 600. The moving image transfer period 664 indicates a period in which signal charges for moving images are transferred from the photodiode 600 to the signal holding unit 607. The moving image reading period 666 is a moving image reading period.

  In this driving example, a still image and a moving image are read out during one cycle of the vertical synchronization signal 650. FIG. 29 illustrates the timing of 16 rows for convenience, but the actual image sensor 184 has several thousand rows. In FIG. 29, the last row is the m-th row.

  The still image that is the first video signal is a signal charge generated during one accumulation period (still image accumulation period 661) that is performed simultaneously for all rows during one cycle (time Tf) of the vertical synchronization signal 650. Is generated based on In addition, the moving image as the second video signal is obtained by adding the signal charges generated during the accumulation period (moving image accumulation period 663) divided into Np times (Np is an integer of 2 or more (Np> 1)). Generated based on the signal charge. In this embodiment, the number Np of accumulation periods performed during one shooting cycle of the moving image that is the second video signal is, for example, 16 times, and these accumulation periods are performed at equal time intervals. The interval Tf (time Tf) of the vertical synchronization signal 650 corresponds to the frame rate of the moving image, and is, for example, 1/60 seconds in this embodiment.

  By doing in this way, it is possible to photograph a moving image and a still image at the same time. In addition, it is possible to acquire a blur-free image with a short accumulation time intended by the photographer as a still image, while acquiring a smooth image with reduced jerkiness as a moving image.

  In the first shooting cycle (Tf) shown in FIG. 29, the still image accumulation time (still image accumulation period 661) is set to a time corresponding to the shutter speed T1 set by the photographer. In this driving example, the shutter speed T1 is 1/2000 seconds. The still image accumulation period is the same for all rows, and is set to end immediately before the start of reading of the first row of still images (still image reading period 665). The end time of the still image accumulation period is the time after the elapse of time Ta from the vertical synchronization signal 650. The time Ta is set to half or less of the interval Tf of the vertical synchronization signal 650. Since the end time of the still image accumulation period (still image accumulation period 661) is the same for all rows, the start time of the still image accumulation period for the vertical synchronization signal 650 is set in accordance with the still image shutter speed T1. It has become. The readout of the still image that is the first video signal (still image readout period 665) is executed during the accumulation period of the moving image that is the second video signal (moving picture accumulation period 663).

  On the other hand, the moving image accumulation period (moving image accumulation period 663) is performed a plurality of times at equal time intervals in one cycle. In this driving example, the time interval is set so that the accumulation period divided into 16 times ends immediately before the start of reading of each row (moving image reading period 666). The time interval of the moving image accumulation period is set to an integral multiple of the interval Th of the horizontal synchronization signal 651. Thereby, the accumulation timing of each row of the moving image is the same. In FIG. 29, the time interval of the moving image accumulation period is shown to be twice the interval Th of the horizontal synchronization signal 651 for convenience. Normally, the time interval of the moving image accumulation period is the interval of the horizontal synchronization signal 651 to an integer not exceeding m / Np, where m is the number of rows of the image sensor 184 and Np is the number of moving image accumulation periods in one cycle. Set to a value multiplied by Th.

  In addition, the accumulation time of one moving image is set to T1 / Np (= 1/32000 seconds). The start time of the accumulation period of each row of moving images is fixed with respect to the vertical synchronization signal 650. The end time of one accumulation period of moving images is set for the vertical synchronization signal 650 in accordance with the still image shutter speed T1 set by the photographer.

  On the other hand, an example in which the photographer sets the shutter speed T2 of the still image to be longer (for example, T2 = 1/500 seconds) when the subject brightness is low is the second that starts from the time t1 in FIG. It is a photographing cycle (Tf).

  As described in the first imaging cycle, the end time of the still image accumulation period is simultaneous for all rows (time after the elapse of time Ta from the vertical synchronization signal 650), and reading of the first row of still images (still image) It is set to end immediately before the start of the image reading period 665). Since the end time of the still image accumulation period is the same for all rows, the start time of the still image accumulation period for the vertical synchronization signal 650 is set in accordance with the still image shutter speed T1. The readout of the still image that is the first video signal (still image readout period 665) is executed during the accumulation period of the moving image that is the second video signal (moving picture accumulation period 663).

  As in the case of the first shooting cycle, the moving image accumulation period is performed a plurality of times at equal time intervals during one cycle. In this driving example, the time interval is set so that the accumulation period divided into 16 times ends immediately before the start of reading of each row (moving image reading period 666). The time interval of the moving image accumulation period is set to an integral multiple of the interval Th of the horizontal synchronization signal 651. Thereby, the accumulation timing of each row of the moving image is the same. In addition, the accumulation time of one moving image is set to T2 / Np (= 1/8000 seconds). The start time of the accumulation period of each row of moving images is fixed with respect to the vertical synchronization signal 650. The end time of one accumulation period of moving images is set for the vertical synchronization signal 650 according to the still image shutter speed T2 set by the photographer.

  Next, an example of a method for controlling the image sensor 184 in the second imaging cycle starting from time t1 in FIG. 29 will be described using the timing chart in FIG. In FIG. 30, the time t1 when the vertical synchronizing signal φV rises is the same as the time t1 when the vertical synchronizing signal 650 rises in FIG.

  Here, it is assumed that the image sensor 184 has m pixel columns in the vertical direction. FIG. 30 shows the timing of the first row and the last m-th row among these. In FIG. 30, a signal φV is a vertical synchronizing signal, and a signal φH is a horizontal synchronizing signal.

  First, at time t1, the vertical synchronization signal φV and the horizontal synchronization signal φH supplied from the timing generation unit 189 transition from the low level to the high level.

  Next, at time t2 synchronized with the vertical synchronization signal φV becoming high level, the reset pulse φRES (1) of the first row supplied from the vertical scanning circuit 307 transits from high level to low level. As a result, the reset transistor 604 of the pixels 303 in the first row is turned off, and the reset state of the floating diffusion region 608 is released. At the same time, the selection pulse φSEL (1) of the first row supplied from the vertical scanning circuit 307 transits from the low level to the high level. Accordingly, the selection transistor 606 of the pixel 303 in the first row is turned on, and the video signal can be read from the pixel 303 in the first row.

  Next, at time t3, the transfer pulse φTX2B (1) of the first row supplied from the vertical scanning circuit 307 transits from the low level to the high level. As a result, the transfer transistor 602B of the pixels 303 in the first row is turned on, and the video signal charges accumulated in the signal holding unit 607B during the immediately preceding shooting cycle (the shooting cycle that ends at time t1) are input to the floating diffusion region 608. Transferred. Then, a signal corresponding to a change in potential of the floating diffusion region 608 is read out to the signal output line 623 via the amplification transistor 605 and the selection transistor 606. The signal read out to the signal output line 623 is supplied to a readout circuit (not shown) and output to the outside as an image signal in the first row of the moving image (corresponding to the moving image reading period 666 in FIG. 29).

  Next, at time t4, the transfer pulse φTX2B (1) of the first row and the transfer pulse φTX2A (φTX2A (1), φTX2A (m)) supplied from the vertical scanning circuit 307 change from the low level to the high level. And transition. As a result, the transfer transistors 602B of the pixels 303 in the first row and the transfer transistors 602A of the pixels 303 in all rows are turned on. At this time, the reset pulses φRES (φRES (1), φRES (m)) of all the rows have already transitioned to the high level, and the reset transistor 604 is in the on state. Thereby, the floating diffusion region 608 of the pixels 303 in all rows, the signal holding unit 607A of the pixels 303 in all rows, and the signal holding unit 607B of the pixels 303 in the first row are reset. At this time, the selection pulse φSEL (1) in the first row has transitioned to the low level, and the pixels 303 in the first row have returned to the non-selected state.

  Next, at time t5, the transfer pulses φTX3 (φTX3 (1), φTX3 (m)) of all rows supplied from the vertical scanning circuit 307 change from the high level to the low level. As a result, the transfer transistors 603 of the pixels 303 in all rows are turned off, and the reset of the photodiodes 600 of the pixels 303 in all rows is released. Then, accumulation of moving image signal charges is started in the photodiodes 600 of the pixels 303 in all rows (corresponding to the moving image accumulation period 663 in FIG. 29).

  Here, the time interval Tb between the time t1 when the vertical synchronization signal φV becomes high level and the time t5 at which the accumulation of moving image signal charges in the photodiodes 600 of the pixels 303 in all rows starts is fixed.

  Note that the start of the accumulation period of the first row of moving images at time t5 in FIG. 30 represents the start of the accumulation period of moving images in the imaging cycle starting from time t1 in FIG. The start of the accumulation period of the m-th row of moving images at time t5 represents the start of the accumulation period of moving images in the shooting cycle that ends at time t1 in FIG.

  In FIG. 29, the still image and moving image accumulation time are different between the shooting cycle that ends at time t1 and the shooting cycle that starts from time t1. Since the accumulation time in the imaging cycle that ends at time t1 is shorter than the accumulation time in the imaging cycle that starts from time t1, the accumulation period of the m-th row of the moving image in the imaging cycle that ends at time t1 ends first.

  Next, immediately before time t6, the transfer pulse φTX1B (m) of the m-th row transitions from the low level to the high level. Accordingly, the transfer transistor 601B of the pixel 303 in the m-th row is turned on, and the signal charge accumulated in the photodiode 600 of the pixel 303 in the m-th row is transferred to the signal holding unit 607B (moving image transfer period 664 in FIG. 29). Equivalent).

  Next, at time t6, the transfer pulse φTX1B (m) of the m-th row transitions from the high level to the low level. Accordingly, the transfer transistor 601B of the pixel 303 in the m-th row is turned off, and the transfer of the signal charge accumulated in the photodiode 600 to the signal holding unit 607B is completed.

  The time from the time t5 to the time t6 corresponds to the accumulation time (= T1 / 16) in each of the Np accumulation periods for the moving image in the imaging cycle that ends at the time t1 in FIG.

  Similarly, at time t6, the transfer pulse φTX3 (m) in the m-th row transitions from the low level to the high level. As a result, the transfer transistor 603 of the m-th row pixel 303 is turned on, and the photodiode 600 of the m-th row pixel 303 is reset.

  Next, immediately before time t7, the transfer pulse φTX1B (1) of the first row supplied from the vertical scanning circuit 307 transits from the low level to the high level. As a result, the transfer transistor 601B of the pixel 303 in the first row is turned on, and the signal charge accumulated in the photodiode 600 of the pixel 303 in the first row is transferred to the signal holding unit 607B.

  Next, at time t7, the transfer pulse φTX1B (1) in the first row changes from the high level to the low level. Accordingly, the transfer transistor 601B of the pixel 303 in the first row is turned off, and the transfer of the signal charge accumulated in the photodiode 600 of the pixel 303 in the first row to the signal holding unit 607B is completed.

  The time from time t5 to time t7 corresponds to the accumulation time (= T2 / 16) in each of the Np accumulation periods for the moving image in the shooting cycle starting from time t1 in FIG.

  Similarly, at time t7, the transfer pulse φTX3 (1) in the first row changes from the low level to the high level. As a result, the transfer transistor 603 of the pixel 303 in the first row is turned on, and the photodiode 600 of the pixel 303 in the first row is reset.

  At the time t8 when a time twice as long as the interval Th of the horizontal synchronization signal φH has elapsed from the time t5 of the start of the first accumulation period of the moving image in the shooting cycle starting from the time t1, the second accumulation period of the moving image is reached. Be started.

  The operation of the second accumulation period of the moving image that starts from time t8 and ends at time t10 is the same as the operation of the first accumulation period of the moving image that starts from time t5 and ends at time t7. .

  Here, in the operations of the first and second accumulation periods of moving images, the signal charges of the moving images generated during these two accumulation periods are added and held in the signal holding unit 607B.

  Next, between the time t10 and the time t11, the third to fifth accumulation periods of the moving image are performed in the same manner as the period from the time t5 to the time t7 described above.

  Next, from time t11, the sixth accumulation period of the moving image is started. Here, the start time t11 of the sixth accumulation period of the moving image is set to a time after time T (= 6 × 2 × Th + Tb) has elapsed from time t1 at which the vertical synchronization signal φV becomes high level. Yes. Here, Th is a time interval of the horizontal synchronizing signal φH, and Tb is a time between time t1 when the vertical synchronizing signal φV becomes high level and time t5 when the first accumulation period of the moving image is started in the photodiode 600. It is an interval.

  The operation of the sixth accumulation period of the moving image starting from time t11 and ending at time t13 is the same as the operation of the first accumulation period of moving image starting from time t5 and ending at time t7, and the description thereof is omitted.

  Next, from time t <b> 14, a still image accumulation period that is the first video signal is started. In this driving example, the number of still image accumulation periods in one shooting cycle is one. The start time of the still image readout period (corresponding to the still image readout period 665 in FIG. 29) with respect to the vertical synchronization signal φV is fixed. Therefore, the end time of the still image accumulation period with respect to the vertical synchronization signal φV is fixed to a time after the elapse of time Ta from the start time, and the still image accumulation time is set to end at time t19. When the shutter speed T2 of the still image is set by the photographer, the start time of the still image accumulation period is controlled in the imaging apparatus of the present embodiment.

  At time t14 that is back by time T2 from time t19 when the still image accumulation period ends, the transfer pulse φTX3 (φTX3 (1), φTX3 (m)) of all rows changes from the high level to the low level. As a result, the transfer transistors 603 of the pixels 303 in all rows are turned off, and the reset of the photodiodes 600 of the pixels 303 in all rows is released. Then, in the photodiodes 600 of the pixels 303 in all rows, a still image signal charge accumulation period is started (corresponding to the still image accumulation period 661 in FIG. 29).

  Further, during the still image signal charge accumulation period, the readout period of the m-th row of the moving image of the immediately preceding shooting cycle that ends at time t1 ends.

  First, at time t15, the reset pulse φRES (m) of the m-th row supplied from the vertical scanning circuit 307 changes from the high level to the low level. Thereby, the reset transistor 604 of the pixel 303 in the m-th row is turned off, and the reset state of the floating diffusion region 608 is released. At the same time, the selection pulse φSEL (m) of the m-th row supplied from the vertical scanning circuit 307 changes from the low level to the high level. Accordingly, the selection transistor 606 of the pixel 303 in the m-th row is turned on, and the video signal can be read from the pixel 303 in the m-th row.

  Next, at time t <b> 16, the transfer pulse φTX <b> 2 </ b> B (m) of the m-th row transitions from the low level to the high level. As a result, the transfer transistor 602B of the pixel 303 in the m-th row is turned on, and the moving image signal charges accumulated in the signal holding unit 607B during the immediately preceding photographing period that ends at time t1 are transferred to the floating diffusion region 608. The Then, a signal corresponding to a change in potential of the floating diffusion region 608 is read out to the signal output line 623 via the amplification transistor 605 and the selection transistor 606. The signal read to the signal output line 623 is supplied to a read circuit (not shown) and output to the outside as an image signal of the m-th row of the moving image (corresponding to the moving image reading period 666 in FIG. 29).

  As a result, the reading of the moving image, which is the second video signal of the immediately preceding shooting cycle that ends at time t1, is completed, and then the still image that is the first video signal of the shooting cycle starting from time t1. Reading is performed (corresponding to the still image reading period 665 in FIG. 29).

  Next, at time t <b> 17, the transfer pulse φTX <b> 2 </ b> B (m) in the m-th row transitions from the low level to the high level. As a result, the transfer transistor 602B of the pixel 303 in the m-th row is turned on. At this time, the reset pulse φRES (m) of the m-th row has already transitioned to the high level, and the reset transistor 604 is in the on state. As a result, the floating diffusion region 608 of the pixel 303 in the m-th row and the signal holding unit 607B of the pixel 303 in the m-th row are reset. At this time, the selection pulse φSEL (m) in the m-th row also changes to the low level, and the pixels 303 in the m-th row have returned to the non-selected state.

  Next, at time t18, the reset pulse φRES (1) in the first row transits from the high level to the low level. As a result, the reset transistor 604 of the pixels 303 in the first row is turned off, and the reset state of the floating diffusion region 608 is released. At the same time, the selection pulse φSEL (1) in the first row transits from the low level to the high level. Accordingly, the selection transistor 606 of the pixel 303 in the first row is turned on, and the video signal can be read from the pixel 303 in the first row.

  Next, immediately before time t19, the transfer pulses φTX1A (φTX1A (1), φTX1A (m)) of all rows supplied from the vertical scanning circuit 307 change from the low level to the high level. Thereby, the transfer transistors 601A of the pixels 303 in all rows are turned on, and the signal charges accumulated in the photodiodes 600 of the pixels 303 in all rows are transferred to the signal holding unit 607A (in the still image transfer period 662 in FIG. 29). Equivalent).

  Next, at time t19, the transfer pulse φTX1A (φTX1A (1), φTX1A (m)) of all rows changes from the high level to the low level. Thereby, the transfer transistors 601A of the pixels 303 in all rows are turned off, and the transfer of the signal charges accumulated in the photodiodes 600 of the pixels 303 in all rows to the signal holding unit 607A is completed.

  The time from time t14 to time t19 corresponds to the still image accumulation time T2 in the shooting cycle starting from time t1.

  Next, at time t20, the transfer pulse φTX2A (1) in the first row changes from the low level to the high level. As a result, the transfer transistor 602A of the pixel 303 in the first row is turned on, and the signal charge accumulated in the signal holding unit 607A of the pixel 303 in the first row is transferred to the floating diffusion region 608. Then, a signal corresponding to a change in potential of the floating diffusion region 608 is read out to the signal output line 623 via the amplification transistor 605 and the selection transistor 606 of the pixel 303 in the first row. The signal read out to the signal output line 623 is supplied to a readout circuit (not shown) and output to the outside as an image signal of the first row of a still image (corresponding to the still image readout period 665 in FIG. 29). As described above, the image signal of the first row of the still image that is the first video signal is stored in the accumulation period (between time t5 and time t26) of the image signal of the first row of the moving image that is the second video signal. Read out.

  Next, the seventh accumulation period of the moving image is started from time t21. Here, the start time t21 of the seventh accumulation period of the moving image is set to a time after time T (= (7 + 2) × 2 × Th + Tb) has elapsed from time t1 when the vertical synchronization signal φV becomes high level. Has been. In this driving example, the two moving image accumulation periods overlap with the still image accumulation period (corresponding to the still image accumulation period 661 in FIG. 29). For this reason, the start time t21 of the seventh accumulation period of the moving image is equivalent to the start time of the ninth accumulation period of the moving image of the shooting cycle starting at time t1.

  Since the operation of the seventh accumulation period of the moving image starting from time t21 and ending at time t23 is the same as the operation of the first accumulation period of moving image starting from time t5 and ending at time t7, the description is omitted. .

  Next, during the period from time t23 to time t24, the eighth to thirteenth accumulation period of the moving image is performed in the same manner as the period from time t5 to time t7 described above.

  Next, from time t24, the last fourteenth accumulation period of the moving image in the shooting cycle starting from time t1 is started. Here, the start time t24 of the fourteenth accumulation period of the moving image is set to the time after the time T (= (14 + 2) × 2 × Th + Tb) has elapsed from the time t1 when the vertical synchronization signal φV becomes high level. Has been.

  The operation of the fourteenth accumulation period of the moving image starting from time t24 and ending at time t26 is the same as the operation of the first accumulation period of moving image starting from time t5 and ending at time t7, and the description thereof will be omitted.

  Next, at time t27, the reset pulse φRES (m) of the m-th row transitions from the high level to the low level. Thereby, the reset transistor 604 of the pixel 303 in the m-th row is turned off, and the reset state of the floating diffusion region 608 is released. At the same time, the selection pulse φSEL (m) in the m-th row transitions from the low level to the high level. Accordingly, the selection transistor 606 of the pixel 303 in the m-th row is turned on, and the video signal can be read from the pixel 303 in the m-th row.

  Next, at time t <b> 28, the transfer pulse φTX <b> 2 </ b> A (m) in the m-th row transitions from the low level to the high level. Accordingly, the transfer transistor 602A of the m-th row pixel 303 is turned on, and the signal charge of the still image accumulated in the signal holding unit 607A of the m-th row pixel 303 is transferred to the floating diffusion region 608. Then, a signal corresponding to a change in the potential of the floating diffusion region 608 is read out to the signal output line 623 via the amplification transistor 605 and the selection transistor 606 of the pixel 303 in the m-th row. The signal read out to the signal output line 623 is supplied to a readout circuit (not shown) and outputted to the outside as an image signal of the m-th row of the still image (corresponding to the still image readout period 665 in FIG. 29). Thus, the m-th row image signal of the still image that is the first video signal is read out during the accumulation period of the m-th row image signal of the moving image that is the second video signal.

  Next, at time t29, the vertical synchronization signal φV supplied from the timing generation unit 189 transitions from the low level to the high level, and the next imaging cycle is started.

  As described above, in this driving example, the end time of the still image accumulation period is fixed with respect to the vertical synchronization signal, and the start time of the moving image accumulation period that is performed a plurality of times during one shooting period is the vertical synchronization signal. It is fixed against it. Thereby, a moving image and a still image can be read out within the same shooting period.

  As a result, when the shutter speed T1 of the moving image is slower than the predetermined shutter speed Tth, a still image having a short accumulation time and a blur-free image and a moving image having a long accumulation period and reduced jerkiness during one shooting period. You can shoot at the same time.

  As described above, according to the present embodiment, when a plurality of images based on signals having different accumulation periods are simultaneously captured using one image sensor, the plurality of images are read without sacrificing the frame rate. be able to.

[Modified Embodiment]
The present invention is not limited to the above embodiment, and various modifications can be made.
For example, the configuration of the imaging apparatus described in the above embodiment is an example, and the imaging apparatus to which the present invention can be applied is not limited to the configuration illustrated in FIGS. 1 and 2. Also, the circuit configuration of each part of the image sensor is not limited to the configuration shown in FIGS. 3, 8, 11, 28, and the like.

  For example, in the pixel circuit shown in FIG. 8, the reset transistor 314, the amplification transistor 315, and the selection transistor 317 are shared by the two pixel elements 303A and 303B, but these transistors may be provided in the pixel elements 303A and 303B, respectively. In the pixel circuit shown in FIG. 8, the number of pixel elements constituting one pixel is not limited to two, and may be three or more.

  In the sample and hold circuit shown in FIG. 8, the capacitor for holding the pixel signal from the pixel element 303A and the capacitor for holding the pixel signal from the pixel element 303B are separately provided. However, these capacitors are shared. May be.

  Further, in the second embodiment, the accumulation period of the still image is set to one time and the accumulation period of the moving image is set to sixteen times. However, the number of the accumulation periods is appropriately selected according to the shooting conditions and the like. It is not limited to these. For example, the number of times that a still image is stored may be at least once and may be two or more. Also, the number of times of accumulation of moving images may be at least two or more.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100 ... Imaging device 152 ... Shooting optical system 178 ... System control CPU
184: Image sensor 187, 188 ... Digital signal processing unit 303 ... Pixel 303A, 303B ... Pixel element 310A, 310B ... Photodiode (photoelectric conversion unit)

Claims (13)

A plurality of pixels each including a first pixel element including a first photoelectric conversion unit and a second pixel element including a second photoelectric conversion unit having a lower sensitivity than the first photoelectric conversion unit;
A first video signal based on a signal charge generated during a first accumulation period in the first pixel element of the plurality of pixels from the plurality of pixels, and the second pixel of the plurality of pixels Reading out a video signal based on a signal charge generated during a second accumulation period longer than the first accumulation period in the element and reading a second video signal synchronized with the first video signal Circuit,
And a control unit that controls the readout circuit such that a first readout period for reading out the first video signal is performed during the second accumulation period. The control unit determines the start timing of the first accumulation period when the length of the first accumulation period is longer than the length of the second read period for reading the second video signal. The imaging device according to claim 1, wherein the imaging device is controlled after the start timing of the second accumulation period. The imaging apparatus according to claim 2, wherein the start timing of the first accumulation period and the start timing of the second accumulation period are the same. The control unit determines the end timing of the first accumulation period when the length of the first accumulation period is equal to or less than the length of the second read period for reading the second video signal. The imaging device according to claim 1, wherein the imaging device is controlled after the end timing of the second readout period. The imaging device according to claim 4, wherein the end timing of the first accumulation period and the end timing of the second readout period are the same. The imaging apparatus according to claim 2, wherein the end timing of the first accumulation period is near the middle of the second accumulation period. The control unit sets the readout circuit so as not to read out the first video signal when the length of the first accumulation period is longer than half the length of the second accumulation period. It controls. The imaging device of any one of Claim 1 thru | or 6 characterized by the above-mentioned. The imaging apparatus according to any one of claims 1 to 7, wherein the first accumulation period is set according to a luminance of a subject. The imaging apparatus according to any one of claims 1 to 8, wherein the second accumulation period is fixed. A plurality of pixels each including a photoelectric conversion unit and first and second signal holding units;
A first video signal generated by transferring a signal charge generated in the photoelectric conversion unit from the plurality of pixels to the first signal holding unit at least once during one imaging cycle; A video signal generated by transferring and adding the signal charge generated in the photoelectric conversion unit to the second signal holding unit at least twice or more during the imaging cycle, and synchronized with the first video signal A read circuit for reading out the second video signal,
An image pickup apparatus comprising: a control unit that controls the readout circuit so that a readout period for reading out the first video signal is performed during an accumulation period of the second video signal. The imaging apparatus according to claim 10, wherein the end timing of the accumulation period of the first video signal is near the middle of the accumulation period of the second video signal. A plurality of pixels each including a first pixel element including a first photoelectric conversion unit and a second pixel element including a second photoelectric conversion unit having a lower sensitivity than the first photoelectric conversion unit; A driving method of an imaging apparatus including an imaging element,
Reading a first video signal based on signal charges generated during a first accumulation period in the first pixel elements of the plurality of pixels;
A video signal based on a signal charge generated during a second accumulation period longer than the first accumulation period in the second pixel element of the plurality of pixels, the video signal being synchronized with the first video signal And reading the second video signal that has been performed,
The method for driving an imaging apparatus, wherein the step of reading out the first video signal is performed during the second accumulation period. A driving method of an imaging apparatus including an imaging element having a plurality of pixels each including a photoelectric conversion unit and a first and second signal holding unit,
Reading out the first video signal generated by transferring the signal charges generated in the photoelectric conversion unit during one imaging period to the first signal holding unit from the plurality of pixels at least once;
A video signal generated by transferring and adding signal charges generated in the photoelectric conversion unit from the plurality of pixels to the second signal holding unit at least twice or more during the one imaging cycle. Reading a second video signal synchronized with the first video signal,
The step of reading out the first video signal is performed during the accumulation period of the second video signal.