JP2017085245A - Imaging apparatus and image distortion detection method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus that is able to detect an image distortion when simultaneously taking two videos by means of a single imager.SOLUTION: An imaging apparatus comprises: a pixel array in which pixels having a first photoelectric conversion part and a second photoelectric conversion part lower in light reception efficiency than the first photoelectric conversion part are arrayed in a matrix; a first reading section provided in each line of the pixel array and configured to read a first video signal from the first photoelectric conversion section; a second reading section that reads a second video signal from the second photoelectric conversion section; and image distortion detection means that detects an image distortion by comparing the first video signal and the second video signal, when a time difference between the start time t59 of a charge accumulation period of each line of the pixel array in the first photoelectric conversion part and the start time t60 of the accumulation period of the other lines differs from the time difference between the start time t61 of the charge accumulation period of each line of the pixel array of the second photoelectric conversion section and the start time t63 of the accumulation period of the other lines.SELECTED DRAWING: Figure 23

Description

  The present invention relates to an imaging apparatus having two photoelectric conversion units having different light receiving efficiencies, and more particularly to an image distortion detection method.

  By simultaneously shooting a moving image and a still image with one camera, it is possible to view the shooting scene as a moving image and simultaneously enjoy a decisive scene in the moving image as a still image. In addition, by shooting a normal frame rate video and a high frame rate video simultaneously with a single camera, a specific scene of the normal frame rate video can be switched to a slow motion video of a high frame rate video to create a high-quality work. I can enjoy it. In this way, by using an imaging device having two photoelectric conversion units with different light receiving efficiencies, it is possible to obtain a video that can convey a dynamic feeling to the viewer, greatly increasing the value of the captured video. Can do.

  As an imaging apparatus having two photoelectric conversion units having different light receiving efficiencies, for example, Patent Document 1 discloses an imaging apparatus including a pair of photodiodes having asymmetric pupil shapes in each pixel. The imaging device described in Patent Document 1 is designed so that one of the pair of photodiodes has a high light receiving efficiency, and the other photodiode has a low light receiving efficiency. Then, by using the two signals from the pair of photodiodes as separate video data, two videos can be simultaneously shot.

JP 2014-048459 A

  In order to control the exposure period of the image sensor that captures two images at the same time, a mechanical shutter system such as a focal plane shutter is not suitable, so an electronic shutter system that can electrically control the readout timing is required. However, a rolling electronic shutter, which is a typical electronic shutter, has a problem that when a subject moves or pans during shooting, a so-called rolling shutter distortion occurs that causes the subject image to be distorted obliquely, resulting in a reduction in image quality. .

  The imaging apparatus according to the present invention includes a first photoelectric conversion unit, a second photoelectric conversion unit having a light receiving efficiency lower than that of the first photoelectric conversion unit, and a first charge that holds charges converted and accumulated by the first photoelectric conversion unit. A pixel array having pixels having a holding unit and a second charge holding unit that holds charges converted and accumulated by the second photoelectric conversion unit arranged in a matrix, and pixels provided in each column of the pixel array. A first readout unit that reads out a first video signal based on the charge held by the first charge holding unit, and a second video signal that is provided in each column of the pixel array and is based on the charge held by the second charge holding unit of the pixel The time difference between the start time of the charge accumulation period in each row of the pixel array and the start time of the accumulation period in other rows is different between the first photoelectric conversion unit and the second photoelectric conversion unit. In, the first video signal and the second video signal Characterized in that it comprises an image distortion detecting means for detecting an image distortion by comparing the.

  According to the present invention, it is possible to obtain an imaging device capable of detecting image distortion when simultaneously capturing two images using a single imaging device.

1 is an external view showing an imaging apparatus according to a first embodiment of the present invention. 1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. 1 is a block diagram illustrating a schematic configuration of an imaging element in an imaging apparatus according to a first embodiment of the present invention. It is sectional drawing which shows the internal structure of the image pick-up element in the imaging device by 1st Embodiment of this invention. It is a graph which shows the relationship between the angle of the light ray which injects into a pixel, and the output from a photodiode. It is a figure which shows the relationship between the imaging optical system and imaging device in the imaging device by 1st Embodiment of this invention. It is the schematic for demonstrating the video signal output from an image pick-up element. It is a circuit diagram showing an example of composition of a pixel in an imaging device by a 1st embodiment of the present invention. FIG. 2 is a plan layout diagram illustrating a main part of a pixel in the imaging apparatus according to the first embodiment of the present invention. FIG. 3 is a plan layout diagram illustrating a main part including a light guide of a pixel in the imaging apparatus according to the first embodiment of the present invention. FIG. 3 is a circuit diagram illustrating a configuration of a readout circuit in the imaging apparatus according to the first embodiment of the present invention. 3 is a timing chart showing a driving sequence of the image sensor in the imaging apparatus according to the first embodiment of the present invention. It is a timing chart for demonstrating the imaging sequence of the imaging device by 1st Embodiment of this invention. It is a figure which shows an example of the time code added to each flame | frame of a moving image and still image shooting data. It is a figure which shows an example of the file structure of the imaging | photography data of a moving image and a still image. It is a figure for demonstrating the setting screen of the imaging condition of a moving image and a still image. It is a figure which shows the example of the ISO sensitivity range of the video signal image | photographed using two photodiodes from which light reception efficiency differs. It is a program AE diagram in the dual video mode of the imaging device according to the first embodiment of the present invention. It is a sequence diagram which shows the accumulation | storage and read-out timing at the time of imaging | photography with a rolling electronic shutter system. It is a figure showing the mode of the display part in the live view display after powering on an imaging device. It is a figure for demonstrating the utilization example of pictureA and pictureB stored in the storage. It is a figure for demonstrating rolling shutter distortion. FIG. 5 is a timing diagram for explaining a method for detecting rolling shutter distortion in the imaging apparatus according to the first embodiment of the present invention. It is a figure which shows the example of the image image | photographed with the detection method of the rolling shutter distortion in the imaging device by 1st Embodiment of this invention. FIG. 10 is a timing diagram for explaining a method of correcting a rolling shutter distortion of picture A in the imaging apparatus according to the second embodiment of the present invention. It is a figure which shows the example of the image which correct | amended rolling shutter distortion in the imaging device by 2nd Embodiment of this invention. It is a timing diagram for demonstrating the method to correct | amend the picture B rolling shutter distortion in the imaging device by 2nd Embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, this invention is not limited to the following embodiment, In the range which does not deviate from the summary, it can change suitably. In the drawings described below, components having the same function are denoted by the same reference numerals, and the description thereof may be omitted or simplified.

(First embodiment)
In general, when the shutter speed at the time of moving image shooting is high, so-called jerkiness such as frame advance appears at the time of reproduction, and the smoothness of the image is lost. In order to obtain such a smooth image with reduced jerkiness, it is necessary to set an accumulation time close to one frame period in a series of photographing. That is, if the frame rate is 30 fps, a relatively long accumulation time such as 1/30 seconds or 1/60 seconds is appropriate. This setting is particularly important in situations where the camera posture is unstable, such as aerial photography.

  On the other hand, in still images, it is required to shoot a video with a so-called stop motion effect that suppresses blur and stops a moment. For this reason, it is necessary to set a short accumulation time of, for example, about 1/1000 second. In addition, since a frame period is short for a high frame rate moving image, for example, if the frame rate is 120 fps, a short accumulation time such as 1/125 seconds or 1/250 seconds is inevitably set.

  Shooting two images of a moving image and a still image or a normal frame rate moving image and a high frame rate moving image simultaneously through a single shooting lens means that the aperture used for the shooting is the same. . At this time, while the two images are shot with different storage time settings, the image sensor obtains the same level of signal charge, and both of them have a good S / N ratio and no noise. Is desirable.

  In addition, as a technique for making movies and home television images more realistic, there is an HDR (high dynamic range) technique for moving images. This expands the luminance reproduction range of the display screen, and provides a more realistic sensation than before, mainly by momentary or partial luminance increase. In order to complete this technology at a high level as a whole from video input to output, it is absolutely necessary to expand the dynamic range on the device side that acquires the video.

  Against this background, a technique has been proposed in which two pixel groups having different sensitivities are provided in the image sensor in the imaging apparatus, and the dynamic range is expanded by combining the outputs from these pixel groups. Similarly, in this technique, it is desirable that intermediate video data with a good S / N ratio and no noise is created from either of the two pixel groups, and finally high quality HDR video can be synthesized.

  However, if one image pickup device having two photoelectric conversion units having different light receiving efficiencies is used to simultaneously shoot two images, it is possible to optimize exposure of the two images at the same time because the aperture is common. difficult. In particular, when different shutter speeds are set for two images, if one of the two images is properly exposed, the other exposure will be over or under.

  Therefore, in the present embodiment, first, a method for optimizing the exposure of two images when simultaneously capturing two images with different shooting conditions using a single image sensor will be described. In the present embodiment, an imaging apparatus in which a photographing optical system for imaging is added to a video processing apparatus for processing a video signal output from the imaging device is an example of a preferred embodiment of the present invention. explain. However, the video processing apparatus is not necessarily configured as a part of the imaging apparatus, and may be configured by hardware different from the imaging element and the imaging optical system. Moreover, you may make it mount all or one part of the function of a video processing apparatus in an image pick-up element.

  FIG. 1 is an external view of a digital still motion camera as an example of the imaging apparatus according to the present embodiment. FIG. 1A shows a front view thereof, and FIG. 1B shows a rear view thereof.

  The imaging apparatus 100 according to the present embodiment includes a housing 151, a photographing optical system 152 provided on the front portion of the housing 151, and a switch ST <b> 154 and a propeller 162 provided on the upper surface of the housing 151. Yes. In addition, the imaging apparatus 100 includes a display unit 153, a switch MV155, a shooting mode selection lever 156, a menu button 157, up / down switches 158 and 159, a dial 160, and a playback button on the rear surface of the casing 151. 161.

  The casing 151 is a container that houses various functional components that constitute the imaging apparatus 100 such as an imaging element. The photographing optical system 152 is an optical system for forming an optical image of a subject. The display unit 153 includes a display device for displaying shooting information and video. The display unit 153 may be provided with a movable mechanism for changing the orientation of the screen as necessary. The display unit 153 has a display luminance range that can display an image with a wide dynamic range without suppressing the luminance range. The switch ST154 is a shutter button used mainly for taking a still image. The switch MV155 is a button for starting and stopping moving image shooting. The shooting mode selection lever 156 is a changeover switch for selecting a shooting mode. The menu button 157 is a button for shifting to a function setting mode for setting a function of the imaging apparatus 100. Up / down switches 158 and 159 are buttons used when changing various setting values. The dial 160 is a dial for changing various setting values. The playback button 161 is a button for shifting to a playback mode in which video recorded on a recording medium stored in the imaging apparatus 100 is played on the display unit 153. The propeller 162 is for floating the imaging device 100 in the air in order to perform shooting from the air.

  FIG. 2 is a block diagram illustrating a schematic configuration of the imaging apparatus 100 according to the present embodiment. As illustrated in FIG. 2, the imaging apparatus 100 includes an aperture 181, an aperture control unit 182, an optical filter 183, an image sensor 184, analog front ends 185 and 186, digital signal processing units 187 and 188, and a timing generation unit 189. ing. Further, the imaging apparatus 100 includes a system control CPU 178, a switch input unit 179, a video memory 190, and a flight control apparatus 200. Further, the imaging apparatus 100 includes a display interface unit 191, a recording interface unit 192, a recording medium 193, a print interface unit 194, an external interface unit 196, and a wireless interface unit 198.

  The image sensor 184 is for converting an optical image of a subject formed through the photographing optical system 152 into an electrical video signal. The image sensor 184 is not particularly limited, but has, for example, a sufficient number of pixels, a signal readout speed, a color gamut, and a dynamic range to satisfy the UHDTV (Ultra High Definition Television) standard. The diaphragm 181 is for adjusting the amount of light passing through the photographing optical system 152. The aperture control unit 182 is for controlling the aperture 181. The optical filter 183 is for limiting the wavelength of light incident on the image sensor 184 and the spatial frequency transmitted to the image sensor 184. The photographing optical system 152, the diaphragm 181, the optical filter 183, and the image sensor 184 are disposed on the optical axis 180 of the photographing optical system 152.

  The analog front ends 185 and 186 are for performing analog signal processing and analog-digital conversion of the video signal output from the image sensor 184. The analog front ends 185 and 186 include, for example, a correlated double sampling (CDS) circuit that removes noise, an amplifier that adjusts a signal gain, an A / D converter that converts an analog signal into a digital signal, and the like. The digital signal processing units 187 and 188 are for performing various corrections on the digital video data output from the analog front ends 185 and 186 and then compressing the video data. The timing generator 189 is for outputting various timing signals to the image sensor 184, the analog front ends 185 and 186, and the digital signal processors 187 and 188. The system control CPU 178 is a control unit that performs various calculations and controls the entire imaging apparatus 100. The video memory 190 is for temporarily storing video data.

  The display interface unit 191 is an interface between the system control CPU 178 and the display unit 153 for displaying captured images on the display unit 153. The recording medium 193 is a recording medium such as a semiconductor memory for recording video data, additional data, and the like, and may be provided in the imaging apparatus 100 or detachable. The recording interface unit 192 is an interface between the system control CPU 178 and the recording medium 193 for recording on or reading from the recording medium 193. The external interface unit 196 is an interface between the system control CPU 178 and an external device for communicating with an external device such as the external computer 197. The print interface unit 194 is an interface between the system control CPU 178 and the printer 195 for outputting a photographed image to a printer 195 such as a small inkjet printer for printing. The wireless interface unit 198 is an interface between the system control CPU 178 and the network 199 for communicating with a network 199 such as the Internet. The switch input unit 179 includes a switch ST154, a switch MV155, and a plurality of switches for switching various modes. The flight control device 200 is a control device for controlling the propeller 162 to fly the imaging device 100 in order to perform shooting from the air.

  FIG. 3 is a block diagram illustrating a configuration example of the image sensor 184. As shown in FIG. 3, the image sensor 184 includes a pixel array 302, a vertical scanning circuit 307, readout circuits 308A and 308B, and timing control circuits 309A and 309B.

  In the pixel array 302, a plurality of pixels 303 are arranged in a matrix. Note that the actual number of pixels 303 belonging to the pixel array 302 is generally large, but here, in order to simplify the drawing, 16 pixels 303 arranged in a matrix of 4 rows × 4 columns. Only shows. Each of the plurality of pixels 303 includes a set of a pixel element 303A and a pixel element 303B. In FIG. 3, the upper half area of the pixel 303 is a pixel element 303A, and the lower half area of the pixel 303 is a pixel element 303B. The pixel element 303A and the pixel element 303B each generate a signal by photoelectric conversion.

  Each column of the pixel array 302 is provided with signal output lines 304A and 304B extending in the column direction. The signal output line 304A of each column is connected to the pixel element 303A belonging to the column. A signal from the pixel element 303A is output to the signal output line 304A. The signal output line 304B of each column is connected to the pixel element 303B belonging to the column. A signal from the pixel element 303B is output to the signal output line 304B. Each column of the pixel array 302 is provided with a power supply line 305 and a ground line 306 extending in the column direction. The power supply line 305 and the ground line 306 in each column are connected to the pixels 303 belonging to the column. The power supply line 305 and the ground line 306 may be signal lines extending in the row direction.

  The vertical scanning circuit 307 is disposed adjacent to the pixel array 302 in the row direction. The vertical scanning circuit 307 controls a readout circuit in the pixel 303 via a control line (not shown) extending in the row direction with respect to the plurality of pixels 303 of the pixel array 302 in the row direction. A predetermined control signal is output. In the figure, reset pulses φRESn and transfer pulses φTXnA, TXnB are shown as control signals (n is an integer corresponding to a row number).

  The readout circuits 308A and 308B are arranged adjacent to the pixel array 302 in the column direction so as to sandwich the pixel array 302. The readout circuit 308A is connected to the signal output line 304A of each column. The readout circuit 308A sequentially activates the signal output lines 304A in each column, thereby sequentially reading out the signals from the signal output lines 304A in each column, and performs predetermined signal processing. Similarly, the readout circuit 308B is connected to the signal output line 304B of each column. The readout circuit 308B sequentially activates the signal output lines 304B in each column sequentially, thereby sequentially reading out the signals from the signal output lines 304B in each column and performing predetermined signal processing. Each of the readout circuits 308A and 308B can include a noise removal circuit, an amplification circuit, an analog-digital conversion circuit, a horizontal scanning circuit, and the like, and sequentially outputs signals that have undergone predetermined signal processing.

  The timing control circuit 309A is connected to the vertical scanning circuit 307 and the readout circuit 308A. The timing control circuit 309A outputs a control signal for controlling the driving timing of the vertical scanning circuit 307 and the readout circuit 308A. The timing control circuit 309B is connected to the vertical scanning circuit 307 and the readout circuit 308B. The timing control circuit 309B outputs a control signal for controlling the driving timing of the vertical scanning circuit 307 and the readout circuit 308B.

  FIG. 4 is a cross-sectional view showing the internal structure of the pixel 303 of the image sensor 184. Each pixel 303 includes two photodiodes 310A and 310B, a light guide 255, and a color filter 256, as shown in FIG. The photodiode 310A constitutes a part of the pixel element 303A, and the photodiode 310B constitutes a part of the pixel element 303B. The photodiodes 310A and 310B are provided in the silicon substrate 251. The light guide 255 is provided in an insulating layer 254 provided on the silicon substrate 251. The insulating layer 254 is made of, for example, silicon oxide, and the light guide 255 is made of a material having a higher refractive index than the insulating layer 254, for example, silicon nitride. A wiring layer 252 is provided on the insulating layer 254 between the light guides 255. On the light guide 255, a color filter 256 having a predetermined spectral transmittance characteristic is provided. FIG. 4 shows an example in which the color filters of two adjacent pixels 303 are configured by color filters 256 and 257 having different spectral transmittance characteristics.

  The light guide 255 has a property of confining light inside due to a difference in refractive index with the insulating layer 254. Thereby, the light incident through the color filter 256 can be guided to the photodiodes 310 </ b> A and 310 </ b> B by the light guide 255. The photodiodes 310A and 310B are arranged asymmetrically with respect to the light guide 255, and the light beam propagated through the light guide 255 is incident on the photodiode 310A with high efficiency and is incident on the photodiode 310B with low efficiency. Further, the light guide 255 is adjusted so that the incident angle characteristics of the light guide 255 are not biased with respect to the incident light beam that can be effectively photoelectrically converted by the photodiodes 310A and 310B.

  FIG. 5 is a graph showing the relationship between the angle of light rays incident on the pixel and the output from the photodiode. In FIG. 5, the horizontal axis represents the angle of light rays incident on the pixel, and the vertical axis represents the output from the photodiode. FIG. 5 shows an output characteristic 261 from the photodiode 310A and an output characteristic 262 from the photodiode 310B.

  As shown in FIG. 5, both the output characteristic 261 and the output characteristic 262 have a slightly symmetrical shape with a peak when the incident angle of the light beam is zero. The peak intensity PB of the output characteristic 262 is about 1/8 of the peak intensity PA of the output characteristic 261. This means that the incident angle dependence of the photodiodes 310A and 310B is small, and the light receiving efficiency of the photodiode 310B is 1/8 of that of the photodiode 310A. That is, if the photodiode 310B is replaced with the ISO sensitivity setting value, the sensitivity is lower by three stages than the photodiode 310A.

  Next, the relationship between the photographing optical system 152 and the image sensor 184 will be described in more detail with reference to FIG. FIG. 6 is a diagram for explaining the relationship between the photographing optical system 152 and the image sensor 184. FIG. 6A is a view of the photographing optical system 152 as seen from the direction of the optical axis 180. FIG. 6B is a diagram showing in more detail the part from the photographing optical system 152 to the image sensor 184 in FIG.

  As shown in FIG. 6B, the imaging element 184 includes a pixel 276 located in the center of the imaging area and a pixel 277 located near the outer edge of the imaging area. In this case, the pixel 276 can receive a light flux from a region surrounded by the light rays 272 and 273. Further, the pixel 277 can receive a light beam from a region surrounded by the light beam 274 and the light beam 275. At this time, since the field lens 270 is disposed between the optical filter 183 and the photographing optical system 152, the light flux received by the pixel 276 and the light flux received by the pixel 277 in the vicinity of the photographing optical system 152 are illustrated in FIG. 6 (a) is overlapped as indicated by a region 271. As a result, the light beam from the photographic optical system 152 can be received with high efficiency in any pixel.

  FIG. 7 is a schematic diagram for explaining a video signal output from the image sensor. Here, it is assumed that a color filter having a predetermined light transmittance characteristic is arranged in the pixel array 302 in the color filter array 281 shown in FIG. FIG. 7A schematically shows a pixel array 302 in which pixels 303 are arranged in a matrix of 6 rows × 8 columns, and colors of color filters arranged in each pixel. In the figure, R represents a red color filter, G1 and G2 represent green color filters, and B represents a blue color filter. The color filter array 281 shown in the figure is a color filter array called a so-called Bayer array, and color filters for each color are arranged for each row by repeating G1BG1B... RG2RG2... G1BG1B.

  From the pixel array 302 having such a color filter array 281, output data 282 and 283 shown in FIGS. 7B and 7C are obtained. In FIG. 7B, g1A and g2A represent outputs from the pixel element 303A of the pixel 303 in which the green color filter is arranged. bA represents the output from the pixel element 303A of the pixel 303 in which the blue color filter is arranged. rA represents the output from the pixel element 303A of the pixel 303 in which the red color filter is arranged. In FIG. 7C, g1B and g2B represent outputs from the pixel element 303B of the pixel 303 in which the green color filter is arranged. bB represents the output from the pixel element 303B of the pixel 303 in which the blue color filter is arranged. rB represents the output from the pixel element 303B of the pixel 303 in which the red color filter is arranged.

  As described with reference to FIG. 3, the image sensor 184 obtains two outputs from the readout circuits 308A and 308B, one of which is the output data 282 shown in FIG. This is the output data 283 shown in FIG. The output data 282 becomes a video signal pictureA after predetermined signal processing. Further, the output data 283 becomes a video signal picture B after predetermined signal processing. In the following description, a video signal based on the output data 282 is referred to as “pictureA”, and a video signal based on the output data 283 is referred to as “pictureB”. Strictly speaking, picture A and picture B are video signals after processing such as predetermined correction, but for convenience of explanation, video signals before or during correction may also be described as picture A and picture B. .

  FIG. 8 is a circuit diagram illustrating a configuration example of the pixel 303. As described above, the pixel 303 includes the pixel element 303A and the pixel element 303B. The pixel element 303A includes a photodiode 310A, a transfer transistor 311A, a floating diffusion region 313A, a reset transistor 314A, and an amplification transistor 315A. The pixel element 303B includes a photodiode 310B, a transfer transistor 311B, a floating diffusion region 313B, a reset transistor 314B, and an amplification transistor 315B. Note that the photodiode 310A corresponds to the photodiode 310A illustrated in FIG. 4, and the photodiode 310B corresponds to the photodiode 310B illustrated in FIG.

  The anode of the photodiode 310A is connected to the ground line 306, and the cathode of the photodiode 310A is connected to the source of the transfer transistor 311A. The drain of the transfer transistor 311A is connected to the source of the reset transistor 314A and the gate of the amplification transistor 315A. A connection node of the drain of the transfer transistor 311A, the source of the reset transistor 314A, and the gate of the amplification transistor 315A constitutes the floating diffusion region 313A. The drain of the reset transistor 314A and the drain of the amplification transistor 315A are connected to the power supply line 305. The source of the amplification transistor 315A constituting the pixel signal output unit 316A is connected to the signal output line 304A.

  Similarly, the anode of the photodiode 310B is connected to the ground line 306, and the cathode of the photodiode 310B is connected to the source of the transfer transistor 311B. The drain of the transfer transistor 311B is connected to the source of the reset transistor 314B and the gate of the amplification transistor 315B. A connection node of the drain of the transfer transistor 311B, the source of the reset transistor 314B, and the gate of the amplification transistor 315B constitutes a floating diffusion region 313B. The drain of the reset transistor 314B and the drain of the amplification transistor 315B are connected to the power supply line 305. The source of the amplification transistor 315B constituting the pixel signal output unit 316B is connected to the signal output line 304B.

  The pixels 303 in each column are connected from the vertical scanning circuit 307 to a reset control line 319 and transfer control lines 320A and 320B arranged in the row direction. The reset control line 319 is connected to the gate of the reset transistor 314A and the gate of the reset transistor 314B. The transfer control line 320A is connected to the gate of the transfer transistor 311A via the contact portion 312A. The transfer control line 320B is connected to the gate of the transfer transistor 311B via the contact portion 312B. The reset control line 319 supplies the reset pulse φRESn output from the vertical scanning circuit 307 to the gate of the reset transistor 314A and the gate of the reset transistor 314B. The transfer control line 320A supplies a transfer pulse φTXnA output from the vertical scanning circuit 307 to the gate of the transfer transistor 311A. The transfer control line 320B supplies the transfer pulse φTXnB output from the vertical scanning circuit 307 to the gate of the transfer transistor 311B. Note that n added to the signs of the reset pulse φRESn, the transfer pulse φTXnA, and the transfer pulse φTXnB is an integer corresponding to the row number. In the drawing, n is represented by a code in which n is replaced with an integer corresponding to the row number.

  The photodiode 310A is a first photoelectric conversion unit that generates and accumulates charges by photoelectric conversion, and the photodiode 310B is a second photoelectric conversion unit that generates and accumulates charges by photoelectric conversion. The floating diffusion regions 313A and 313B are regions that retain charges. The transfer transistor 311A is for transferring the charge generated by the photodiode 310A to the floating diffusion region 313A. The transfer transistor 311B is for transferring the charge generated by the photodiode 310B to the floating diffusion region 313B.

  When the high-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned on, and the photodiode 310A and the floating diffusion region 313A are connected. Similarly, when the high-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned on, and the photodiode 310B and the floating diffusion region 313B are connected. When the high level reset pulse φRESn is output from the vertical scanning circuit 307, the reset transistors 314A and 314B are turned on, and the photodiodes 310A and 310B and the floating diffusion regions 313A and 313B are reset.

  When the low-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned off, and the photodiode 310A starts accumulating signal charges generated by photoelectric conversion. Next, when the high-level transfer pulse φTXnA is output from the vertical scanning circuit 307, the transfer transistor 311A is turned on, and the signal charge of the photodiode 310A is transferred to the floating diffusion region 313A. Then, the amplification transistor 315A amplifies the input based on the voltage value of the floating diffusion region 313A corresponding to the amount of signal charge transferred from the photodiode 310A, and outputs the amplified signal to the signal output line 304A.

  Similarly, when the low-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned off, and the photodiode 310B starts accumulating signal charges generated by photoelectric conversion. Next, when a high-level transfer pulse φTXnB is output from the vertical scanning circuit 307, the transfer transistor 311B is turned on, and the signal charge of the photodiode 310B is transferred to the floating diffusion region 313B. Then, the amplification transistor 315B amplifies the voltage of the floating diffusion region 313B corresponding to the amount of signal charge transferred from the photodiode 310B and outputs the amplified voltage to the signal output line 304B.

  9 and 10 are plan layout diagrams showing the main part of the pixel 303. FIG. 9 shows photodiodes 310A and 310B, transfer transistors 311A and 311B, and floating diffusion regions 313A and 313B among the components of the pixel 303. Other circuit elements including the reset transistors 314A and 314B and the amplification transistors 315A and 315B are represented as a readout circuit 321 in the drawing, and detailed illustration is omitted. Further, the signal output lines 304A and 304B and the power supply line 305 arranged in the vertical direction of the pixel 303 are omitted, and the contact portions of the reset control line 319, the power supply line 305, and the ground line 306 are omitted. FIG. 10 shows the light guide 255 described in FIG. 4 in addition to the components shown in FIG. In the light guide 255, a hatched portion indicates a low refractive index region, and a white portion indicates a high refractive index region, that is, a light guide portion.

  9 and 10, the contact portion 312A is a contact portion that connects the transfer control line 320A and the gate of the transfer transistor 311A. The contact portion 312B is a contact portion that connects the transfer control line 320B and the gate of the transfer transistor 311B. The photodiodes 310 </ b> A and 310 </ b> B are photoelectric conversion units that perform photoelectric conversion. The photodiodes 310 </ b> A and 310 </ b> B are a first conductivity type (for example, P type) semiconductor region and a second conductivity type (for example, a PN junction with the first conductivity type semiconductor region). N-type) semiconductor region (N-type electron storage region). The second conductivity type semiconductor region of the photodiode 310A and the second conductivity type semiconductor region of the photodiode 310B are separated by a separation unit 322.

  The transfer transistors 311A and 311B, the contact portions 312A and 312B, and the transfer control lines 320A and 320B are arranged in line symmetry or substantially line symmetry with respect to the separation portion 322 between the photodiodes 310A and 310B, respectively. On the other hand, the light guide 255 is disposed at a position biased with respect to the separation portion 322 as shown in FIG. That is, the photodiode 310A occupies a large area of the bottom portion of the light guide 255, whereas the photodiode 310B is slightly hooked on the bottom portion of the light guide 255. As a result, the light receiving efficiency of the photodiode 310A is high, and the light receiving efficiency of the photodiode 310B is low.

  In the image sensor 184 according to the present embodiment, the ratio of the light receiving efficiencies of the photodiodes 310A and 310B is set to about 8: 1, that is, the sensitivity difference is set to about three stages. Then, while shooting two images with different storage time settings, the pixel element obtains the same level of signal charge, and both have a good S / N ratio and no noise, or This makes it possible to synthesize high HDR video. Details will be described later.

  FIG. 11 is a circuit diagram illustrating a configuration example of a readout circuit of the image sensor 184. In FIG. 11, “A” is added to the end of the reference numerals of some components assuming the reading circuit 308A. In the readout circuit 308B, it should be understood that “B” is appended to the end of the reference numerals of the corresponding components.

  As shown in FIG. 11, the read circuit 308A includes a clamp capacitor C0, a feedback capacitor Cf, an operational amplifier 406, a reference voltage source 407, and a switch 423. One input terminal of the operational amplifier 406 is connected to the signal output line 304A via a clamp capacitor C0. A feedback capacitor Cf and a switch 423 are connected in parallel between the one input terminal and the output terminal of the operational amplifier 406. The other input terminal of the operational amplifier is connected to the reference voltage source 407. The reference voltage source 407 is for supplying the reference voltage Vref to the operational amplifier 406. The switch 423 is a switch controlled by the signal PC0R, and is turned on when the signal PC0R is at a high level, and short-circuits both ends of the feedback capacitor Cf.

  The readout circuit 308A also includes switches 414, 415, 418, and 419, a capacitor CTSA, a capacitor CTNA, horizontal output lines 424 and 425, and an output amplifier 421. The switches 414 and 415 are switches that control writing of pixel signals to the capacitors CTSA and CTNA. The switch 414 is a switch controlled by the signal PTSA, and is turned on when the signal PTSA is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTSA. The switch 415 is a switch controlled by the signal PTNA and is turned on when the signal PTNA is at a high level, and connects the output terminal of the operational amplifier 406 and the capacitor CTNA.

  The switches 418 and 419 are switches for controlling the output of the pixel signals held in the capacitors CTSA and CTNA to the output amplifier 421. The switches 418 and 419 are turned on in response to a control signal from the horizontal shift register. As a result, the signal written in the capacitor CTSA is output to the output amplifier 421 via the switch 418 and the horizontal output line 424. Further, the signal written in the capacitor CTNA is output to the output amplifier 421 via the switch 419 and the horizontal output line 425. The signal PC0R, the signal PTNA, and the signal PTSA are signals that are supplied from the timing generation unit 189 under the control of the system control CPU 178.

  The readout circuit 308B has a configuration similar to that of the readout circuit 308A. Further, the signal PTNB and the signal PTSB in the following description are signals supplied from the timing generation unit 189 under the control of the system control CPU 178, and play the same role as the signal PTNA and the signal PTSA in the read circuit 308A. ing.

  Next, reset, accumulation, and readout operations in the image sensor 184 will be sequentially described using the readout operation from the pixels 303 in the first row as an example with reference to the timing chart of FIG.

  First, at time t1, the vertical scanning circuit 307 transitions the transfer pulses φTX1A and TX1B output to the transfer control lines 320A and 320B from the low level to the high level. As a result, the transfer transistors 311A and 311B are turned on. At this time, a high level reset pulse φRES1 is output from the vertical scanning circuit 307 to the reset control line 319, and the reset transistors 314A and 314B are also in the ON state. As a result, the photodiodes 310A and 310B are connected to the power supply line 305 via the transfer transistors 311A and 311B and the reset transistors 314A and 314B, and are in a reset state. At this time, the floating diffusion regions 313A and 313B are also reset.

  Next, at time t2, the vertical scanning circuit 307 changes the transfer pulse φTX1B from the high level to the low level. As a result, the transfer transistor 311B is turned off, and accumulation of signal charges by photoelectric conversion is started in the photodiode 310B.

  Next, at time t3, the vertical scanning circuit 307 changes the transfer pulse φTX1A from the high level to the low level. As a result, the transfer transistor 311A is turned off, and the photodiode 310A starts to accumulate signal charges by photoelectric conversion.

  Next, at time t4, the vertical scanning circuit 307 causes the reset pulse φRES1 to transition from the high level to the low level. Thereby, the reset transistors 314A and 314B are turned off, and the reset of the floating diffusion regions 313A and 313B is released.

  As a result, the potential of the floating diffusion region 313A is read to the signal output line 304A as the reset signal level via the amplification transistor 315A and input to the read circuit 308A. Further, the potential of the floating diffusion region 313B is read out as a reset signal level pixel signal to the signal output line 304B via the amplification transistor 315B and input to the readout circuit 308B.

  At time t4, the high-level signal PC0R is output from the timing generation unit 189 to the read circuit 308A and the read circuit 308B, and the switch 423 is in the on state. Therefore, the pixel signal at the reset signal level is input from the pixel element 303A to the readout circuit 308A in a state where the operational amplifier 406 buffers the output of the reference voltage Vref. Although not shown in the drawing, the reset signal level pixel signal is similarly input to the readout circuit 308B from the pixel element 303B.

  Next, at time t5, the signal PC0R output from the timing generation unit 189 to the read circuit 308A and the read circuit 308B is changed from the high level to the low level, and the switch 423 is turned off.

  Next, at time t6, the signal PTNA output from the timing generation unit 189 to the reading circuit 308A is changed from the low level to the high level to turn on the switch 415, and the output of the operational amplifier 406 at that time is written into the capacitor CTNA. Similarly, the signal PTNB output from the timing generation unit 189 to the reading circuit 308B is changed from the low level to the high level to turn on the switch 415, and the output of the operational amplifier 406 at that time is written to the capacitor CTNB.

  Next, at time t7, the signal PTNA output from the timing generation unit 189 to the reading circuit 308A is changed from the high level to the low level, the switch 415 is turned off, and the writing to the capacitor CTNA is completed. Similarly, the signal PTNB output from the timing generation unit 189 to the reading circuit 308B is changed from the high level to the low level, the switch 415 is turned off, and the writing to the capacitor CTNB is completed.

  Next, at time t8, the vertical scanning circuit 307 changes the transfer pulses φTX1A and φTX1B from the low level to the high level, and turns on the transfer transistors 311A and 311B. As a result, the signal charge accumulated in the photodiode 310A is transferred to the floating diffusion region 313A, and the signal charge accumulated in the photodiode 310B is transferred to the floating diffusion region 313B.

  By simultaneously setting the transfer pulses φTX1A and φTX1B to the high level at time t8, the end timings of the accumulation times of the photodiodes 310A and 310B are aligned.

  Next, at time t9, the vertical scanning circuit 307 changes the transfer pulses φTX1A and φTX1B from the high level to the low level, and turns off the transfer transistors 311A and 311B. Thereby, the reading of the signal charge accumulated in the photodiode 310A to the floating diffusion region 313A and the reading of the signal charge accumulated in the photodiode 310B to the floating diffusion region 313B are completed.

  As a result, the potential of the floating diffusion region 313A changed by the signal charge is read as an optical signal level to the signal output line 304A via the amplification transistor 315A and input to the reading circuit 308A. Further, the potential of the floating diffusion region 313B changed by the signal charge is read as an optical signal level to the signal output line 304B via the amplification transistor 315B and is input to the reading circuit 308B.

  In the readout circuit 308 </ b> A, the operational amplifier 406 outputs a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio between the clamp capacitor C <b> 0 and the feedback capacitor Cf. Similarly, in the readout circuit 308B, a voltage obtained by applying an inversion gain to the voltage change at the capacitance ratio of the clamp capacitor C0 and the feedback capacitor Cf is output from the operational amplifier 406.

  Next, at time t10, the signal PTSA output from the timing generation unit 189 to the reading circuit 308A is changed from the low level to the high level to turn on the switch 414, and the output of the operational amplifier 406 at that time is written into the capacitor CTSA. Similarly, the signal PTSB output from the timing generator 189 to the reading circuit 308B is changed from the low level to the high level to turn on the switch 414, and the output of the operational amplifier 406 at that time is written to the capacitor CTSB.

  Next, at time t11, the signal PTSA output from the timing generation unit 189 to the reading circuit 308A is changed from the high level to the low level, the switch 414 is turned off, and writing to the capacitor CTSA is completed. Similarly, the signal PTSB output from the timing generation unit 189 to the reading circuit 308B is changed from the high level to the low level to turn off the switch 414, and the writing to the capacitor CTSB is completed.

  Next, at time t12, the vertical scanning circuit 307 changes the reset pulse φRES1 from the low level to the high level to turn on the reset transistors 314A and 314B. As a result, the floating diffusion regions 313A and 313B are connected to the power supply line 305 via the reset transistors 314A and 314B to be in a reset state.

  FIG. 13 is a timing chart for explaining an imaging sequence in the imaging apparatus according to the present embodiment. The “time code” at the top of the drawing indicates the time since the power is turned on, and “00: 00: 00: 00” indicates “hour: minute: second: frame”.

Time t31 is the power-on time of the imaging apparatus 100.
At time t32, the switch MV155, which is a moving image shooting button, is turned on when operated by the user, and in response to this, imaging of picture B and imaging of picture A are started. In response to the operation of the switch MV155, which is a button for moving image shooting, the picture data of picture B is written to the recording medium 193 through predetermined signal processing.

  In the period from time t33 to time t34 and in the period from time t35 to time t36, the switch ST154 used for taking a still image is operated. In response to this, during these periods, picture A is also written to the recording medium 193 through predetermined signal processing. The picture A video data may be written to the recording medium 193 not only during the period from time t33 to time t34 and from time t35 to time t36, but also during the same period as the picture B video data.

  For both picture A and picture B, each video data recorded on the recording medium 193 is a moving image of, for example, 60 fps at the same frame rate, and an NTSC time code is added. The value of the time code added to each frame of the moving image data is as shown in FIG. 14, for example.

  FIG. 15 is a diagram illustrating an example of a file structure of picture data of picture A and picture B. Here, an example of the MP4 file is shown as the format of the video data, but the format of the video data is not limited to this. The MP4 file format is standardized by ISO / IEC 144961 / AMD6. All information is stored in a structure called a box, and is composed of multiplexed video and audio bitstreams (media data) and management information (metadata) for these media data. Each Box is an identifier of 4 characters and represents each Box type. The file type Box 501 (ftyp) is a box at the head of the file for identifying the file. The media data box 502 (mdat) stores a multiplexed video and audio bit stream. The movie box 503 (moov) stores management information for reproducing the bit stream stored in the media data box 502. The skip box 504 (skip) is a box for skipping and skipping data stored in the skip box 504 during reproduction.

  In the skip box 504, the clip name 508 of the clip including the video data file and the UMID (Unique Material Identifier) 509 (CLIP-UMID) of the clip attached to the material are stored. In the skip box 504, the time code value (time code start value) 510 of the clip start frame and the serial number 511 of the recording medium on which the material file is recorded are stored. In this figure, the skip box 504 includes free space 505, user data 506, and metadata 507. Since special data such as the UMID of the material file and the serial number of the recording medium are stored in the skip box 504, they are not affected when played back by a general-purpose viewer.

  The same CLIP-UMID is set in each MP4 file of picture A and picture B. As a result, the CLIP-UMID is used to search for the same CLIP-UMID file from one material file, and mechanical association can be performed without manual confirmation.

  FIG. 16 is a diagram for explaining a screen for setting picture conditions for picture A and picture B. Suppose that the shooting mode selection lever 156 is rotated 90 degrees clockwise from the position of FIG. 1B, for example, to enter a dual video mode in which two videos can be shot simultaneously. On the display unit 153, the Bv value 521, the F number 522, the ISO sensitivities 523 and 524 of the picture A and the picture B, and the shutter speeds 525 and 526 corresponding to the luminance of the subject at that time are displayed. Also, currently set picture modes 527 and 528 are displayed for each of picture A and picture B. In the picture mode, the up / down switches 158 and 159 and the dial 160 can be used to select one that meets the purpose of shooting from a plurality of options.

  As described above, the difference in light receiving efficiency between the photodiode 310A and the photodiode 310B is set to three stages. For this reason, there is a three-stage difference in the ISO sensitivity range between picture A and picture B. As shown in FIG. 17, picture A is ISO 100 to ISO 102400, and picture B is ISO 12 to ISO 12800.

  FIG. 18 is a program AE (Automatic Exposure) diagram in the dual video mode. The horizontal axis shows the Tv value and the corresponding shutter speed, and the vertical axis shows the Av value and the corresponding aperture value. Further, the oblique direction is an equal Bv line. The relationship between the Bv value of picture A and the ISO sensitivity is represented in a gain notation area 556, and the relationship between the Bv value of picture B and ISO sensitivity is represented in a gain notation area 557. In FIG. 18, each Bv value is represented by a numerical value surrounded by a square in order to distinguish it from other parameters.

  How the shutter speed, aperture value, and ISO sensitivity change as the brightness changes from high to low will be described with reference to FIG.

  First, in the case of Bv13, ISO sensitivity is set to ISO100 in picture A. An equal Bv line of picture A intersects the program diagram 558 of picture A at a point 551, and a shutter speed 1/4000 and an aperture value F11 are determined from the point 551. On the other hand, in picture B, the ISO sensitivity is set to ISO12. The picture B equal Bv line intersects the picture B program diagram 559 at a point 552, and the shutter speed 1/500 and the aperture value F11 are determined from the point 552.

  In the case of Bv10, in picture A, the ISO sensitivity is increased by one step and set to ISO 200. The equal Bv line of picture A intersects the program diagram 558 of picture A at a point 553, and the shutter speed 1/1000 and the aperture value F11 are determined from the point 553. On the other hand, in picture B, the ISO sensitivity is set to ISO12. The picture B equal Bv line intersects the picture B program diagram 559 at a point 560, and the shutter speed 1/60 and the aperture value F11 are determined from the point 560.

  In the case of Bv6, the ISO sensitivity is set to ISO200 in picture A. The picture B equal Bv line intersects with the picture A program diagram 558 at a point 554, and the shutter speed 1/1000 and the aperture value F2.8 are determined from the point 554. On the other hand, in picture B, the ISO sensitivity is set to ISO12. The picture B equal Bv line intersects the picture B program diagram 559 at a point 555, and the shutter speed 1/60 and the aperture value F2.8 are determined from the point 555.

  In the case of Bv5, in picture A, the ISO sensitivity is increased by one step and set to ISO 400. The picture B equal Bv line intersects with the picture A program diagram 558 at a point 554, and the shutter speed 1/1000 and the aperture value F2.8 are determined from the point 554. On the other hand, in picture B, the ISO sensitivity is set to ISO25. The picture B equal Bv line intersects the picture B program diagram 559 at a point 555, and the shutter speed 1/60 and the aperture value F2.8 are determined from the point 555.

  Thereafter, as the brightness decreases, both the picture A and the picture B increase the gain without changing the shutter speed and the aperture value, and the ISO sensitivity increases.

  By performing the exposure operation shown in this program AE diagram, picture A maintains a shutter speed of 1/1000 or more in the entire luminance range described, and picture B maintains a shutter speed of 1/60 in many luminance ranges. . As a result, it is possible to obtain a high-quality moving picture without frame-like jerkiness with picture B while obtaining a stop motion effect with picture A.

  FIG. 19 is a sequence diagram showing storage and readout timings when shooting is performed using the rolling electronic shutter method. FIG. 19 shows the vertical synchronization signal 481 in the nth frame, and the accumulation period 482 of the upper end line of picture A and the accumulation period 483 of the lower end line of picture A are shown by thick solid lines. Similarly, the accumulation period 486 for the upper end line of picture B and the accumulation period 487 for the lower end line of picture B are indicated by thick solid lines. The horizontal axis represents time. Although FIG. 19 shows a timing sequence for an arbitrary column of the pixel array 302, pixels in the same row (same line) are controlled in synchronism with each other, so that the other columns have the same timing. It becomes a sequence.

  When the transfer transistor 311A shown in FIG. 8 is turned on and the accumulation period ends, the charges photoelectrically converted and accumulated by the photodiode 310A are transferred to and held in the first charge holding portion (floating diffusion region 313A). Similarly, the charges photoelectrically converted and accumulated by the photodiode 310B are transferred to and held in the second charge holding portion (floating diffusion region 313B). The accumulation period of each line is sequentially started from the upper end line to the lower end line at a predetermined time interval, and is sequentially ended at the time interval. Since the sensitivity of the photodiode 310B is lower than that of the photodiode 310A, the accumulation time of the picture B is set longer than that of the picture A.

  Further, in FIG. 19, a readout period 484 of the upper end line of picture A and a readout period 485 of the lower end line of picture A, which are readout periods of the video signal held in the first charge holding unit, are indicated by dotted lines. In addition, a readout period 488 for the upper end line of picture B and a readout period 489 for the lower end line of picture B, which are readout periods of the video signal held in the second charge holding unit, are similarly indicated by dotted lines. This read period includes transfer to the read circuits 308A and 308B via the amplification transistors 315A and 315B, read by the read circuits 308A and 308B, and transfer to the digital signal processing units 187 and 188. The readout period of each line is sequentially started from the upper end line to the lower end line at a predetermined time interval after the accumulation time of each line is finished, and is sequentially ended at the time interval.

  From time t53 to time t54 is the accumulation period 482 of the upper end line of picture A, and from time t56 to time t57 is the accumulation period 483 of the lower end line of picture A. Similarly, time t51 to time t54 are the accumulation period 486 of the upper end line of picture B, and time t52 to time t57 are the accumulation period 487 of the lower end line of picture B. Time t54 to time t55 are readout periods 484 and 488 for the upper end line of picture A and picture B, and time t57 to time t58 are readout periods 485 and 489 for the lower end line of picture A and picture B.

  In this way, by setting the picture A accumulation period by the photodiode 310A having a high light receiving efficiency short and setting the picture B accumulation period by the photodiode 310B having a low light receiving efficiency long, it is possible to obtain a video signal having the same intensity. Can do. As a result, it is possible to obtain an image with a good S / N ratio and no noise feeling without increasing the gain of both picture A and picture B.

  FIG. 20 is a diagram illustrating a state of the display unit 153 during live view display after power is turned on to the imaging apparatus. On the display unit 153, the sports scene of the person 163 captured through the photographing optical system 152 is displayed. In FIG. 20, the shooting mode selection lever 156 is in a dual video mode set at a position rotated 90 degrees clockwise from the state of FIG. Therefore, the display unit 153 displays a picture A accumulation period 164, a picture B accumulation period 165, and an F number 166.

  FIG. 21 is a diagram for explaining an example of using picture A and picture B stored in a storage in a tablet terminal, a personal computer, a television monitor, and the like. It is assumed that the data files of pictureA and pictureB are stored in a storage on the network. In FIG. 21, a frame group 581 is a frame group of picture A stored in an MP4 file, and a frame group 571 is a frame group of picture B stored in another MP4 file. These MP4 files are associated with each other by setting the same CLIP-UMID at the time of shooting.

  First, when reproduction of a moving image is started, frames are sequentially reproduced at a frame rate determined from the first frame 572 of the frame group 571 of picture B. Since picture B is shot with a setting that prevents the shutter speed from becoming too fast (in this example, 1/60 seconds), the reproduced video is high-quality without jerkiness like frame advance. In FIG. 21, the motion of the subject is expressed to such an extent that the periphery of the subject of the frame 573 is blurred and the jerkiness does not appear.

  When the user performs a pause operation at the point of time when playback has progressed to frame 573, a frame 582 having the same time code as that of picture B is automatically retrieved from the picture A data file and displayed. Picture A is a powerful image that is captured at a high shutter speed (in this example, 1/1000 second) at which a stop motion effect is easily obtained, and that captures a moment in a sports scene. The two images of picture A and picture B are taken with different accumulation time settings, but the gain of picture A is not increased, but the image sensor 184 obtains the same level of signal charge. For this reason, both picture A and picture B are images having a good S / N ratio and no noise.

  When printing is instructed, the data of the frame 582 of picture A is output to the printer 195 via the print interface unit 194. Therefore, the printed matter is also powerful with a stop motion effect reflecting picture A. When the user cancels the pause, the picture B automatically returns to the frame group 571 of the picture B, and the reproduction is resumed from the frame 574. At this time, the reproduced video is of high quality without jerkiness like frame advance.

  As described above, by using picture B as a video signal for a moving image, it is possible to obtain a moving image without jerkiness. Further, by using picture A as a video signal for a still image, a still image with a stop motion effect and a video for printing can be obtained. In the imaging apparatus of the present embodiment, these two effects can be realized using one imaging element.

  In this embodiment, in order to improve the operability at the time of reproduction, the still image file and the moving image file are separated and filed in separate folders on the recording medium 193. For example, as described above, both picture A and picture B are recorded separately in a moving image folder and a still image folder at the same frame rate and the same MP4 format. When the play button 161 is operated, picture B recorded as moving image file data is continuously reproduced as a moving image. Also, picture A recorded as file data for a still image is reproduced one by one as a still image together with the image feed operation.

  Regarding the display of still images during moving image reproduction, a still image picture A having the same time code as that of picture B of the moving image being reproduced may be displayed at the timing when the switch ST154 is operated. Alternatively, the up / down switches 158 and 159 may be operated to display still images before and after the switches. The combination of picture A and picture B used for moving image shooting and still image shooting is determined at the time of shooting. Then, each of photographed picture A and picture B is filed in a moving image folder and a still image folder.

  As described above, the present embodiment includes a pixel having the first photoelectric conversion unit (photodiode 310A) and the second photoelectric conversion unit (photodiode 310B) having a light receiving efficiency lower than that of the first photoelectric conversion unit. . The camera further includes shooting condition setting means (switch input means, system control CPU) for setting shooting conditions including an accumulation period of moving image and still image shooting. Thereby, a combination of photodiodes used for moving image shooting and still image shooting can be selected according to the shooting conditions. For example, when the subject is dark, degradation of the S / N ratio of the video signal can be reduced by using picture A by the photodiode 310A with high light receiving efficiency for both moving images and still images. Alternatively, when the subject is bright, degradation of the S / N ratio of the video signal can be reduced by using picture B by the photodiode 310B having low light receiving efficiency for both moving images and still images.

Next, a method for detecting rolling shutter distortion when two images are simultaneously captured using a single image sensor will be described.
First, rolling shutter distortion will be described. In the rolling electronic shutter method, video signals held in pixels are read in order from the upper end line to the lower end line of the pixel array 302. That is, the reading of the video signal of the next line is on standby until the reading of the video signal of the previous line is completed. As a result, the accumulation period starting after the end of reading of each line is also delayed, so the start timing of the accumulation period differs for each line. As a result, image quality degradation called rolling shutter distortion occurs.

  FIG. 22 is a diagram for explaining rolling shutter distortion. FIG. 22A shows subjects 701 and 702 to be photographed. On the other hand, FIG. 22B shows subjects 701 and 702 photographed by the rolling electronic shutter method. The subject 701 is stationary with respect to the shooting angle of view indicated by the solid frame, but the subject 702 is moved in the direction of the arrow with respect to the shooting angle of view. In the rolling electronic shutter method, as shown in FIG. 22B, a non-moving subject 701 is photographed without distortion, but a moving subject 702 is photographed with distortion. This is rolling shutter distortion.

  Note that FIG. 22 illustrates the case where the subject moves with respect to the shooting angle of view, but the shooting angle of view moves with respect to the subject as when the photographer pans the camera in the horizontal direction. Even in such a case, rolling shutter distortion occurs. Rolling shutter distortion also occurs in a mechanical shutter system such as a focal plane shutter. However, in the mechanical shutter system, since the moving time of the shutter curtain from the upper end to the lower end of one frame is shorter than that in the rolling electronic shutter system, the rolling electronic shutter distortion is suppressed and is difficult to be identified.

  The present embodiment has an image distortion detection mode for detecting rolling shutter distortion. In the image distortion detection mode, rolling shutter distortion is detected by comparing picture A and picture B. This image distortion detection mode is set by the photographer operating the menu button 157 and the up / down switches 158 and 159 shown in FIG. 1 before photographing.

  Next, the accumulation and readout timing in the image distortion detection mode will be described. FIG. 23 is a timing chart for explaining a method for detecting rolling shutter distortion in the imaging apparatus according to the first embodiment of the present invention. FIG. 23 is a diagram showing a timing sequence for storing and reading picture A and picture B, similarly to FIG. 19, and particularly shows a timing sequence in the image distortion detection mode.

  23 is the same as that in FIG. 19, and therefore, the same reference numerals as those in FIG. Hereinafter, the sequence of pictureA will be described. In the image distortion detection mode, the accumulation and readout timings of charges photoelectrically converted by the photodiodes differ between picture A and picture B. FIG. 23 shows the vertical synchronization signal 481 in the nth frame, and the accumulation period 491 of the upper end line of picture A and the accumulation period 492 of the next line are shown by thick solid lines. The charges photoelectrically converted and accumulated by the photodiode 310A in the accumulation periods 491 and 492 are transferred to and held in the first charge holding portion (floating diffusion region 313A) provided in the pixel 303 shown in FIG.

  FIG. 23 shows a readout period of the charge signal held in the first charge holding unit (floating diffusion region 313A), which is a readout period 493 of the upper end line of picture A and a readout period 494 of the next line. Shown with dotted lines. This readout period includes transfer from the pixel element 303A shown in FIG. 3 to the first readout unit (readout circuit 308A), readout by the first readout unit, and transfer to the digital signal processing unit.

  From time t59 to time t61 is the accumulation period 491 of the upper end line, and from time t60 to time t62 is the accumulation period 492 of the next line. Also, the time t61 to time t63 are the reading period 493 for the upper end line, and the time t63 to time t64 are the reading period 494 for the next line.

  In the image distortion detection mode shown in FIG. 23, as compared with the normal shooting mode shown in FIG. 19, from the start time t59 of the accumulation period 491 of the upper end line of picture A to the start time t60 of the accumulation period 492 of the next line. The time is getting shorter. On the other hand, the time from the start time t61 of the readout period 493 for the upper end line of picture A to the start time t63 of the readout period 494 for the next line does not change. This is because when the reading circuit 308 is shared by different lines in the same column, it is necessary to sequentially perform the reading process line by line, and the entire reading period of one frame cannot be shortened. Therefore, also in the image distortion detection mode, the reading process is not performed immediately after the end of the accumulation period of each line, but waits until the reading process of the previous line ends. In this manner, in the image distortion detection mode, the time difference between the start time of the charge accumulation period accumulated synchronously in each row of the pixel array and the start time of the accumulation period of other rows (hereinafter simply referred to as “accumulation period”). The time difference between the start times ”is different between picture A and picture B. As a result, the waiting period from the end of the accumulation period to the start of reading also differs.

  As shown in FIG. 23, the picture B is sequentially read by the second reading unit (reading circuit 308B) without a waiting period after transferring the charge to the second charge holding unit (floating diffusion region 313B) in each line. On the other hand, the picture A is sequentially read by the first reading unit (reading circuit 308A) after the transfer of the charge to the first charge holding unit (floating diffusion region 313A), with the standby time being longer than that of the previous line in each line. It is.

  FIG. 24 is a diagram illustrating an example of an image captured by the rolling shutter distortion detection method in the imaging apparatus according to the first embodiment of the present invention. The image by picture A shown in FIG. 24A and the image by picture B shown in FIG. 24B are obtained by photographing the subjects 701 and 702 shown in FIG. 22A according to the sequence shown in FIG. In the image by picture B shown in FIG. 24B, the rolling shutter distortion is generated in the moving subject 702, as in FIG. 22B. On the other hand, in the image by picture A shown in FIG. 24A, the difference between the start time of the accumulation period in each line and the other lines is shortened, so that the rolling shutter distortion of the subject 702 is reduced compared to FIG. ing. As described above, in the image distortion detection mode, the magnitude of the rolling shutter distortion is different between the image by the picture A and the image by the picture B.

  Therefore, in the present embodiment, an image distortion detection unit that detects rolling shutter distortion by comparing an image based on picture A and an image based on picture B is provided. The image distortion detection means is executed by the system control CPU 178 and compares, for example, an image based on picture A and an image based on picture B by image matching. Then, a vector unique to rolling shutter distortion that occurs when a moving subject 702 is photographed is detected. FIG. 24C shows an image obtained by superimposing the image of FIG. 24A and the image of FIG. In FIG. 24C, the outline 702a of the subject 702 in the image by picture A is indicated by a dotted line, and the outline 702b of the subject 702 in the image by picture B is indicated by a solid line. In FIG. 24D, the contours 702a and 702b are extracted and shown, and the motion vectors of several points corresponding to the contours 702a and 702b are indicated by arrows. As shown in FIG. 24D, the motion vector increases from the upper end direction to the lower end direction, which is a feature of rolling shutter distortion. The image distortion detection means determines that rolling shutter distortion has occurred when such a motion vector is detected and its magnitude exceeds a preset threshold value.

  As described above, the imaging apparatus according to the present embodiment includes the first photoelectric conversion unit (photodiode 310A) and the second photoelectric conversion unit (photodiode 310B) having lower light receiving efficiency than the first photoelectric conversion unit. . In addition, image distortion detection means (system control CPU) for detecting image distortion (rolling shutter distortion) by comparing the first video signal (picture A) and the second video signal (picture B) is provided. Here, when the image distortion is detected, the time difference between the start time of the charge accumulation period in each row of the pixel array and the start time of the accumulation period in another row is determined by the first photoelectric conversion unit. And the second photoelectric conversion unit are set differently. As a result, image distortion can be detected when two images are simultaneously captured using a single image sensor.

(Second Embodiment)
An imaging apparatus according to the second embodiment of the present invention will be described with reference to FIGS. In the present embodiment, a method for correcting rolling shutter distortion when rolling shutter distortion is detected will be described. In this embodiment, in addition to the image distortion detection unit described in the first embodiment, an image distortion correction unit that corrects rolling shutter distortion is provided. The image distortion correction unit is executed by the system control CPU 178 in the same manner as the image distortion detection unit.

  FIG. 25 is a timing chart for explaining a method of correcting the picture A rolling shutter distortion in the imaging apparatus according to the second embodiment of the present invention. FIG. 25 shows an accumulation and readout timing sequence for correcting rolling shutter distortion when it is determined by the image distortion correcting means that rolling shutter distortion has occurred. 25 is the same as that shown in FIG. 19, and therefore, the same reference numerals as those in FIG.

  FIG. 25 shows the vertical synchronization signal 481 in the nth frame, and the accumulation period 495 of the upper end line of picture A and the accumulation period 496 of the next line are shown by thick solid lines. In FIG. 25, the readout period 497 for the upper end line of picture A and the readout period 498 for the next line are indicated by dotted lines. Time t65 is the start time of the accumulation period 495 of the upper end line, and time t66 is the start time of the accumulation period 496 of the next line. Further, time t67 is the start time of the reading period 497 for the upper end line, and time t68 is the start time of the reading period 498.

  In the picture A timing sequence shown in FIG. 25, the slope of the line connecting the start times of the charge accumulation periods of the respective lines is larger than that in the picture A timing sequence shown in FIG. That is, the time difference between the start times of the accumulation periods in each line of the pixel array is shortened. Specifically, the time from the start time t65 of the storage period 495 shown in FIG. 25 to the start time t66 of the next storage period 496 is the start of the next storage period 492 from the start time t59 of the storage period 491 shown in FIG. It is set shorter than the time up to time t60. However, the start time of the readout period for each line is the same as that before the timing sequence change, and the readout of the next line is performed after the readout of the previous line is completed. By photographing the subjects 701 and 702 shown in FIG. 22A according to such a timing sequence, an image by picture A shown in FIG. 26 is obtained. In the image by picture A shown in FIG. 26, the rolling shutter distortion of the subject 702 is reduced compared to FIG. 24A because the time difference between the start times of the accumulation periods in each line is shortened. In this way, for picture A, the rolling shutter distortion is reduced by shortening the time difference between the start times of the accumulation periods and increasing the waiting time from the end of the accumulation period to the start of reading. be able to.

  The magnitude of the rolling shutter distortion depends on the moving speed of the subject with respect to the shooting angle of view. Therefore, the rolling shutter distortion can be reduced by shortening the time difference of the start time of the accumulation period for each line in accordance with the vector change of the rolling shutter distortion detected by the image distortion detecting means.

  Further, since the image distortion detection means can know the range in which the rolling shutter distortion occurs, the image distortion correction means can also correct the picture B rolling shutter distortion. For example, the image distortion of picture B is corrected by replacing the range in which the rolling shutter distortion of the image by picture B is generated with an image by picture A in the same range. In this way, rolling shutter distortion of both picture A and picture B can be reduced. The picture B image distortion correction means is not limited to a method for replacing an image in a range where the rolling shutter distortion occurs, and for example, the image distortion can be corrected by performing geometric transformation. is there.

  By the way, in order to remove the rolling shutter distortion, it seems that the accumulation period should be started at the same time in all lines. However, if the standby period in which charges are held in the floating diffusion region is long, noise such as dark current occurs. Therefore, it is necessary to set the standby period from the end of the accumulation period to the start of reading in consideration of the balance with the generated noise. It is also conceivable that the speed of the subject on the image sensor surface changes during shooting. Therefore, when the rolling shutter distortion is small, it is desirable to control the timing sequence in accordance with the amount of rolling shutter distortion, such as increasing the time difference of the start time of the accumulation period for each line to reduce the noise effect.

  Next, with respect to picture B, a method of correcting rolling shutter distortion by shortening the time difference between the start times of the accumulation times as in picture A will be described. FIG. 27 is a timing chart for explaining a method of correcting the picture B rolling shutter distortion in the imaging apparatus according to the second embodiment of the present invention. FIG. 27A in the upper stage shows a picture B accumulation period 463 and a read timing sequence similar to those in the prior art. Further, FIG. 27B in the middle stage shows a timing sequence of picture B in which the time difference between the start times of the accumulation periods 464 of the respective lines is shortened.

  The frame period t70 to t72 of the corrected picture B shown in FIG. 27 (b) is longer than the frame period t70 to t71 of the picture B before correction shown in FIG. 27 (a). This is because the waiting period until the start of reading is lengthened while the length of the accumulation period 464 is maintained. As a result, in FIG. 27B, the frame rate of picture B is lowered. On the other hand, in picture A, since the accumulation periods 495 and 496 are short and there is a margin time during the frame period during which neither accumulation nor readout is performed, the frame period does not become long even after correction by the image distortion correction means.

  Therefore, in the present embodiment, the accumulation period 466 is set shorter as shown in FIG. 27C in accordance with the reduction in the time difference between the start times of the accumulation periods of picture B. As a result, the corrected frame period can be accommodated at the same time t70 to time t71 as shown in FIG. 27A, as shown in FIG. 27C, so that rolling shutter distortion is reduced while maintaining the frame rate. can do. Moreover, since the readout timing can be matched between picture A and picture B, the control of the image sensor can be simplified.

  By the way, in the normal mode other than the image distortion detection mode, after the reading process from the charge holding portions (floating diffusion regions 313A and 313B) of the previous line is completed, the reading process in the next line is performed. For example, the readout processing of the double correlation sampling circuit for removing noise in the readout circuit 308 described with reference to FIG. 19 is performed within the readout period 488 indicated by the dotted line in the timing sequence of FIG. In this manner, in the normal mode, readout is not performed simultaneously for pixels on different lines in the same column of the image sensor, so the readout circuit 308 is shared by pixels on different lines in the same column. Here, the pixels on different lines in the same column are pixels corresponding to, for example, photodiode groups arranged in the vertical direction connected by the same vertical scanning signal output line 304A shown in FIG.

On the other hand, in the image distortion detection mode, if the next line read process can be performed before the previous line read process is completed, the accumulation time start time is also shortened for picture B without shortening the accumulation period. The time difference can be shortened. Therefore, in the present embodiment, it is considered that the second reading unit (reading circuit 308B) is configured using a plurality of reading units that can independently read different rows in the same column.
For example, it is assumed that, after the reading process for a certain line is completed, the reading process for the five lines below is started. When n is a natural number from 1 to N and the total number of lines of the image sensor is (5N + 1), the same readout circuit 308 is shared only for a plurality of lines (5n + 1). Similarly, independent readout circuits are shared for pixels in a plurality of lines of (5n + 2) rows, (5n + 3) rows, and (5n + 4). Thereby, even while the (5n + 1) rows are being read, the reading processing of the (5n + 2) rows, the (5n + 3) rows, and the (5n + 4) rows can be performed in parallel. In addition, the number of readout circuits 308 can be reduced and an increase in circuit scale can be suppressed as compared with the case where a readout circuit is provided for each pixel.

  As described above, in this embodiment, the image distortion correction unit that corrects image distortion of at least one of the first photoelectric conversion unit and the second photoelectric conversion unit based on the detection result of the image distortion by the image distortion detection unit is provided. ing. Thus, image distortion can be corrected when two images are simultaneously captured using a single image sensor.

  Note that the image distortion correction unit executed by the system control CPU 178 may record an image corrected for rolling shutter distortion on a recording medium, or may use other methods. For example, picture A and picture B are recorded on a recording medium without being subjected to image distortion correction, and the image is captured by an external video processing apparatus such as a personal computer having image distortion detection means and image distortion correction means. Correction processing may be performed.

(Modified embodiment)
The present invention is not limited to the above embodiment, and various modifications can be made. For example, the configuration of the imaging apparatus described in the above embodiment is an example, and the imaging apparatus to which the present invention can be applied is not limited to the configuration illustrated in FIGS. 1 and 2. Further, the circuit configuration of each part of the image sensor is not limited to the configuration shown in FIGS. Further, the photoelectric conversion unit is not necessarily limited to the photodiode as shown in FIG. 4, and any photoelectric conversion unit may be used as long as it has a photoelectric conversion function.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100 ... Imaging device 152 ... Shooting optical system 178 ... System control CPU (Image distortion detection means, Image distortion correction means)
184: Image sensors 187, 188 ... Digital signal processing unit 255 ... Light guide 302 ... Pixel array 303 ... Pixels 303A, 303B ... Pixel elements 308A, 308A ... Read circuit (read unit)
310A, 310B ... Photodiode (photoelectric conversion unit)
313A, 313B: Floating diffusion region (charge holding unit)

Claims (11)

A first photoelectric conversion unit; a second photoelectric conversion unit having a light receiving efficiency lower than that of the first photoelectric conversion unit; a first charge holding unit for holding charges photoelectrically converted and accumulated in the first photoelectric conversion unit; A pixel array in which pixels having a second charge holding unit that holds charges converted and accumulated by two photoelectric conversion units are arranged in a matrix;
A first readout unit that is provided in each column of the pixel array and reads out a first video signal based on a charge held by the first charge holding unit of the pixel;
A second reading unit that is provided in each column of the pixel array and reads a second video signal based on the charge held by the second charge holding unit of the pixel;
In the state where the time difference between the start time of the charge accumulation period in each row of the pixel array and the start time of the accumulation period of another row is different between the first photoelectric conversion unit and the second photoelectric conversion unit, Image distortion detection means for detecting image distortion by comparing one video signal and the second video signal;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the accumulation period of the second photoelectric conversion unit is set longer than the accumulation period of the first photoelectric conversion unit.   The image distortion detection means makes the time difference of the start time of the accumulation period of the first photoelectric conversion unit shorter than the time difference of the start time of the accumulation period of the second photoelectric conversion unit, and the first photoelectric conversion unit The imaging apparatus according to claim 1, wherein a waiting period from the end of the accumulation period to the start of reading of the first video signal is lengthened.   The image distortion detection means generates image distortion when the magnitude of a motion vector obtained by image matching the first video signal and the second video signal exceeds a preset threshold value. The imaging device according to claim 1, wherein the imaging device is determined to be.   The image distortion correction means for correcting image distortion of at least one of the first photoelectric conversion unit and the second photoelectric conversion unit based on a detection result of the image distortion by the image distortion detection unit. Item 5. The imaging device according to any one of Items 1 to 4.   The image distortion correcting unit corrects the image distortion by replacing the second video signal in a range where the image distortion is determined to be generated by the image distortion detecting unit with the first video signal. The imaging device according to claim 5.   The image distortion correction unit is configured to perform the accumulation period for a row in which image distortion is determined by the image distortion detection unit in at least one of the first photoelectric conversion unit and the second photoelectric conversion unit. The image pickup apparatus according to claim 5, wherein the image distortion is corrected by shortening the time difference between the start times.   The image distortion correction unit is configured to reduce the time difference between the start times of the accumulation periods of the first photoelectric conversion unit, and the first video signal after the accumulation period of the first photoelectric conversion unit ends. The image pickup apparatus according to claim 7, wherein a waiting period until the start of reading is increased.   The image distortion correction means sets the second video image by shortening the accumulation period of the second photoelectric conversion unit in response to shortening the time difference of the start time of the accumulation period of the second photoelectric conversion unit. The imaging apparatus according to claim 7 or 8, wherein a frame rate of the signal is maintained.   10. The imaging apparatus according to claim 7, wherein the second readout unit is configured by using a plurality of readout units that can independently read out different rows of the same column. A first photoelectric conversion unit; a second photoelectric conversion unit having a light receiving efficiency lower than that of the first photoelectric conversion unit; a first charge holding unit for holding charges photoelectrically converted and accumulated in the first photoelectric conversion unit; In a pixel array in which pixels having a second charge holding unit that holds charges converted and accumulated by two photoelectric conversion units are arranged in a matrix,
Setting the time difference between the start time of the charge accumulation period in each row of the pixel array and the start time of the accumulation period of the other row to be different between the first photoelectric conversion unit and the second photoelectric conversion unit; ,
Reading a first video signal based on the charge held by the first charge holding unit of the pixel;
Reading a second video signal based on the charge held by the second charge holding unit of the pixel;
Detecting image distortion by comparing the first video signal and the second video signal;
An image distortion detection method comprising:

Images