JP2015080180A - Image-capturing device and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To equalize processing loads required for reading out a pixel signal and simplify timing control in a rolling shutter operation.SOLUTION: A short-period vertical synchronization signal VDS and a long-period vertical synchronization signal VDL with a cycle equal to N cycles of the short-period vertical synchronization signal VDS are generated. Long-time exposure is executed on long-time exposure lines set at prescribed line intervals in accordance with the VDL, and short-time exposure and intermediate-time exposure are executed on other lines of an image-capturing element in accordance with the VDS. As for the long-time exposure lines, pixel signal readout by long-time exposure is started in synchronism with the VDL, and for the other lines, pixel signal readout by short-time exposure is started in accordance with the VDS that does not overlap a period for pixel signal readout by long-time exposure. Readout of the pixel signal obtained by the intermediate-time exposure is started in synchronism with the VDS that does not overlap the period for readout of the pixel signal obtained by the long-time exposure and for readout of the pixel signal obtained by the short-time exposure.

Description

  The present invention relates to an imaging apparatus and a control method thereof.

  A CCD (Charge Coupled Device) type image sensor (hereinafter referred to as “CCD sensor”) is generally used as an image pickup element used in the image pickup apparatus. However, in recent years, CMOS (Complementary Metal Oxide Semiconductor) type image sensors (hereinafter referred to as “CMOS sensors”) are attracting attention as the number of pixels of an image pickup device increases.

  An imaging element such as a CMOS sensor performs photoelectric conversion in which a photoelectric conversion element of a pixel accumulates charges corresponding to the amount of incident light and outputs an electrical signal corresponding to the accumulated charges. An image sensor such as a CMOS sensor has an electronic shutter function. The electronic shutter function starts exposure by resetting the photoelectric conversion element of the pixel, and ends exposure by reading out the electric charge accumulated in the photoelectric conversion element. Thus, since the start and end of exposure are controlled only by the function of the image sensor, accurate exposure time control from a low-speed shutter to a high-speed shutter can be realized.

  Further, one of the features of the CMOS sensor is a rolling shutter operation (also called a focal plane shutter operation). In the rolling shutter operation, unlike the CCD sensor, a plurality of two-dimensionally arranged pixels are sequentially scanned for each line to reset the charge of the pixels. Then, after a predetermined exposure time has elapsed, scanning is sequentially performed for each line, and the accumulated charges are read and signals are output.

  As described above, the rolling shutter operation is an operation having a time difference between charge readout and signal output for each line. As a result, the exposure time is shifted for each line in one imaging operation.

  By the way, in such an imaging apparatus using an imaging element such as a CMOS sensor, there is a problem that the dynamic range tends to be insufficient when imaging a subject in which a bright place and a dark place are mixed. For example, if the exposure time is controlled to be short in accordance with a bright place, a sufficient exposure time cannot be obtained in a dark part, and therefore, the image quality is deteriorated due to blackout or S / N deterioration. Conversely, if the exposure time is controlled to be longer in a dark place, the accumulated charge amount of the photoelectric conversion element reaches a saturation level, and overexposure occurs when the subject area with a certain brightness or more is set to a saturated luminance level. Occur.

  A dynamic range expansion process is a method for accurately reproducing the gradation of bright and dark areas. The dynamic range expansion process realizes a high S / N by controlling the exposure time long for pixels with a small amount of incident light on the image sensor, and avoids saturation for pixels with a large amount of incident light.

  As one method of dynamic range expansion processing, a multiple exposure method is known in which a plurality of images having different exposure times are continuously captured by the same image sensor and combined. In the multiple exposure method, a long-exposure image and a short-exposure image are taken separately and continuously. A long exposure image is used for a dark image region, and a short exposure image is used for a bright image region that is overexposed in a long exposure image, and then the composition processing is performed. Thus, one image with an expanded dynamic range is generated.

  However, in the multiple exposure method, it is necessary to alternately perform long-time exposure and short-time exposure and alternately read out signals obtained from the respective exposures. Further, when the multiple exposure method is applied to moving image shooting, in order to make the frame rate uniform, it is necessary to make the one frame period in the long exposure and the one frame period in the short exposure the same. Then, the exposure time of long exposure is 1 frame period at the maximum, and the exposure time can be set to 1 frame period at the maximum even in the operation of a normal image sensor, so the effect of improving the S / N by long exposure is expected. Can not. Furthermore, since the exposure time for short exposure is set to a maximum of half of one frame period when the exposure time ratio is 2: 1, a waste time that does not contribute to exposure also occurs.

  In the multiple exposure method, a time difference of one frame period occurs between the long exposure and the short exposure. Therefore, if there is a moving subject, the long exposure image and the short exposure image are not the same. As a result, there is a problem that a moving subject is blurred or a false color is generated in an image whose dynamic range is enlarged.

  In response to such a problem of the multiple exposure method, another method of dynamic range expansion processing is disclosed in Patent Documents 1 and 2. In Patent Documents 1 and 2, different exposure times are set for pixels grouped in two every two lines. Then, for example, high-sensitivity pixel information is acquired from long-time exposure pixels, low-sensitivity pixel information is acquired from short-time exposure pixels, and an image with an expanded dynamic range is generated based on the pixel information of these different sensitivities. According to this method, an image with an expanded dynamic range can be generated based on one captured image.

  Patent Document 2 also proposes a method for eliminating the blurring of a moving subject, which is a problem of the multiple exposure method, by aligning the centers of the exposure times of long exposure and short exposure (Patent Document 2). FIG. 13).

JP 2012-105225 A JP 2011-244309 A

  However, in the method described in FIG. 6 of Patent Document 1 and FIG. 9 of Patent Document 2, all lines of signals of pixels having different exposure times for every two lines are read out simultaneously. Therefore, it is necessary to switch the correction processing and signal processing for the long exposure pixels and the correction processing and signal processing for the short exposure pixels every two lines. Such processing places a heavy burden on the signal processing circuit.

  In the method described in FIG. 13 of Patent Document 2, it is necessary that the center of short-time exposure always matches the center of long-time exposure. In this case, it is necessary to change the reset and readout timing for aligning the center of the short exposure each time the exposure time is changed and the long exposure reset and readout timing is changed. Therefore, such a method has a problem that timing control becomes complicated.

  The present invention is an improvement of a rolling shutter operation for dynamic range expansion processing, and realizes a technique that achieves both leveling of processing load for pixel signal readout and simplification of timing control.

  According to an aspect of the present invention, an imaging device in which a plurality of pixels having photoelectric conversion elements are arranged in a matrix, a short-term vertical synchronization signal that is a vertical synchronization signal for short-time exposure, and one cycle of the short-term vertical Synchronization signal generating means for generating a long-term vertical synchronization signal for long exposure equal to a predetermined period of the synchronization signal, and a long-exposure line set at a predetermined line interval of the image sensor in accordance with the long-term vertical synchronization signal Exposure control means for performing short-time exposure and medium-time exposure on each of the other lines of the image sensor according to the short-term vertical synchronization signal, and the exposure control means, For the long exposure line, readout of the pixel signal obtained by the long exposure is started in synchronization with the long vertical synchronization signal, and for the other lines, the long The readout of the pixel signal obtained by the short-time exposure is started in synchronization with a short-term vertical synchronization signal that does not overlap with the readout period of the pixel signal of the line for intermediate exposure, and the readout of the pixel signal obtained by the long-time exposure is started. And readout of the pixel signal obtained by the medium-time exposure in synchronization with a short-term vertical synchronization signal that does not overlap with the period of readout and the readout period of the pixel signal obtained by the short-time exposure. Is provided.

  According to the present invention, it is possible to level the processing load and simplify the timing control for reading out pixel signals in the rolling shutter operation for dynamic range expansion processing.

1 is a diagram illustrating a configuration of an imaging apparatus according to an embodiment. 1 is a diagram illustrating a schematic configuration of an image sensor according to an embodiment. FIG. 3 is a diagram illustrating control timing of the image sensor according to the first embodiment. FIG. 3 is a diagram illustrating control timing of the image sensor according to the first embodiment. FIG. 3 is a diagram illustrating control timing of the image sensor according to the first embodiment. The figure which shows the control timing of the image pick-up element which concerns on 2nd Embodiment. The figure which shows the control timing of the image pick-up element which concerns on 2nd Embodiment. The figure explaining the dynamic range expansion process which concerns on embodiment.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiment described below is an example of the configuration of the present invention, and should be appropriately modified or changed depending on the configuration of the apparatus to which the present invention is applied and various conditions. The present invention is described below. The form is not limited.

<First Embodiment>
In the first embodiment, for use in dynamic range expansion processing (hereinafter referred to as “HDR processing”), the operation of the image sensor controlled to output signals of pixels having different exposure times for every two lines. explain.

  FIG. 1 is a diagram illustrating a configuration of an imaging apparatus according to the present embodiment. The imaging device of this embodiment can be applied to a digital still camera, a digital video camera, and the like. The imaging apparatus shown in FIG. 1 includes an optical system 11, an imaging device 12, a signal processing unit 13, a compression / decompression unit 14, a synchronization control unit 15, an operation unit 16, an image display unit 17, and an image recording unit 18. The optical system 11 includes a lens for condensing light from a subject on the image sensor 12, a drive mechanism for moving the lens to perform zooming and focusing, a mechanical shutter mechanism, an aperture mechanism, and the like. Among these, the movable part is driven based on a control signal from the synchronization control part 15.

  The imaging element 12 is an XY address type CMOS sensor in which a plurality of pixels having photoelectric conversion elements are arranged in a matrix. The image sensor 12 includes a CDS (Correlated Double Sampling) circuit, an AGC (Auto Gain Control) circuit, an A / D converter, and the like, and is controlled by a control signal from the synchronization control unit 15. Here, the CMOS sensor performs an imaging operation such as exposure, signal readout, and reset in accordance with a control signal from the synchronization control unit 15. The digital image signal is output through noise removal by the CDS circuit, gain control by the AGC circuit, and analog-digital conversion by the A / D converter.

  The signal processing unit 13 performs white balance adjustment, color correction, gamma correction, AF (Auto Focus), AE on the digitized image signal input from the image sensor 12 under the control of the synchronization control unit 15. Apply signal processing such as (Auto Exposure). In the present embodiment, the signal processing unit 13 performs HDR processing.

  As a method of HDR processing, for example, as shown in FIG. 4 of Patent Document 1, a signal of a long exposure pixel and a signal of a short exposure pixel are used by using a gain value for compensating a sensitivity ratio and a weighting coefficient according to brightness. There is a method for synthesizing signals. Alternatively, as shown in FIG. 7 of Patent Document 2, a method of selecting either a long exposure pixel signal or a short exposure pixel signal in accordance with the brightness may be employed.

  The compression / decompression unit 14 compresses and encodes the image signal from the signal processing unit 13 in a predetermined still image data format such as JPEG (Joint Photographic Coding Experts Group) under the control of the synchronization control unit 15. Do. The compression / decompression unit 14 also performs decompression / decoding processing on the encoded data of the still image supplied from the synchronization control unit 15. Further, the compression / decompression unit 14 can also perform compression encoding / decompression decoding processing of moving images by MPEG (Moving Picture Experts Group) or the like.

  The synchronization control unit 15 is a microcontroller composed of, for example, a CPU, a ROM, a RAM, and the like, and comprehensively controls each unit of the imaging apparatus by executing a program stored in the ROM.

  The operation unit 16 includes various operation keys such as a shutter release button, a lever, a dial, and the like, and outputs a control signal according to an input operation by the user to the synchronization control unit 15.

  The image display unit 17 includes a display device such as an LCD (Liquid Crystal Display), an interface circuit for the display device, and the like. The image display unit 17 generates an image signal to be displayed on the display device from the image signal supplied from the synchronization control unit 15, and supplies the signal to the display device to display an image.

  The image recording unit 18 is realized as, for example, a portable semiconductor memory, an optical disk, an HDD (Hard Disk Drive), a magnetic tape, or the like. The image recording unit 18 receives the image data file encoded by the compression / decompression unit 14 from the synchronization control unit 15 and stores it. The image recording unit 18 also reads designated data based on the control signal from the synchronization control unit 15 and outputs the data to the synchronization control unit 15.

  The configuration of the imaging apparatus in the present embodiment is generally as described above. Next, a basic operation of the imaging apparatus in the present embodiment will be described.

  Prior to capturing a still image, image signals output from the image sensor 12 are sequentially supplied to the signal processing unit 13. The signal processing unit 13 performs image quality correction processing on the digital image signal from the image sensor 12 and supplies the image display unit 17 through the synchronization control unit 15 as a camera-through image signal. As a result, the camera-through image is displayed, and the user can adjust the angle of view while viewing the display image. In this state, when the shutter release button of the operation unit 16 is pressed, the image processing signal for one frame from the image sensor 12 is taken into the signal processing unit 13 under the control of the synchronization control unit 15. The signal processing unit 13 performs image quality correction processing on the captured image signal for one frame, and supplies the processed image signal to the compression / decompression unit 14. The compression / decompression unit 14 compresses and encodes the input image signal, and supplies the generated encoded data to the image recording unit 18 through the synchronization control unit 15. As a result, the data file of the captured still image is recorded in the image recording unit 18.

  On the other hand, when playing back a still image data file recorded in the image recording unit 18, the synchronization control unit 15 receives the selected data file from the image recording unit 18 in response to an operation input from the operation unit 16. The data is read and supplied to the compression / decompression unit 14 to execute the decompression decoding process. The decoded image signal is supplied to the image display unit 17 via the synchronization control unit 15, whereby a still image is reproduced and displayed.

  When recording a moving image, the image signal sequentially processed by the signal processing unit 13 is subjected to compression encoding processing by the compression / decompression unit 14, and the encoded data of the generated moving image is sequentially converted into the image recording unit 18. Transfer to and record. In addition, a moving image data file is read from the image recording unit 18, supplied to the compression / decompression unit 14, decompressed and decoded, and supplied to the image display unit 17, thereby displaying a moving image.

  FIG. 2 is a diagram illustrating a configuration of the image sensor 12 according to the present embodiment. The imaging element 12 (CMOS sensor) shown in FIG. 2 includes a pixel region 200 composed of a plurality of pixels. The image sensor 12 also includes a vertical selection unit 220, a pixel control line 230 and a vertical signal line 240 connected to each pixel. The imaging device 12 further includes a column signal processing unit 250, a horizontal memory unit 260, a horizontal selection unit 270, an output unit 280, and a TG 290 (Timing Generator).

  The pixel region 200 includes CMOS sensor pixels (not shown) having photoelectric conversion elements and transistors, and the pixels are arranged in a matrix in the horizontal and vertical directions. In FIG. 2, each pixel is indicated by P (0101) to P (1616), with the upper two digits as row numbers and the lower two digits as column numbers. FIG. 2 shows an example of a 16 × 16 array, but the present invention is not limited to a specific number of matrices.

  The vertical selection unit 220 selects the pixel array of the pixel region 200 row by row, and controls the reset operation and readout operation of the selected pixel row. The pixel control line 230 is commonly connected to each pixel row, and transmits a control signal in units of rows by the vertical selection unit 220. The vertical signal lines 240 are commonly connected to the respective pixel columns, and the signals of the pixels in the row selected by the pixel control line 230 are read out to the corresponding vertical signal lines 240, respectively. The column signal processing unit 250 includes a CDS circuit (not shown), an AGC circuit, and an A / D converter that are provided in each of the vertical signal lines 240. For each pixel unit pixel signal sent through the vertical signal line 240, the CDS circuit removes fixed pattern noise caused by variations in the threshold values of the transistors in the pixel circuit, and calculates S / N. Sample and hold to keep it good. The AGC circuit performs gain control. The A / D converter performs analog-to-digital conversion.

  The horizontal memory unit 260 is provided for each vertical signal line 240 and stores pixel signals in units of rows digitized by the corresponding column signal processing units 250. The horizontal selection unit 270 selects the corresponding horizontal memory unit 260 for each column, and outputs the stored digitized pixel signal to the signal processing unit 13 via the output unit 280. The TG 290 outputs various clock signals and control signals necessary for the operation of each unit of the image sensor 12 based on the control signal from the synchronization control unit 15.

  Hereinafter, the exposure and read timing control of the image sensor 12 in the embodiment will be described in detail. As described above, in the method described in FIG. 6 of Patent Document 1 and FIG. 9 of Patent Document 2, all the lines of signals of pixels having different exposure times for every two lines are read out simultaneously. Therefore, it is necessary to switch the correction processing and signal processing for the long exposure pixels and the correction processing and signal processing for the short exposure pixels every two lines. Such processing places a heavy burden on the signal processing circuit.

  In addition, when performing HDR processing, simultaneously with the correction processing and signal processing for the pixels, HDR processing is performed together with the positional deviation correction processing between the long exposure pixels and the short exposure pixels for every two lines. . This further increases the burden on the system of the entire imaging apparatus.

  Further, in this method, since all the lines of pixel signals having different exposure times are read every two lines using a rolling shutter operation, it takes time to read from the first line to the last line. For this reason, there is also a problem that a phenomenon called rolling distortion is likely to occur, in which a moving subject is distorted by the amount of exposure time shifted for each line.

  In the method described in FIG. 13 of Patent Document 2, it is necessary that the center of short-time exposure always matches the center of long-time exposure. In this case, it is necessary to change the reset and readout timing for aligning the center of the short exposure each time the exposure time is changed and the long exposure reset and readout timing is changed. Therefore, such a method has a problem that timing control becomes complicated.

  Further, in this method, when the exposure time is shortened, it may be considered that the time taken to read all the lines to be exposed for a short time exceeds half of the difference between the long time exposure and the short time exposure. In this case, since the end of reading of a line that performs short-time exposure collides with the start of reading of a line that performs long-time exposure, there is also a problem that reading cannot be performed for each line.

  Therefore, in the present embodiment, the processing described below will be performed to achieve leveling of the processing load required for reading out pixel signals while simplifying timing control.

  FIG. 3 is a diagram illustrating the control timing of the image sensor 12 according to the present embodiment. In the present embodiment, for HDR processing, long exposure lines (hereinafter referred to as “long exposure lines”) are set in the pixel rows of the pixel region 200 at predetermined line intervals (for example, two line intervals). The Each of the other lines is subjected to short time exposure and medium time exposure. In FIG. 3, VDL is a vertical synchronization signal (long-term vertical synchronization signal) for long exposure, and VDS is a vertical synchronization signal (short-term vertical synchronization signal) for short exposure. The vertical synchronization period of the long exposure frame is indicated by Tfr1, and the vertical synchronization period of the short exposure frame is indicated by Tfrs. The vertical synchronization period Tfrl corresponding to one cycle of the long-term vertical synchronization signal is equal to a predetermined cycle of the short-term vertical synchronization signal, for example, N cycles (= N · Tfrs, where N is a natural number of 2 or more). In the example of FIG. 3, N = 2, that is, two periods. In the present embodiment, these short-term vertical synchronization signal and long-term vertical synchronization signal are generated by the TG 290 as the synchronization signal generating means. HD indicates a horizontal synchronization signal, which is a period in which pixel reset operations and readout operations are performed in units of rows.

  Line 01 to Line 16 indicate the operation states of the pixel rows P (01−−) to P (16−−) in the pixel region 200. In order to represent the pixel row by the row number, the column number is displayed as “-”. The long-time exposure lines are 01, 02, 05, 06, 09, 10, 13, 14 and the short-time exposure lines are 03, 04, 07, 08, 11, 12 so that the exposure time differs every two lines. , 15 and 16 lines.

  In the present embodiment, a medium-time exposure that is intermediate between the short-time exposure and the long-time exposure is further set and executed for the short-time exposure line. Then, after reading the pixel signals of each line to the column signal processing unit 250, the corresponding 1HD period is displayed as RO as a period for outputting the pixel signals for one line from the output unit 280. Here, the period for reading the pixel signals of each line to the column signal processing unit 250 is sufficiently shorter than the period for outputting the pixel signals for one line from the output unit 280, so that the next exposure frame starts from the RO period. Shall begin. In addition, each pixel is provided with a 2 × 2 color filter such as a Bayer array so that dynamic range expansion processing can be easily performed from a signal with different exposure times for every two lines.

  ReadOut indicates the timing at which the output unit 280 outputs the read output signal Fr_L Readout of the long exposure frame, the output signal Fr_M Readout of the medium exposure frame, and the output signal Fr_S Readout of the short exposure frame.

  First, at the timing t00, the 01 line is read out in synchronization with the first long-term vertical synchronization signal VDL, thereby resetting the pixels and starting the 01 line exposure. Subsequently, at timing t01, the 02 lines are read to reset the pixels and start the 02 lines. In the same manner after t02, the 05 line, 06 line, 09 line, 10 line, 13 line and 14 line are reset, and exposure of each line is started.

  In this way, every time 1 HD passes, the 01, 02, 05, 06, 09, 10, 13, and 14 lines, which are long exposure lines, are reset, so that the line sequential rolling shutter operation of the long exposure frame is performed. Can start.

  Next, at time t05 after the elapse of Tfrs from timing t00, readout of 03 lines is performed in synchronization with the first short-term vertical synchronization signal VDS next to VDL, thereby resetting the pixels and starting exposure of 03 lines. . Subsequently, at timing t06, the pixels are reset by reading 04 lines, and exposure of 04 lines is started. In the same way after t07, the 07 line, 08 line, 11 line, 12 line, 15 line and 16 line are reset, and the exposure of each line is started.

  Thus, every time 1 HD elapses, the 03, 04, 07, 08, 11, 12, 15, and 16 lines, which are short-time exposure lines, are reset, so that the line-sequential rolling shutter operation of the short-time exposure frame is performed. Can start. Then, at time t10 after 2Tfrs elapses from timing t00, the pixel signals of the 03 line are output in the RO period in synchronization with the second short-term vertical synchronization signal VDS. As a result, the pixels are reset, and medium time exposure of 03 lines is started. Subsequently, at timing t11, the pixel signals of the 04 line are output during the RO period. As a result, the pixels are reset, and medium time exposure of 04 lines is started. In the same way after t12, pixel signals of 07 line, 08 line, 11 line, 12 line, 15 line and 16 line are output in each RO period. As a result, each pixel is reset, and medium-time exposure is started for 07 line, 08 line, 11 line, 12 line, 15 line and 16 line.

  Thus, every time 1 HD elapses, the 03, 04, 07, 08, 11, 12, 15, and 16 lines that are the short exposure lines are read out, thereby completing the line sequential rolling shutter operation of the short exposure frames. To do. At the same time, it is possible to start a line-sequential rolling shutter operation in a medium time exposure frame in which medium time exposure is performed on a short time exposure line. At this time, the long exposure lines 01, 02, 05, 06, 09, 10, 13, and 14 continue to be exposed. The output signal of the short exposure frame at this time is Fr_S Readout. Here, in this short exposure frame, since the pixel is reset by reading out the pixel, the longest exposure time is Tfrs of the vertical synchronization period (short-term vertical synchronization period) of the short exposure frame. It has become.

  Next, at t15 after Tfrl (= 3Tfrs) has elapsed from timing t00, the pixel signal of the 01 line is output in the RO period in synchronization with the second long-term vertical synchronization signal VDL. Subsequently, at timing t16, the pixel signals of the 02 line are output in the RO period. In the same manner after t17, pixel signals of 05 line, 06 line, 09 line, 10 line, 13 line and 14 line are output in each RO period.

  In this way, every time 1 HD elapses, the 01, 02, 05, 06, 09, 10, 13, 14 lines that are long exposure lines are read to complete the line sequential rolling shutter operation of the long exposure frame. To do. At this time, exposure is continued for the 03, 04, 07, 08, 11, 12, 15, and 16 lines, which are short-time exposure lines in which medium-time exposure is performed. The output signal of the long exposure frame at this time becomes Fr_L Readout. Here, in this long exposure frame, the pixels are reset by reading out the pixels, so the longest exposure time is Tfrl of the vertical synchronization period of the long exposure frame.

  Further, at t20 after the elapse of 4 Tfrs from timing t00, the pixel signal of the 03 line is output in the RO period in synchronization with the third short-term vertical synchronization signal VDS next to the second long-term vertical synchronization signal VDL. Subsequently, at timing t21, the pixel signals of the 04 line are output in the RO period. In the same way after t22, pixel signals of 07 line, 08 line, 11 line, 12 line, 15 line and 16 line are output in each RO period.

  In this way, every time 1 HD elapses, the 03, 04, 07, 08, 11, 12, 15, and 16 lines, which are the short-time exposure lines for which the medium-time exposure is performed, are read out, so that The line sequential rolling shutter operation ends. The output signal of the medium time exposure frame at this time becomes Fr_S Readout. Here, in this medium time exposure frame, since the pixel is reset by reading out the pixel, the longest exposure time is Tfrm which is 2 Tfrs of the vertical synchronization period of the short time exposure frame. .

  In the operation of FIG. 3, the longest exposure time Tfrl of the long exposure frame is three times longer than the longest exposure time Tfrs of the short exposure frame. Further, the longest exposure time Tfrm of the medium-time exposure frame is twice as long as the longest exposure time Tfrs of the short-time exposure frame.

  In addition, in the operation of FIG. 3, it is understood that the medium time exposure using the short time exposure line is controlled so as to be executed in the idle time of the short time exposure. Further, since the readout is performed separately for the short exposure line and the long exposure line every two lines, the readout time of each frame is about half that of the operation of reading all the lines at once.

  FIG. 4 is a diagram illustrating control timings of the long exposure frame, the medium exposure frame, and the short exposure frame according to the present embodiment. In FIG. 4, the operation for each line in FIG. 3 is shown by the operation for each frame, and in order to apply it to moving image shooting, the control timing enables continuous shooting. In order to enable continuous shooting, the control timing after timing t36 returns to t30 and is repeated. The same operations and configurations as those in FIG. 3 are denoted by the same reference numerals.

  From the timing t30 to t36 and after the timing t36 are the first long-term vertical synchronization period Tfrl1 and the second long-term vertical synchronization period Tfrl2, respectively. The timing t30 to t32 is the first short-term vertical synchronization period Tfrs1. From the timing t32 to t34 is the second short-term vertical synchronization period Tfrs2. The timing t34 to t36 is the third short-term vertical synchronization period Tfrs3. The timing t36 to t38 is the fourth short-term vertical synchronization period Tfrs4. The period from timing t38 to t40 is the fifth short-term vertical synchronization period Tfrs5.

  Fr_L indicates the operation state of the 01, 02, 05, 06, 09, 10, 13, and 14 lines constituting the long exposure frame. Fr_M and Fr_S indicate the operation states of the 03, 04, 07, 08, 11, 12, 15, and 16 lines, which are the short-time exposure lines constituting the medium-time exposure frame and the short-time exposure frame, respectively.

Here, in Fr_L, a rolling shutter operation is performed continuously every long vertical synchronization period Tfrl in synchronization with VDL.
In Fr_S, the exposure of the rolling shutter operation is started in synchronization with the VDS after Tfrs from the VDL, and the reading of the rolling shutter operation is started in synchronization with the VDS after Tfrs.
Further, in Fr_M, exposure of the rolling shutter operation is started in synchronization with VDS after 2 Tfrs from VDL, which is the free time of Fr_S, and reading of the rolling shutter operation is started in synchronization with VDS after 2 Tfrs.
This operation is repeatedly performed in synchronization with the VDL.

  That is, for the short exposure line, reading of the pixel signal is started in synchronization with the VDS that does not overlap with the reading period of the pixel signal obtained by the long exposure. In addition, readout of the pixel signal obtained by the medium-time exposure is started in synchronization with the VDS that does not overlap with the readout period of the pixel signal obtained by the long-time exposure and the readout period of the pixel signal obtained by the short-time exposure.

  Specifically, in the first short-term vertical synchronization period Tfrs1 constituting the first long-term vertical synchronization period Tfrl1, the output signal Fr_L Readout1 of the long-time exposure frame in which exposure is started one round before is output. Next, in the second short-term vertical synchronization period Tfrs2, the output signal Fr_M Readout2 of the medium time exposure frame in which the exposure is started one round before is output. Thereafter, in the third short-term vertical synchronization period Tfrs3, the short-exposure frame output signal Fr_S Readout3 is output.

  Similarly, in the fourth short-term vertical synchronization period Tfrs4 constituting the second long-term vertical synchronization period Tfr12, the long-exposure frame output signal Fr_L Readout4 is output. Next, in the fifth short-term vertical synchronization period Tfrs5, an output signal Fr_M Readout5 of the medium time exposure frame is output.

  Accordingly, the output signal Fr_L Readout of the long exposure frame, the output signal Fr_M Readout of the medium exposure frame, and the output signal Fr_S Readout of the short exposure frame are repeatedly output in synchronization with the short-term vertical synchronization signal VDS. For this reason, the reading of the output signal from the image sensor is leveled compared to the operation of reading all lines at once after the long exposure.

Next, exposure control using an electronic shutter will be described.
First, in the long-time exposure frame Fr_L in the first long-term vertical synchronization period Tfrl1, the 01-line pixels are reset and the 01-line exposure is started in synchronization with the horizontal synchronization signal HD at the timing t31. The reset operation at this time is indicated by a broken line 310. Subsequently, each time 1HD elapses, the lines 02, 05, 06, 09, 10, 13, and 14 are reset to start the line-sequential rolling shutter operation of the long exposure frame Fr_L. The reset operation at this time is also indicated by broken lines.

  Next, at t36 after Tfrl_exp has elapsed from timing t31, the 01 line is read in synchronization with VDL. In the continuous shooting, after timing t36, the process returns to t30 and repeats, but here, it is continuously displayed until the operation of the long exposure frame Fr_L is completed. At this time, Tfrl_exp serving as the exposure time is 21 HD. Subsequently, every time 1 HD elapses, the lines 02, 05, 06, 09, 10, 13, and 14 are read out to complete the line sequential rolling shutter operation of the long exposure frame Fr_L. As a result, the long exposure frame Fr_L outputs a signal having the exposure time Tfrl_exp for all lines. The output signal of the long exposure frame Fr_L at this time becomes Fr_L Readout4 in the fourth short-term vertical synchronization period Tfrs4.

  The above is the operation of the exposure control using the electronic shutter that is performed in the first long-term vertical synchronization period Tfrl1.

  In the short exposure frame Fr_S in the first long-term vertical synchronization period Tfrl1, at the timing t33, the 03-line pixels are reset in synchronization with the horizontal synchronization signal HD, and the 03-line exposure is started. The reset operation at this time is indicated by a broken line 320. Subsequently, each time 1HD elapses, the 04, 07, 08, 11, 12, 15, and 16 lines are reset to start the line-sequential rolling shutter operation of the short exposure frame Fr_S. The reset operation at this time is also indicated by broken lines.

  Next, at time t34 after the elapse of Tfrs_exp from timing t33, reading of the 03 line is performed in synchronization with VDS. At this time, Tfrs_exp serving as the exposure time is 7HD. Subsequently, every time 1 HD elapses, the lines 04, 07, 08, 11, 12, 15, and 16 are read to finish the line sequential rolling shutter operation of the short exposure frame Fr_S. As a result, the short exposure frame Fr_S outputs a signal having the exposure time Tfrs_exp on all lines. The output signal of the short exposure frame Fr_S at this time becomes Fr_S Readout3 in the third short-term vertical synchronization period Tfrs3 of the short exposure frame.

  In the middle time exposure frame Fr_M, at the timing t35, the pixels on the 03 line are reset in synchronization with the horizontal synchronization signal HD, and the exposure on the 03 line is started. The reset operation at this time is indicated by a broken line 330. Subsequently, each time 1 HD elapses, the 04, 07, 08, 11, 12, 15, and 16 lines are reset to start the line-sequential rolling shutter operation of the medium time exposure frame Fr_M. The reset operation at this time is also indicated by broken lines. Next, at time t38 after Tfrm_exp elapses from timing t35, reading of the 03 line is performed in synchronization with VDS. In the continuous shooting, after timing t36, the process returns to t30 and repeats, but here, it is continuously displayed until the operation of the medium time exposure frame Fr_M is completed. At this time, Tfrm_exp serving as the exposure time is 14HD. Subsequently, each time 1HD elapses, the lines 04, 07, 08, 11, 12, 15, and 16 are read out to complete the line sequential rolling shutter operation of the medium time exposure frame Fr_M. As a result, the medium time exposure frame Fr_M outputs a signal having the exposure time Tfrm_exp on all lines. The output signal of the medium-time exposure frame Fr_M at this time becomes Fr_M Readout5 in the fifth short-term vertical synchronization period Tfrs5 of the short-time exposure frame.

  The above is the exposure control operation using the electronic shutter, which is performed in synchronization with the long-term vertical synchronization signal VDL.

  Here, the ratio of the exposure time Tfrs_exp for the short exposure frame, the exposure time Tfrm_exp for the medium exposure frame, and the exposure time Tfrl_exp for the long exposure frame is 7HD: 14HD: 21HD = 1: 2: 3. By these electronic shutter operations, the exposure time ratio of the short exposure frame, the medium exposure frame, and the long exposure frame is controlled to be 1: 2: 3. Therefore, the gain value for compensating the sensitivity ratio is set to 3 times, 3/2 times, and 1 time, respectively, and the dynamic exposure is performed by synthesizing the short exposure pixel signal, the medium exposure pixel signal, and the long exposure pixel signal. A range expansion process may be executed.

  FIG. 5 is a diagram showing control timings of the long exposure frame, the medium exposure frame, and the short exposure frame according to the present embodiment. In FIG. 5, the operation for each frame in FIG. 4 is shown in a simplified manner. With reference to FIG. 5, the case where the exposure time by an electronic shutter is changed will be described. The same operations and configurations as those in FIG. 4 are denoted by the same reference numerals.

  First, in the long exposure frame Fr_L, at timing t51, the exposure is started by resetting the pixels in order from the 01th line, and the rolling shutter operation is started. The reset operation at this time is indicated by a broken line 410. Next, at time t55 after the elapse of Tfrl_expl from timing t51, pixel signals are read line by line from line 01, and the rolling shutter operation ends. At this time, Tfrl_expl as an exposure time is 12 HD. The output signal of the long exposure frame Fr_L at this time is Fr_L Readout4.

  Further, in the short exposure frame Fr_S, at the timing t52, the pixels of the 03 line are reset, the pixels are sequentially reset from the 03 line to start the exposure, and the rolling shutter operation is started. The reset operation at this time is indicated by a broken line 420. Next, at t53 after the elapse of Tfrs_expl from timing t52, the pixel signal is read out line by line from line 03, and the rolling shutter operation ends. At this time, Tfrs_expl as an exposure time is 4HD. The output signal of the short exposure frame Fr_S at this time is Fr_S Readout3.

  Then, in the medium time exposure frame Fr_M, at the timing t54, the pixels of the 03 line are reset, the pixels are sequentially reset from the 03 line to start the exposure, and the rolling shutter operation is started. The reset operation at this time is indicated by a broken line 430. Next, at t56 after the elapse of Tfrm_expl from timing t54, the pixel signal is read out line by line from line 03, and the rolling shutter operation ends. At this time, Tfrm_exp1, which is the exposure time, is 8 HD. At this time, the output signal of the medium time exposure frame Fr_M is Fr_M Readout5.

  The above is the operation before changing the exposure time. Here, the ratio of the exposure time Tfrs_expl of the short exposure frame, the exposure time Tfrm_expl of the medium exposure frame, and the exposure time Tfrl_exp1 of the long exposure frame is 4HD: 8HD: 12HD = 1: 2: 3.

  Subsequently, in a reset operation 440 starting from timing t58, a rolling shutter operation for the long exposure frame Fr_L is started. Next, the rolling shutter operation is ended in the reading operation starting from t59 after the elapse of Tfrl_exps from the timing t58. At this time, Tfrl_exps as an exposure time is 6HD, and the output signal is Fr_L Readout7.

  Further, in the reset operation 450 starting from the timing t57, the rolling shutter operation of the short exposure frame Fr_S is started. Next, the rolling shutter operation is ended in the read operation starting from t58 after Tfrs_exps has elapsed from timing t57. At this time, the exposure time Tfrs_exps is 2HD, and the output signal is Fr_S Readout6.

  Then, in the reset operation 460 starting from the timing t60, the rolling shutter operation of the medium time exposure frame Fr_M is started. Next, the rolling shutter operation is terminated in the read operation starting from t61 after Tfrm_exps has elapsed from timing t60. At this time, the exposure time Tfrm_exps is 4HD, and the output signal is Fr_M Readout8.

  The above is the operation after changing the exposure time. Here, the ratio of the exposure time Tfrs_exps for the short exposure frame, the exposure time Tfrm_exps for the medium exposure frame, and the exposure time Tfrl_exps for the long exposure frame is 2HD: 4HD: 6HD = 1: 2: 3.

  Thus, even when the exposure time is changed by the electronic shutter, the ratio of the exposure times of the short exposure frame, the medium exposure frame, and the long exposure frame is always controlled to be 1: 2: 3. be able to. As a result, the gain value for compensating the sensitivity ratio is set to 3 times, 3/2 times, and 1 time, respectively, and the signal of the short exposure pixel, the signal of the medium exposure pixel, and the signal of the long exposure pixel are synthesized dynamically. Range expansion processing can be executed.

Next, the photographing operation and the HDR process in this embodiment will be described.
The HDR processing is performed in the signal processing unit 13 in FIG. In the signal processing unit 13, the pixel signal output from the image sensor 12 is subjected to pixel signal correction processing, positional deviation correction processing, HDR processing, and image signal processing, whereby an image with an expanded dynamic range is displayed. Generate. Here, a case will be described in which the long exposure frame output signal Fr_L Readout1, the medium exposure frame output signal Fr_M Readout2, and the short exposure frame output signal Fr_S Readout3 in FIG. 5 are used.

  First, pixel signal correction processing is performed on the output signal Fr_L Readout1 of the long exposure frame output in synchronization with the VDS. As pixel signal correction processing, correction processing such as scratch correction, fixed pattern correction, and shading correction is performed. Next, since the positions of the long-time exposure line and the short-time exposure line are shifted every two lines, a positional shift correction process is performed. The misalignment correction processing uses the 01, 02, 05, 06, 09, 10, 13, 14 lines of the long exposure frame, and the positions of the 03, 04, 07, 08, 11, 12, 15, 16 lines. A pixel signal at the time of long exposure is obtained. For example, a method may be used in which the 03, 04 lines at the time of short exposure are interpolated using the 01, 02, 05, 06 lines of the long exposure frame. In the present embodiment, in consideration of the use of the short exposure line for both short exposure and medium time exposure and the fact that there are many main subjects on the short exposure side, the misalignment correction is performed so as to match the short exposure line. Process. Further, although only 16 lines are shown in the pixel area 200 of FIG. 2, an actual image sensor has 960 lines or more or 2160 lines or more according to the moving image shooting standard. Therefore, even if it is divided into a short exposure frame and a long exposure frame, sufficient misalignment correction can be performed. The pixel signals of 03, 04, 07, 08, 11, 12, 15, and 16 lines corresponding to the long exposure frames thus obtained are stored in a memory (not shown) provided in the signal processing unit 13. Then, similar pixel signal correction processing is performed on the output signal Fr_M Readout2 of the medium time exposure frame output in synchronization with the next VDS. Further, as part of the HDR processing, a gain value of 3/2 is applied to compensate for the ratio 2: 3 of the exposure time between the medium time exposure frame and the long time exposure frame. Since the medium-time exposure frame is a short-time exposure line, no positional deviation correction processing is required. The pixel signals of 03, 04, 07, 08, 11, 12, 15, and 16 lines corresponding to the medium-time exposure frame thus obtained are stored in a memory (not shown) provided in the signal processing unit 13.

  Further, similar pixel signal correction processing is performed on the output signal Fr_S Readout3 of the short exposure frame that is output in synchronization with the next VDS. Further, as part of the HDR process, a gain value of 3 times is applied in order to compensate the ratio 1: 3 of the exposure time between the short exposure frame and the long exposure frame.

Next, HDR processing is performed using the pixel signals shown below.
(A) Stored pixel signals of 03, 04, 07, 08, 11, 12, 15, 16 lines of the long exposure frame.
(B) Pixel signals of 03, 04, 07, 08, 11, 12, 15, and 16 lines corresponding to the stored medium time exposure frames.
(C) Pixel signals of 03, 04, 07, 08, 11, 12, 15, and 16 lines corresponding to the stored short-time exposure frames.

  A gain value that compensates for the exposure time is already applied to the pixel signal of the short exposure frame and the pixel signal of the medium exposure frame. Therefore, in the HDR processing here, the signal of the long exposure pixel, the signal of the medium exposure pixel, and the signal of the short exposure pixel are combined using the weighting coefficient corresponding to the brightness. For example, the composition method according to the brightness can be performed as follows. The sum of the weighting coefficient of the signal of the long-time exposure pixel, the weighting coefficient of the signal of the medium-time exposure pixel, and the weighting coefficient of the signal of the short-time exposure pixel is set to a constant value 1. When the image is bright, the weighting coefficient of the signal of the short-time exposure pixel is increased, and when the image is dark, the weighting coefficient of the signal of the long-time exposure pixel is increased.

  Up to this point, since signal processing is performed for each pixel, finally, image signal processing is performed on an image whose dynamic range has been expanded. As the image signal processing, signal processing such as white balance adjustment processing, color correction processing, and gamma correction processing is performed. In this way, when a signal of a long exposure frame is output, pixel signal correction processing and positional deviation correction processing for the long exposure frame are performed. When outputting the signal of the medium time exposure frame, the pixel signal correction process for the medium time exposure frame and the gain correction for compensating the exposure time are performed. When outputting a signal for a short exposure frame, pixel signal correction processing for the short exposure frame, gain correction for compensating the exposure time, HDR processing, and image signal processing are performed. In this way, various processes are distributed.

  Further, in moving image shooting, an image of the third short-term vertical synchronization period Tfrs3 is created using Fr_L Readout1, Fr_M Readout2, and Fr_S Readout3. An image of the next fourth short-term vertical synchronization period Tfrs4 is created using Fr_M Readout2, Fr_S Readout3, and Fr_L Readout4 instead of Fr_L Readout1. As described above, by performing the HDR processing using the newly output image signal, it is possible to improve the moving image resolution at the time of moving image shooting.

  In addition, when creating a still image from the output signals of three consecutive exposure frames, an output signal that is output in the order of a short exposure frame, a long exposure frame, and a medium exposure frame with a large overlap of exposure times can be used. Good. For example, the output signals Fr_S Readout3, Fr_L Readout4, and Fr_M Readout5 or the output signals Fr_S Readout6, Fr_L Readout7, and Fr_M Readout8 may be used. As a result, a still image with an expanded dynamic range with less blur can be created.

  As described above, according to the present embodiment, an imaging device that performs HDR processing using signals of pixels having different exposure times for every two adjacent lines is provided. In synchronism with the frame period during the short exposure, the short exposure line, the medium exposure line, and the long exposure line are read out in separate frames. At the same time, the short-time exposure line and the medium-time exposure line are read in separate frames in synchronization with the frame period at the time of short-time exposure within the frame period at the time of long-time exposure. Thereby, simplification of the reading control system and reduction and leveling of the data rate during reading and signal processing can be realized.

  Further, the vertical synchronization period of the long exposure frame is three times the vertical synchronization period of the short exposure frame in accordance with the ratio 1: 2: 3 of the exposure time of the short exposure frame, the medium exposure frame, and the long exposure frame. Set to. At the same time, medium-time exposure is performed on the short-time exposure line during the short-time exposure frame. As a result, the exposure time of the short exposure frame and the exposure time of the medium exposure frame can be effectively used with respect to the exposure time of the long exposure frame, and the complexity of the exposure time control can be avoided. .

  Further, by reading out the short exposure line in synchronization with the frame period during the short exposure within the frame period during the long exposure, the long exposure and the short exposure can be performed in an overlapping manner. For this reason, even when the frame rate is made uniform, long-time exposure exceeding one frame at the time of short-time exposure is realized. In addition, there is an effect of avoiding the generation of dead time that does not contribute to the exposure seen in the multiple exposure method.

  Then, by reading out the short time exposure line and the long time exposure line in separate frames, the occurrence of rolling distortion can be reduced to half.

  Further, it is possible to execute a long exposure and a short exposure in an overlapping manner. Therefore, it is possible to avoid the problem that the timing control of reset and readout becomes complicated in order to align the centers of the exposure times of the long exposure and the short exposure as in the conventional example.

  Furthermore, the short exposure line and the long exposure line can be read out in separate frames. Therefore, there is also a problem that when the exposure time of the long exposure and the short exposure is aligned, the end of reading the short exposure line and the start of reading the long exposure line interfere when the exposure time is short. Can be avoided.

Next, a modification of this embodiment will be described.
In the above-described misalignment correction processing, the long exposure lines 01, 02, 05, 06, 09, 10, 13, 14 are used, and 03, 04, 07, 08, 11, corresponding to the short exposure lines are used. The pixel signals of 12, 15, and 16 lines were calculated. Instead, all the lines from the 01 line to the 16 lines may be used as the long exposure frame. At that time, using the 01, 02, 05, 06, 09, 10, 13, 14 lines of the long exposure frame, the 03, 04, 07, 08, 11, 12, 15, 16 lines corresponding to the long exposure frame are used. Is calculated. Thereby, all lines from 01 line to 16 lines are used as the long exposure frame.

  Similarly, the 03, 04, 07, 08, 11, 12, 15, 16 lines of the short exposure lines for the medium time exposure and the short exposure are used to correspond to the short exposure lines for the medium time exposure and the short exposure. The pixel signals of the 01, 02, 05, 06, 09, 10, 13, and 14 lines are calculated. All lines from 01 to 16 lines are used as the medium time exposure frame and the short time exposure frame.

  As a result, an image with an expanded dynamic range using all 16 × 16 pixels can be generated. This method is suitable when a still image is created from the output signals of three consecutive exposure frames.

Next, another modification of the present embodiment will be described.
In the image signal processing described above, signal processing such as white balance adjustment processing, color correction processing, and gamma correction processing is performed after the HDR processing, but the image signal processing may be distributed. At that time, white balance adjustment processing and color correction processing are performed on the output signal (first image data) of the long-exposure frame after the positional deviation correction processing is performed and stored in the memory. Similarly, the white balance adjustment process and the color correction process are performed on the output signal (second image data) of the medium time exposure frame after the gain value for compensating the exposure time is applied and stored in the memory. . Then, a white balance adjustment process and a color correction process are performed on the output signal (third image data) of the short-time exposure frame subjected to the pixel signal correction process. Thereafter, HDR processing is performed, and gamma correction processing is performed on the output signal (fourth image data).

Next, still another modification of the present embodiment will be described with reference to FIG.
FIG. 8A is a diagram of pixel characteristics that represents the relationship between the imaging surface illuminance Epix of the pixel region 200 and the pixel output signal Psig. Here, the pixel characteristics Exp1, Exp2, Exp3, and Exp4 are controlled such that the exposure frame period is Tfrs, 2Tfrs, 3Tfrs, and 4Tfrs, respectively, and the exposure time ratio by the electronic shutter is 1: 2: 3: 4. Yes. FIG. 8A shows the relationship between the imaging surface illuminance Epix and the pixel output signal Psig in this case. Accordingly, when the pixel characteristic Exp1 is used as a reference, the gradients of the pixel characteristics Exp2, Exp3, and Exp4 are doubled, tripled, and quadrupled, respectively. PSAT indicates the saturation signal amount of the pixel.

  Since the pixel characteristics Exp1, Exp2, Exp3, and Exp4 reach the saturation signal amount PSAT at the imaging surface illuminances E1, E2, E3, and E4, respectively, the output signal does not increase at higher illuminance. The broken line indicates the case where it is assumed that the pixel characteristics Exp1, Exp2, Exp3, and Exp4 are not saturated.

  In the present embodiment, the pixel characteristics Exp1, Exp2, and Exp3 correspond to the output signal of the short exposure frame, the output signal of the intermediate exposure frame, the output signal of the intermediate exposure frame, and the output signal of the long exposure frame, respectively. .

  FIG. 8B is a diagram of image signal characteristics representing the relationship between the imaging surface illuminance Eplx and the image signal Ssig used for image signal processing for the pixel characteristic Exp1. In FIG. 8A, since the pixel is saturated at the illuminance E1 on the imaging surface, the image signal is also saturated at SSAT. Then, it can be seen that the image signal is output with gradation from 0 to SSAT corresponding to the illuminance of the imaging surface from 0 to E1. This is a pixel characteristic when HDR processing is not performed.

  FIG. 8C is a diagram of image signal characteristics representing the relationship between the imaging surface illuminance Epix and the image signal Ssig used for image signal processing when HDR processing is performed using the pixel characteristics Exp1 and Exp2. The HDR processing method is realized by adding the pixel characteristics Exp1 and Exp2 and normalizing the maximum saturation signal to be Ssig. For imaging surface illuminance 0 to E2, pixel characteristics Exp1 and Exp2 in FIG. 8A are added. From the imaging surface illuminance E2 to E1, since the pixel characteristic Exp2 is saturated in FIG. 8A, PSAT and the pixel characteristic Exp1 are added. Since the pixel characteristics Exp1 and Exp2 are both saturated in FIG. 8A, the imaging surface illuminance E1 or higher is 2PSAT, which is the maximum saturation signal amount. The HDR processing of FIG. 8C is realized by normalizing the added pixel characteristics so that the maximum saturation signal amount 2PSAT becomes SSAT. Accordingly, it can be seen that the image signal is output with gradation from 0 to SSAT corresponding to the imaging surface illuminance of 0 to E1. Furthermore, since the maximum saturation signal amount 2PSAT is normalized so as to be SSAT, it can be seen that the dynamic range is doubled.

  FIG. 8D shows the imaging surface illuminance Epix and the image signal Ssig used for the image signal processing when the HDR processing is performed using the pixel characteristics Exp1, Exp2, and Exp3 for the imaging device of the present embodiment. It is a figure of the image signal characteristic showing the relationship. The HDR processing method is realized by adding the pixel characteristics Exp1, Exp2, and Exp3 and normalizing the maximum saturation signal to be Ssig. For imaging surface illuminances from 0 to E3, pixel characteristics Exp1, Exp2, and Exp3 in FIG. 8A are added. Since the pixel characteristic Exp3 is saturated in FIG. 8A from the imaging surface illuminance E3 to E2, PSAT and the pixel characteristics Exp1 and Exp2 are added. Since the pixel characteristics Exp2 and Exp3 are both saturated in FIG. 8A from the imaging surface illuminance E2 to E1, 2PSAT and the pixel characteristic Exp1 are added. Since the pixel characteristics Exp1, Exp2, and Exp3 are all saturated in FIG. 8A, the imaging surface illuminance E1 or higher is 3PSAT, which is the maximum saturation signal amount. The HDR process shown in FIG. 8D is realized by normalizing the added pixel characteristics so that the maximum saturation signal amount 3PSAT becomes SSAT. Then, the image pickup device 12 may be controlled to operate as shown in FIG. 4 and to perform the HDR processing in the signal processing unit 13 as shown in FIG. 8D. Accordingly, it can be seen that the image signal is output with gradation from 0 to SSAT corresponding to the imaging surface illuminance of 0 to E1. Furthermore, since the maximum saturation signal amount 3PSAT is normalized so as to become SSAT, it can be seen that the dynamic range is expanded three times. FIG. 8D shows input / output characteristics known as gamma characteristics, which can expand the dynamic range.

<Second Embodiment>
Next, a second embodiment will be described with reference to FIGS. 1, 2, and 6 to 8. In the present embodiment, the basic configuration and operation of the imaging apparatus and the basic configuration and operation of the imaging element are the same as those in the first embodiment, and therefore, description will be made with reference to the drawings and symbols. In the first embodiment, an image with a wide dynamic range is generated by setting two frames, a long exposure line and a short exposure line, which have different exposure times every two lines, and a medium exposure frame for the short exposure line. did. In the present embodiment, a method for changing the input / output characteristics of the HDR processing by changing the frame periods of the long exposure frame and the medium exposure frame will be described.

  FIG. 6 is a diagram showing control timings of the long exposure frame, the medium exposure frame, and the short exposure frame according to the present embodiment. In FIG. 6, since the long-term vertical synchronization period Tfrl is four times as long as the short-term vertical synchronization period Tfrs, it is possible to have HDR input / output characteristics different from those of the first embodiment. In FIG. 6, the control timing is such that continuous shooting is possible in order to apply to moving image shooting. In order to enable continuous shooting, the control timing after timing t77 or the control timing after timing t84 is returned to t70 and repeated.

  In FIG. 6, VDL represents a vertical synchronization signal (long-term vertical synchronization signal) of a long exposure frame, and VDS represents a vertical synchronization signal (short-term vertical synchronization signal) of a short exposure frame. HD indicates a horizontal synchronization signal, which is a period in which pixel reset operations and readout operations are performed in units of rows.

Timings t70 to t77 and timings t77 to t84 indicate the first long-term vertical synchronization period Tfrl1 and the second long-term vertical synchronization period Tfrl2, respectively.
The period from timing t70 to t71 is the first short-term vertical synchronization period Tfrs1.
The timing t71 to t73 is the second short-term vertical synchronization period Tfrs2.
The timing t73 to t75 is the third short-term vertical synchronization period Tfrs3.
The timing t75 to t77 is the fourth short-term vertical synchronization period Tfrs4.
The timing from t77 to t78 is the fifth short-term vertical synchronization period Tfrs5.
From the timing t78 to t80 is the sixth short-term vertical synchronization period Tfrs6.
The timing t80 to t82 is the seventh short-term vertical synchronization period Tfrs7.
The timing t82 to t84 is the eighth short-term vertical synchronization period Tfrs8.

  Fr_L indicates the operation state of the 01, 02, 05, 06, 09, 10, 13, and 14 lines constituting the long exposure frame. Fr_M and Fr_S indicate the operation states of the 03, 04, 07, 08, 11, 12, 15, and 16 lines, which are the short-time exposure lines constituting the medium-time exposure frame and the short-time exposure frame, respectively. Thereby, similarly to FIG. 3, the long exposure line and the short exposure line can be controlled so that the exposure time differs every two lines.

  Here, in the long exposure frame Fr_L, the rolling shutter operation is continuously performed every Tfrl in the long vertical synchronization period in synchronization with the VDL. In the short-time exposure frame Fr_S, the exposure of the rolling shutter operation is started in synchronization with VDS after 2 Tfrs from VDL, and the reading of the rolling shutter operation is started in synchronization with VDS after Tfrs. Further, in Fr_M, the exposure of the rolling shutter operation is started in synchronization with the VDS of the short-time exposure frame 3Tfrs after VDL, which is the free time of Fr_S, and the rolling shutter operation is read in synchronization with the VDS after 2Tfrs. Be started. This operation is repeatedly performed in synchronization with the VDL.

  Specifically, in Tfrs1 constituting Tfrl1, an output signal Fr_L Readout1 of a long exposure frame in which exposure is started one round before is output. Next, in Tfrs2, an output signal Fr_M Readout2 of a medium time exposure frame in which exposure is started one round before is output. In the present embodiment, since Tfrl is set to four times Tfrs, there is no signal output at Tfrs3.

  Next, in Tfrs4, an output signal Fr_S Readout4 of the short exposure frame is output. Similarly, an output signal Fr_L Readout5 for the long exposure frame is output at Tfrs5 constituting Tfrl2. Next, in Tfrs6, an output signal Fr_M Readout6 of the medium time exposure frame is output.

  No signal is output at Tfrs7, and an output signal Fr_S Readout8 for a short exposure frame is output at the next Tfrs8. Thereby, Fr_L Readout, Fr_M Readout, and Fr_S Readout are repeatedly output in synchronization with VDS. For this reason, the reading of the output signal from the image sensor is leveled compared to the operation of reading all lines at once after the long exposure.

Next, exposure control using an electronic shutter will be described.
First, in the long exposure frame Fr_L in Tfrl1, the 01-line pixels are reset and the 01-line exposure is started in synchronization with the horizontal synchronization signal HD at a timing t72. The reset operation at this time is indicated by a broken line 510. Subsequently, each time 1HD elapses, the lines 02, 05, 06, 09, 10, 13, and 14 are reset to start the line-sequential rolling shutter operation of the long exposure frame Fr_L. The reset operation at this time is also indicated by a broken line.

  Next, at t77 after Tfrl_exp has elapsed from timing t72, the 01 line is read in synchronization with VDL. In the continuous shooting, after timing t77, the process returns to t70 and repeats, but here, it is continuously displayed until the operation of the long exposure frame Fr_L is completed. At this time, Tfrl_exp serving as the exposure time is 16 HD. Subsequently, every time 1 HD elapses, the lines 02, 05, 06, 09, 10, 13, and 14 are read out to complete the line sequential rolling shutter operation of the long exposure frame Fr_L. As a result, the long exposure frame Fr_L outputs a signal having the exposure time Tfrl_exp for all lines. The output signal of the long exposure frame Fr_L at this time becomes Fr_L Readout5 at Tfrs5 of the short exposure frame. The above is the operation of the exposure control using the electronic shutter performed at Tfr1 of the long exposure frame.

  In the short-time exposure frame Fr_S in Tfrl1, the 03-line pixels are reset and the 03-line exposure is started in synchronization with the horizontal synchronization signal HD at the timing t74. The reset operation at this time is indicated by a broken line 520. Subsequently, each time 1HD elapses, the 04, 07, 08, 11, 12, 15, and 16 lines are reset to start the line-sequential rolling shutter operation of the short exposure frame Fr_S. The reset operation at this time is also indicated by a broken line. Next, at time t75 after the elapse of Tfrs_exp from timing t74, reading of the 03 line is performed in synchronization with VDS. At this time, Tfrs_exp serving as the exposure time is 4HD. Subsequently, every time 1 HD elapses, the lines 04, 07, 08, 11, 12, 15, and 16 are read to finish the line sequential rolling shutter operation of the short exposure frame Fr_S. As a result, the short exposure frame Fr_S outputs a signal having the exposure time Tfrs_exp on all lines. The output signal of the short exposure frame Fr_S at this time is Fr_S Readout4 in Tfrs4 of the short exposure frame.

  In the middle time exposure frame Fr_M, at the timing t76, the pixels on the 03 line are reset in synchronization with the horizontal synchronization signal HD, and the exposure on the 03 line is started. The reset operation at this time is indicated by a broken line 530. Subsequently, each time 1 HD elapses, the 04, 07, 08, 11, 12, 15, and 16 lines are reset to start the line-sequential rolling shutter operation of the medium time exposure frame Fr_M. The reset operation at this time is also indicated by a broken line. Next, at t78 after Tfrm_exp elapses from timing t76, the 03 lines are read in synchronization with VDS. In continuous shooting, after timing t77, the process returns to t70 and repeats, but here, it is continuously displayed until the operation of the medium-time exposure frame Fr_M is completed. At this time, Tfrm_exp serving as the exposure time is 8 HD. Subsequently, each time 1HD elapses, the lines 04, 07, 08, 11, 12, 15, and 16 are read out to complete the line sequential rolling shutter operation of the medium time exposure frame Fr_M. As a result, the medium time exposure frame Fr_M outputs a signal having the exposure time Tfrm_exp on all lines. The output signal of the medium time exposure frame Fr_M at this time becomes Fr_M Readout 6 at Tfrs 6 of the short time exposure frame.

  The above is the exposure control operation using the electronic shutter, which is performed in synchronization with the long-term vertical synchronization signal VDL. Here, the ratio of the exposure time Tfrs_exp for the short exposure frame, the exposure time Tfrm_exp for the medium exposure frame, and the exposure time Tfrl_exp for the long exposure frame is 4HD: 8HD: 16HD = 1: 2: 4. By these electronic shutter operations, the exposure time ratio of the short exposure frame, the medium exposure frame, and the long exposure frame is controlled to be 1: 2: 4. For this reason, the gain value for compensating the sensitivity ratio is set to 4 times, 2 times, and 1 time, respectively, and the signal of the short exposure pixel, the signal of the medium exposure pixel, and the signal of the long exposure pixel are synthesized to The enlargement process may be executed. Thereby, it is possible to provide input / output characteristics of HDR processing different from those of the first embodiment. Regarding the shooting operation and the HDR processing in the present embodiment, the gain values for compensating the sensitivity ratio of the short exposure frame, the medium exposure frame, and the long exposure frame are set to 4 times, 2 times, and 1 time, respectively. Since it may be carried out similarly to the embodiment, the description is omitted.

  In addition, in the misalignment correction process, an image with an expanded dynamic range using 16 × 16 all pixels can be generated in the same manner as in the modification of the first embodiment. Further, the image signal processing can be distributed as in the other modified examples of the first embodiment.

  As described above, in the present embodiment, the short-time exposure line, the medium-time exposure line, and the long-time exposure line are read in separate frames in synchronization with the frame period at the time of short-time exposure. Thereby, simplification of the reading control system and reduction and leveling of the data rate during reading and signal processing can be realized. In addition, other effects similar to those of the first embodiment can be obtained.

Next, a modification of this embodiment will be described.
FIG. 7 is a diagram illustrating a modification of the control timing of the long exposure frame, the medium exposure frame, and the short exposure frame. In FIG. 7, the long-term vertical synchronization period Tfrl is four times the short-term vertical synchronization period Tfrs. Further, the medium-term vertical synchronization period Tfrm is three times the short-term vertical synchronization period Tfrs. For this reason, the input / output characteristics of the HDR processing different from the above example can be provided. Further, in FIG. 7, the control timing is such that continuous shooting is possible in order to apply to moving image shooting. In order to enable continuous shooting, the control timing after timing t96 or the control timing after timing t102 is returned to t90 and repeated. The same operations and configurations as those in FIG. 6 are denoted by the same reference numerals.

In FIG. 7, timings t90 to t96 and timings t96 to t102 indicate the first long-term vertical synchronization period Tfrl1 and the second long-term vertical synchronization period Tfrl2, respectively.
The timing t90 to t91 is the first short-term vertical synchronization period Tfrs1.
The timing t91 to t93 is the second short-term vertical synchronization period Tfrs2.
The timing t93 to t95 is the third short-term vertical synchronization period Tfrs3.
The timing t95 to t96 is the fourth short-term vertical synchronization period Tfrs4.
The timing t96 to t97 is the fifth short-term vertical synchronization period Tfrs5.
The period from timing t97 to t99 is the sixth short-term vertical synchronization period Tfrs6.
The timing t99 to t101 is the seventh short-term vertical synchronization period Tfrs7.
The timing t101 to t102 is the eighth short-term vertical synchronization period Tfrs8.

  Here, it is assumed that the long exposure frame Fr_L and the short exposure frame Fr_S perform the same rolling shutter operation as in FIG. In the medium time exposure frame Fr_M, the exposure of the rolling shutter operation is started in synchronization with the VDS after 3 Tfrs from the VDL which is the free time of the short exposure frame Fr_S, and the rolling shutter operation is read in synchronization with the VDS after 3 Tfrs. Is started. This operation is repeatedly performed in synchronization with the VDL.

  Specifically, in Tfrs1 constituting Tfrl1, an output signal Fr_L Readout1 of a long exposure frame in which exposure is started one round before is output. In this modification, Tfrl is set to 4 times Tfrs, and Tfrm is set to 3 times Tfrs, so there is no signal output at Tfrs2. Then, in Tfrs3, an output signal Fr_M Readout3 of the medium time exposure frame in which the exposure is started one round before is output. Next, in Tfrs4, an output signal Fr_S Readout4 of the short exposure frame is output.

  Similarly, an output signal Fr_L Readout5 for the long exposure frame is output at Tfrs5 constituting Tfrl2. In Tfrs6, there is no signal output, and in Tfrs7, an output signal Fr_M Readout7 of the medium time exposure frame is output. Next, at Tfrs8, an output signal Fr_S Readout8 of the short exposure frame is output. This operation is repeatedly performed in synchronization with the VDL. Thus, in synchronization with VDS, the output signal Fr_L Readout of the long exposure frame, the output signal Fr_M Readout of the medium exposure frame, and the output signal Fr_S Readout of the short exposure frame are repeatedly output. For this reason, the reading of the output signal from the image sensor is leveled compared to the operation of reading all lines at once after the long exposure.

  Further, in this modification, Tfrl is set to 4 times Tfrs, so the free time of the short exposure frame Fr_S is 3 times Tfrs. Therefore, in this modification, Tfrm is set to be three times Tfrs, so that the dead time of the short exposure line that does not contribute to exposure is eliminated.

Next, exposure control using an electronic shutter will be described.
First, in the long exposure frame Fr_L at Tfrl1, the pixels of the long exposure line are reset line by line in synchronization with the horizontal synchronization signal HD at timing t92, and exposure is started. Thereby, the line sequential rolling shutter operation of the long exposure frame Fr_L is started. The reset operation at this time is indicated by a broken line 610.

  Next, the pixels of the long-time exposure line are read out line-sequentially in synchronization with the VDL at t96 after Tfrl_exp has elapsed from timing t92. Thereby, the line sequential rolling shutter operation of the long exposure frame Fr_L is completed. At this time, Tfrl_exp serving as the exposure time is 16 HD. Then, the output signal of the long exposure frame Fr_L becomes Fr_L Readout5 at Tfrs5.

  Next, in the short exposure frame Fr_S in Tfrl1, the pixels of the short exposure line are reset in line sequence in synchronization with the horizontal synchronization signal HD at timing t94, and exposure is started. Thereby, the line sequential rolling shutter operation of the short exposure frame Fr_S is started. The reset operation at this time is indicated by a broken line 620. Next, the pixels of the short-time exposure line are read out line-sequentially in synchronization with VDS at t95 after Tfrs_exp has elapsed from timing t94. Thereby, the line-sequential rolling shutter operation of the short exposure frame Fr_S is completed. At this time, Tfrs_exp serving as the exposure time is 4HD. Then, the output signal of the short-time exposure frame Fr_S becomes Fr_S Readout4 in Tfrs4.

  Next, in the medium time exposure frame Fr_M, in synchronization with the horizontal synchronization signal HD at timing t96, the pixels of the medium time exposure line are reset in line sequence to start exposure. Thereby, the line sequential rolling shutter operation of the medium time exposure frame Fr_M is started. The reset operation at this time is indicated by a broken line 630. Next, the pixels of the medium-time exposure line are read out line-sequentially in synchronization with the VDS at t99 after the elapse of Tfrm_exp from the timing t96. Thereby, the line-sequential rolling shutter operation of the medium time exposure frame Fr_M is completed. At this time, Tfrm_exp serving as the exposure time is 12 HD. Then, the output signal of the medium time exposure frame Fr_M becomes Fr_M Readout 7 at Tfrs7 of the short time exposure frame.

  The above is the operation of the exposure control using the electronic shutter that is performed in synchronization with the VDL. Here, the ratio of the exposure time Tfrs_exp for the short exposure frame, the exposure time Tfrm_exp for the medium exposure frame, and the exposure time Tfrl_exp for the long exposure frame is 4HD: 12HD: 16HD = 1: 3: 4. By these electronic shutter operations, the exposure time ratio of the short-time exposure frame, the medium-time exposure frame, and the long-time exposure frame is controlled to be 1: 3: 4. For this reason, the gain value for compensating the sensitivity ratio is set to 4 times, 4/3 times, and 1 time, respectively, and the signal of the short-time exposure pixel, the signal of the medium-time exposure pixel, and the signal of the long-time exposure pixel are synthesized dynamically. A range expansion process may be executed. Thereby, the input / output characteristics of the HDR processing different from those of the first embodiment and the present embodiment can be provided.

  Next, another modification of the present embodiment will be described. The basic operation of the HDR process is the same as that of still another modification example of the first embodiment described with reference to FIG.

  FIG. 8A is a diagram of pixel characteristics that represents the relationship between the imaging surface illuminance Epix of the pixel region 200 and the pixel output signal Psig. In the present embodiment, the pixel characteristics Exp1, Exp2, and Exp4 correspond to the output signal of the short exposure frame, the output signal of the intermediate exposure frame, the output signal of the intermediate exposure frame, and the output signal of the long exposure frame, respectively. .

  FIG. 8E illustrates the imaging surface illuminance Epix and the image signal Ssig used for the image signal processing when the HDR processing is performed using the pixel characteristics Exp1, Exp2, and Exp4 for the imaging device of the present embodiment. It is a figure of the image signal characteristic showing the relationship. The HDR processing method is realized by adding the pixel characteristics Exp1, Exp2, and Exp4 and normalizing the maximum saturation signal to be Ssig.

For imaging surface illuminances from 0 to E4, pixel characteristics Exp1, Exp2, and Exp4 in FIG. 8A are added.
From the imaging surface illuminances E4 to E2, since the pixel characteristic Exp4 is saturated in FIG. 8A, PSAT and the pixel characteristics Exp1 and Exp2 are added.
Since the pixel characteristics Exp2 and Exp4 are both saturated in FIG. 8A from the imaging surface illuminance E2 to E1, 2PSAT and the pixel characteristic Exp1 are added.
Since the pixel characteristics Exp1, Exp2, and Exp4 are all saturated in FIG. 8A, the imaging surface illuminance E1 or higher is 3PSAT, which is the maximum saturation signal amount.

  The HDR processing shown in FIG. 8E is realized by normalizing the added pixel characteristics so that the maximum saturation signal amount 3PSAT becomes SSAT. Then, the image pickup device 12 may be operated as shown in FIG. 6 and the image pickup apparatus may be controlled so that the signal processing unit 13 performs the HDR processing as shown in FIG. Accordingly, it can be seen that the image signal is output with gradation from 0 to SSAT corresponding to the imaging surface illuminance of 0 to E1. Furthermore, since the maximum saturation signal amount 3PSAT is normalized so as to become SSAT, it can be seen that the dynamic range is expanded three times.

  FIG. 8F shows an image used for imaging surface illuminance Epix and image signal processing when HDR processing is performed using the pixel characteristics Exp1, Exp3, and Exp4 for the imaging apparatus according to the modification of the present embodiment. It is a figure of the image signal characteristic showing the relationship of signal Ssig. The HDR processing method is realized by adding the pixel characteristics Exp1, Exp3, and Exp4 and normalizing the maximum saturation signal to be Ssig.

For imaging surface illuminances from 0 to E4, pixel characteristics Exp1, Exp3, and Exp4 in FIG. 8A are added.
For the imaging surface illuminances E4 to E3, since the pixel characteristic Exp4 is saturated in FIG. 8A, PSAT and the pixel characteristics Exp1 and Exp3 are added.
Since the pixel characteristics Exp3 and Exp4 are both saturated in FIG. 8A from the imaging surface illuminance E3 to E1, 2PSAT and the pixel characteristic Exp1 are added.
Since the pixel characteristics Exp1, Exp3, and Exp4 are all saturated in FIG. 8A, the imaging surface illuminance E1 or higher is 3PSAT, which is the maximum saturation signal amount.

  By normalizing the pixel characteristics thus added so that the maximum saturation signal amount 3PSAT becomes SSAT, the HDR processing of FIG. 8F is realized. Then, the image sensor 12 may be operated as illustrated in FIG. 7 and the image processing apparatus may be controlled so that the signal processing unit 13 performs the HDR processing as illustrated in FIG. Accordingly, it can be seen that the image signal is output with gradation from 0 to SSAT corresponding to the imaging surface illuminance of 0 to E1. Furthermore, since the 3 maximum saturation signal amounts PSAT are normalized so as to be SSAT, it can be seen that the dynamic range is expanded three times.

  8 (e) and 8 (f) are input / output characteristics known as gamma characteristics, which can expand the dynamic range and have different HDR processing input / output characteristics. Become.

Next, still another modification of the present embodiment will be described.
Here, the case where the input / output characteristics of the HDR processing are switched in accordance with the subject will be described with reference to FIGS. 8E and 8F. First, the luminance distribution of the subject or photometric area is calculated from the digital image signal input to the signal processing unit 13 before shooting a still image or during moving image shooting. When it is determined that the calculated luminance distribution is wider than a predetermined threshold value, the image sensor 12 is operated as shown in FIG. 6, and the main luminance distribution of the subject enters the imaging surface illuminance from 0 to E2. In this way, an aperture and an electronic shutter are set. Then, the imaging apparatus is controlled so that the signal processing unit 13 performs the HDR process as shown in FIG. As a result, many gradations can be assigned to a wide imaging surface illuminance from imaging surface illuminance 0 to E2, so that a good image with a sufficient dynamic range can be captured for the subject.

  Next, when it is determined that the calculated luminance distribution is narrower than a predetermined threshold value, the image sensor 12 is operated as shown in FIG. 7, and the main luminance distribution of the subject is converted to illuminance on the imaging surface from 0 to E4. Set the aperture and electronic shutter so that Then, the imaging device is controlled so that the HDR processing is performed in the signal processing unit 13 as shown in FIG. As a result, the inclination from the imaging surface illuminance of 0 to E4 increases, and many gradations are assigned during that time, so that the dynamic range can be expanded and a good image that is shadowed can be taken.

  In this way, a good image can be obtained by performing still image shooting or moving image shooting after performing HDR processing according to the subject.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed. In this case, the program and the storage medium storing the program constitute the present invention.

Claims (11)

An image sensor in which a plurality of pixels having photoelectric conversion elements are arranged in a matrix; and
Synchronization signal generating means for generating a short-term vertical synchronization signal that is a vertical synchronization signal for short-time exposure, and a long-term vertical synchronization signal for long-time exposure in which one period is equal to a predetermined period of the short-term vertical synchronization signal;
In accordance with the long-term vertical synchronization signal, long-time exposure is performed on a long-exposure line set at a predetermined line interval of the image sensor, and in accordance with the short-term vertical synchronization signal, each other line of the image sensor is short-time. Exposure control means for performing exposure and medium time exposure;
Have
The exposure control means includes
For the long exposure line, start reading the pixel signal obtained by the long exposure in synchronization with the long vertical synchronization signal,
For the other lines, readout of the pixel signal obtained by the short-time exposure is started in synchronization with a short-term vertical synchronization signal that does not overlap with the readout period of the pixel signal of the long-time exposure line, and the long-time exposure The readout of the pixel signal obtained by the medium-time exposure is synchronized with the readout period of the pixel signal obtained by the exposure and the short-term vertical synchronization signal that does not overlap with the readout period of the pixel signal obtained by the short-time exposure. An imaging apparatus characterized by starting. The exposure control means includes
A rolling shutter operation by long exposure for the long exposure line is started in synchronization with a first long vertical synchronization signal;
A rolling shutter operation by short-time exposure for the other lines is started in synchronization with a first short-term vertical synchronization signal next to the first long-term vertical synchronization signal;
The readout of the pixel signal by the short-time exposure for the other line is started in synchronization with the second short-term vertical synchronizing signal next to the first short-term vertical synchronizing signal, and the medium time for the other line is started. Start rolling shutter operation by exposure,
Reading out the pixel signal by the long exposure for the long exposure line in synchronization with the second long vertical synchronization signal next to the first long vertical synchronization signal;
2. The readout of a pixel signal by medium-time exposure for the other lines is started in synchronization with a third short-term vertical synchronization signal next to the second long-term vertical synchronization signal. Imaging device.   The imaging apparatus according to claim 1, wherein the exposure control unit makes a ratio of exposure times in the long-time exposure, the medium-time exposure, and the short-time exposure constant in each frame.   The first image data obtained by the long-time exposure, the second image data obtained by the medium-time exposure, and the third image data obtained by the short-time exposure are combined to produce the first image data. 4. The signal processing unit according to claim 1, further comprising a signal processing unit that generates fourth image data having an expanded dynamic range with respect to the second and third image data. 5. Imaging device.   The imaging apparatus according to claim 4, wherein the signal processing unit further performs white balance adjustment processing, color correction processing, and gamma correction processing on the fourth image data.   The signal processing means further performs white balance adjustment processing and color correction processing on the first, second, and third image data, respectively, and the white balance adjustment processing and color correction processing are performed. The imaging according to claim 4, wherein the fourth image data is created by combining the first, second, and third image data, and a gamma correction process is performed on the fourth image data. apparatus.   The imaging apparatus according to claim 1, wherein a ratio of exposure times of the short time exposure, the medium time exposure, and the long time exposure is 1: 2: 3.   The imaging apparatus according to claim 1, wherein a ratio of exposure times of the short time exposure, the medium time exposure, and the long time exposure is 1: 2: 4.   The imaging apparatus according to claim 1, wherein a ratio of exposure times of the short time exposure, the medium time exposure, and the long time exposure is 1: 3: 4. An imaging device in which a plurality of pixels having photoelectric conversion elements are arranged in a matrix, a short-term vertical synchronization signal that is a vertical synchronization signal for short-time exposure, and a length equal to a predetermined period of the short-term vertical synchronization signal A control method for an imaging apparatus having synchronization signal generating means for generating a long-term vertical synchronization signal for time exposure,
The exposure control means performs long exposure with a long exposure line set at a predetermined line interval of the image sensor in accordance with the long-term vertical synchronization signal, and performs other exposure of the image sensor in accordance with the short-term vertical synchronization signal. It has an exposure control process that carries out short-time exposure and medium-time exposure respectively in the line,
The exposure control step includes
For the long exposure line, start reading the pixel signal obtained by the long exposure in synchronization with the long vertical synchronization signal,
For the other lines, readout of the pixel signal obtained by the short-time exposure is started in synchronization with a short-term vertical synchronization signal that does not overlap with the readout period of the pixel signal of the long-time exposure line, and the long-time exposure The readout of the pixel signal obtained by the medium-time exposure is synchronized with the readout period of the pixel signal obtained by the exposure and the short-term vertical synchronization signal that does not overlap with the readout period of the pixel signal obtained by the short-time exposure. A control method for an imaging apparatus, characterized by starting.   The program for making a computer perform each process of the control method of the imaging device of Claim 10.

Images