JP6700723B2 - Imaging device and driving method of imaging device - Google Patents

Description

  The present invention relates to an image pickup apparatus and a method for driving an image pickup element.

  In general, a vehicle-mounted or surveillance imaging device is required to have a wide dynamic range performance. In addition, a wide dynamic range performance is required for a digital still camera and a digital video camera. On the other hand, a method is known in which an image with a short exposure time and an image with a long exposure time are combined to generate an image with a wide dynamic range.

  On the other hand, many light sources such as LED light sources and fluorescent lamps blink in synchronization with a commercial power source. Therefore, when an image is captured with a short exposure time, a light source that appears to be illuminated by human eyes may be captured as a non-illuminated image in a still image. In addition, in a moving image, an image may be captured as if it were blinking violently.

  Patent Document 1 describes that the fluctuation detection unit that detects the energy fluctuation cycle of illumination detects the fluctuation cycle of the light source. The entire exposure period is determined in accordance with this fluctuation cycle, and this exposure period is alternately divided into a valid exposure period and an invalid exposure period at a constant rate. An image pickup apparatus is shown that acquires an image of a short exposure time that matches a fluctuation cycle of a light source by converting only an electric charge accumulated during an effective exposure period among electric charges accumulated by an imaging unit into an electric signal. There is.

JP, 2007-6049, A

  In order to realize both the expansion of the dynamic range and the suppression of the influence of the blinking of the light source, the imaging device of Patent Document 1 alternately performs imaging with a short exposure time and imaging with a long exposure time for each frame. It is possible. However, the present inventor has found that it is difficult to correctly acquire an image with such a device. When the imaging device of Patent Document 1 performs imaging with a short exposure time, it is necessary to discharge and transfer charges from the photoelectric conversion unit a plurality of times, and thus a memory such as a floating diffusion is required. The charges generated in the photoelectric conversion unit during the effective exposure period of the nth frame are intermittently transferred to the memory during the same nth frame, and then sequentially transferred from the memory to the subsequent storage unit in the (n+1)th frame. To be done. The image pickup apparatus of Patent Document 1 alternately divides one frame into a valid exposure period and an invalid exposure period at a constant rate in order to capture a short exposure time. Therefore, when imaging is performed for a short exposure time in the (n+1)th frame, the charges in the nth frame and the charges in the (n+1)th frame may overlap in the memory. As a result, it is difficult to capture images with different exposure times in successive frames. In order to obtain a single image with a wide dynamic range by synthesizing a plurality of images with different exposure times while ensuring the frame rate, it is desirable to capture and output images with different exposure times in consecutive frames. It is an object of the present invention to provide a technique for acquiring an image with a wide dynamic range in an imaging device while suppressing the influence of blinking of a light source.

In view of the above problem, an imaging device according to an embodiment of the present invention includes an imaging element and a control unit that controls the imaging element, and the imaging element has a plurality of pixels and each pixel of the plurality of pixels. Is a photoelectric conversion unit, a charge holding unit, an output unit, a first transfer unit that transfers the charges generated in the photoelectric conversion unit to the charge holding unit, and the charge held in the charge holding unit to the output unit. A second transfer unit for transferring, the control unit transfers a charge generated by the photoelectric conversion unit to the charge holding unit by the first transfer unit and holds the charge holding unit, and a holding operation for holding the charge holding unit. A transfer operation of transferring the stored charge to the output section by the second transfer section is repeatedly performed by the image pickup device, and the control section performs the holding operation to transfer the charge generated in the photoelectric conversion section during the first exposure time. In the first holding operation of performing the first sampling operation of transferring and holding the charge to the charge holding unit by the first transfer unit a plurality of times in the period of one frame, and the photoelectric conversion in the second exposure time longer than the first exposure time. A plurality of holding operations including a second holding operation of performing the second sampling operation of transferring the charges generated by the conversion unit to the charge holding unit by the first transfer unit and holding the charges once or more in one frame period. the operation, to perform the image pickup device, of the period of one frame, the period from the start of the first first sampling operation until the end of the last first sampling operation, out of the period of one frame It is characterized in that it is shorter than the period from the start of the first second sampling operation to the end of the final second sampling operation .

  By the above means, a technique for acquiring an image with a wide dynamic range is provided in the imaging device while suppressing the influence of the blinking of the light source.

FIG. 3 is a block diagram of the image pickup apparatus according to the first embodiment of the present invention. FIG. 3 is a timing diagram illustrating the operation of the image pickup apparatus in FIG. 1. FIG. 2 is an equivalent circuit diagram of a pixel portion of the image pickup device of FIG. 1. FIG. 2 is a diagram showing a configuration of an image processing unit of the image pickup apparatus of FIG. 1. FIG. 3 is a diagram illustrating a method of synthesizing images by the image pickup apparatus in FIG. 1. FIG. 6 is a timing diagram illustrating an operation of the image pickup apparatus according to the second embodiment of the present invention. FIG. 9 is a timing diagram illustrating an operation of the image pickup apparatus according to the third embodiment of the present invention. The block diagram of the imaging device which concerns on the 4th Embodiment of this invention. FIG. 9 is an equivalent circuit diagram of a pixel portion of the image pickup device in FIG. 8. FIG. 9 is a timing diagram illustrating the operation of the image pickup apparatus in FIG. 8.

  Hereinafter, specific embodiments of an image pickup apparatus according to the present invention will be described with reference to the accompanying drawings. In the following description and drawings, common reference numerals are given to common configurations across a plurality of drawings. Therefore, common configurations will be described with reference to a plurality of drawings, and descriptions of configurations with common reference numerals will be appropriately omitted.

First Embodiment With reference to FIGS. 1 to 5, a structure of an image pickup apparatus and a driving method of an image pickup element included in the image pickup apparatus according to some embodiments of the present invention will be described. FIG. 1 is a block diagram schematically showing a configuration example of an image pickup apparatus 101 according to the first embodiment of the present invention. In the present embodiment, the imaging apparatus 101 has, for example, a moving image for mounting on a vehicle or for monitoring, which includes a display unit that displays an image with an expanded dynamic range and a built-in recording medium unit that records an image with an expanded dynamic range. It is a recording camera. The image capturing apparatus 101 includes an image sensor 102, a control unit 103, an image processing unit 104, an image display unit 105, and an image recording unit 106.

  In the image sensor 102, a plurality of pixels are arranged in a two-dimensional array. Although four pixels 120 are shown in FIG. 1, a plurality of these pixels 120 are two-dimensionally arranged in an array on the image sensor 102. The pixel 120 of the image sensor 102 includes a photoelectric conversion unit 107, a reset unit 108, a first transfer unit 109, a charge holding unit 110, a second transfer unit 111, and an output unit 112. The photoelectric conversion unit 107 has a function of converting incident light into an electric signal such as an electric charge, and is a photodiode, for example. The reset unit 108 resets the potential due to the electric charge of the photoelectric conversion unit 107. The charges generated by the photoelectric conversion unit 107 are transferred to the charge holding unit 110 by the first transfer unit 109. The charges are temporarily held by the charge holding unit 110. The charges held by the charge holding unit 110 are transferred by the second transfer unit 111 to the output unit 112 that is a subsequent circuit of the image sensor 102. The output unit 112 outputs an image from the image sensor 102 to the image processing unit 104. A series of operations in which the image sensor 102 generates electric charges according to incident light and holds the electric charges in the electric charge holding unit 110 is called a sampling operation. The sampling operation of this embodiment includes a reset operation for resetting the potential of the photoelectric conversion unit 107 due to electric charges, an exposure operation for waiting for the electric charges to be accumulated in the photoelectric conversion unit 107, and an electric charge accumulated in the photoelectric conversion unit 107. Is transferred to and held in the charge holding unit 110. The time required for the exposure operation is called exposure time.

  Next, the control unit 103 will be described. The control unit 103 generates a pulse signal for driving the image sensor 102. The control unit 103 includes a sampling operation switching unit 113, a first pulse generation unit 114, and a second pulse generation unit 115. The sampling operation switching unit 113 has a function of sequentially switching, for each frame of an image, the timing related to the above-described sampling operation among the timings for supplying the pulse signal for driving the image sensor 102. The first pulse generator 114 and the second pulse generator 115 generate pulse signals for sampling operation. The pulse signals generated by the respective pulse generators are sequentially switched by the sampling operation switching unit 113 in each of a plurality of consecutive frames, and are sent from the control unit 103 to the image sensor 102.

  The first pulse generation unit 114 generates a pulse signal that causes the image sensor 102 to perform a first charge transfer operation including a first holding operation and a first transfer operation. The first holding operation is an operation of performing a sampling operation with a first exposure time that is short in exposure time a plurality of times during a period of one frame. A sampling operation with a short exposure time is called a short time sampling operation. The short time sampling operation is also called a first sampling operation. The first transfer operation is an operation of transferring the signal charge held in the charge holding portion 110 by the first holding operation to the output portion 112. By the first charge transfer operation, signal charges generated by a plurality of short-time sampling operations are superimposed on the charge holding unit 110, and then transferred to the output unit 112. That is, the image pickup device 102 outputs an image subjected to multiple exposure in a short exposure time.

  The second pulse generation unit 115 generates a pulse signal that causes the image sensor 102 to perform the second charge transfer operation including the second holding operation and the second transfer operation. The second holding operation is an operation of performing the sampling operation at the second exposure time, which has a long exposure time, once or more in one frame period. A sampling operation with a long exposure time is called a long time sampling operation. The long-time sampling operation is also called the second sampling operation. The exposure time of the long time sampling operation is longer than the exposure time of the short time sampling operation. In the present embodiment, in the second holding operation, the long-time sampling operation is performed only once in one frame period. The second transfer operation is an operation of transferring the signal charge held in the charge holding portion 110 by the second holding operation to the output portion 112. By the second charge transfer operation, the signal charge generated by one long time sampling operation is held in the charge holding unit 110, and then transferred to the output unit 112. That is, the image sensor 102 outputs an image that has been exposed for a single time with a long exposure time.

  In the present embodiment, the pulse signal for performing the first transfer operation and the pulse signal for performing the second transfer operation may be equal to each other. Therefore, the control unit 103 may have another pulse generation unit that generates this pulse signal instead of the first pulse generation unit 114 and the second pulse generation unit 115. In this case, the sampling operation switching unit 113 supplies the pulse signal generated by the other pulse generation unit to the image sensor 102.

  The pulse signal sent to the image sensor 102 by the sampling operation switching unit 113 causes the image sensor 102 to repeatedly perform the above-described charge transfer operation. Further, the first charge transfer operation and the second charge transfer operation are periodically switched and performed for each frame. For this reason, an image that has been subjected to multiple exposure with a short exposure time and an image that has been subjected to a single exposure with a long exposure time are sequentially output from the image sensor 102 periodically for each frame and sent to the image processing unit 104.

  The image processing unit 104 processes an image captured by the image sensor 102. The image processing unit 104 has a function of receiving a signal from the image sensor 102, synthesizing at least two images having different exposure times, and generating an image with an expanded dynamic range. The image processing unit 104 also has a function of converting a pixel array signal of an image sensor such as a Bayer array into a video signal such as a general RGB signal or a YCbCr signal. Further, the image processing unit 104 has an interface (IF) function of generating a display signal to the image display unit 105, an IF function of generating a recording signal to the image recording unit 106, and the like. The image display unit 105 includes, for example, a liquid crystal element or an EL element, and displays the image processed by the image processing unit 104. The image recording unit 106 is composed of, for example, a hard disk or a non-volatile memory, and records the image processed by the image processing unit 104.

  FIG. 2 is a timing diagram illustrating the operation of the image pickup apparatus 101 according to this embodiment. The period T210(n), the period T210(n+1), the period T210(n+2), the period T210(n+3), and the period T210(n+4) respectively indicate the periods of the nth, n+1, n+2, n+3, and n+4th frame. .. Each frame outputs one image from the image sensor 102 to the image processing unit 104.

  The waveform 201 is a schematic waveform obtained by full-wave rectifying a commercial AC power source of 50 Hz or 60 Hz, for example. The period T211 indicates the AC cycle of the commercial power supply, and is 1/50 seconds or 1/60 seconds, for example.

  A waveform 202 indicates a lighting state of an LED light source such as a traffic signal using the power source of the waveform 201. The "H" level indicates a lighting state, and the "L" level indicates a non-lighting state. Here, when the waveform 201 of the full-wave rectified power supply is equal to or higher than the threshold level TH, the waveform 202 is turned on, and when the waveform 201 is lower than the threshold level TH, the waveform 202 is turned off. A waveform 203 schematically shows a lighting state when the blinking phase is different from that of the waveform 202. A period T212 indicates a blinking cycle of the light source, which is a half cycle of the period T211. A period T213A and a period T213B are periods in which the light source is on, and a period T214A and a period T214B are periods in which the light source is off, respectively.

  A signal waveform 204 shows a waveform of a pulse signal output from the sampling operation switching unit 113 to the reset unit 108 for resetting the potential of the photoelectric conversion unit 107. The "H" level is reset and the "L" level is not reset. In the present embodiment, the photoelectric conversion units 107 of all the pixels 120 arranged in an array on the image sensor 102 are collectively and simultaneously reset. Therefore, the reset signals of the “H” level of the signal waveform 204 are input to the reset units 108 of all the pixels 120 at the same timing.

  The signal waveform 205 shows the waveform of the pulse signal output from the sampling operation switching unit 113 to the first transfer unit 109. When the signal waveform 205 is at “H” level, the first transfer unit 109 is turned on, and the charges generated by the photoelectric conversion unit 107 are transferred to the charge holding unit 110. Further, when the signal waveform 205 is at “L” level, the first transfer unit 109 is turned off, and the charges are not transferred from the photoelectric conversion unit 107 to the charge holding unit 110. In this embodiment, the charges generated by the photoelectric conversion unit 107 are simultaneously transferred to the charge holding unit 110 in all the pixels 120 arranged in an array. Therefore, the “H” level transfer signal of the signal waveform 205 is input to the first transfer units 109 of all the pixels 120 at the same timing. By using the reset signal to the photoelectric conversion unit 107 shown in the signal waveform 204 and the transfer signal to the first transfer unit 109 shown in the signal waveform 205, the imaging device 101 uses all the pixels 120 arranged in an array. To realize a global electronic shutter that simultaneously performs the exposure of.

  A period T215 and a period T216 each indicate a period of the above-described short-time sampling operation. A period T217 indicates a period from the start of the first sampling operation of the short-time sampling operation performed a plurality of times in one frame to the end of the last sampling operation. In this embodiment, the sampling operation is performed twice during the period T217, that is, the period T215 and the period T216. The charges generated in these two periods T215 and T216 are sampled, transferred to the charge holding unit 110, and added. As a result, multiple images are formed in the charge holding unit 110 in a short exposure time.

  A period T218 indicates a period of the above long-time sampling operation. During the period T218, the long-time sampling operation is performed, and the charge generated in this period is sampled. As a result, a single exposure image is formed in the charge holding unit 110 for a long exposure time.

  Here, the sampling operation of the short time sampling operation and the long time sampling operation is started when the reset signal shown in the signal waveform 204 from the reset unit 108 to the photoelectric conversion unit 107 is changed from the “L” level to the “H” level. To do. The sampling operation ends when the transfer signal shown in the signal waveform 205 from the sampling operation switching unit 113 to the first transfer unit 109 changes from the “H” level to the “L” level.

  The signal waveform 206, the signal waveform 207, and the signal waveform 208 are from the sampling operation switching unit 113 to the second transfer because of the first and second transfer operations of transferring the charge temporarily held in the charge holding unit 110 to the output unit 112. 7 shows a waveform of a pulse signal output to the unit 111. When the signal waveform 206 to the signal waveform 208 are at “H” level, the second transfer unit 111 is turned on, and the charges temporarily held in the charge holding unit 110 are transferred to the output unit 112. Further, when the signal waveform 206 to the signal waveform 208 are at the “L” level, the second transfer unit 111 is turned off, and the charge is not transferred from the charge holding unit 110 to the output unit 112. Here, the signal waveform 206 is a waveform of a signal for transferring the charge held in the charge holding unit 110 of each pixel 120 in the first row of the image sensor 102 to the output unit 112. Similarly, a signal waveform 207 shows a waveform of a signal for transferring the charge held in the charge holding unit 110 of each pixel 120 in the second row of the image sensor 102 to the output unit 112. A signal waveform 208 shows a waveform of a signal for transferring the charge held in the charge holding portion 110 of each pixel 120 in the last row of the image sensor to the output portion 112. A period T219 is a transfer period in which an image for one frame is sequentially output from the charge holding portion 110 to the output portion 112 in the vertical direction row by row with the signal waveform 206 to the signal waveform 208.

  As described above, the charges generated by the photoelectric conversion units 107 of the pixels 120 arranged in an array on the image sensor 102 are temporarily stored in the charge holding unit 110 by the global electronic shutter function. After that, the frame images for one screen are output by sequentially outputting the charge holding unit 110 to the output unit 112.

  The charges sampled in the period T217 of the n-th frame shown in the period T210(n) are transferred to the output portion 112 in each row during the period T219 of the period T210(n+1) of the n+1-th frame, and the image pickup element It is output from 102 to the image processing unit 104. Next, the charges sampled in the period T218 of the (n+1)th frame are transferred to the output unit 112 for each row during the period T219 of the period T210(n+2) of the (n+2)th frame, and further from the image sensor 102 to the image processing unit. It is output to 104. By the sampling operation switching unit 113 sequentially switching the sampling signal to the image sensor 102 for each frame, the image sensor 102 causes the image processing unit 104 to provide an image of a short exposure time and an image of a long exposure time to each frame. Output. Specifically, the image sensor 102 outputs an image with a short exposure time in odd-numbered frames, and outputs an image with a long exposure time in even-numbered frames. The image processing unit 104 uses the images with different exposure times to perform image composition for expanding the dynamic range.

  Here, the interval of a plurality of short-time sampling operations performed in the periods T215 and T216, that is, the interval from the start of one short-time sampling operation to the start of the next short-time sampling operation is 1 It should be half or less of the frame period. For example, the interval may be 1/2 or less of the blinking cycle of the light source for imaging. In the period T217, the blinking of the light source is sampled and added a plurality of times at a frequency twice or more the blinking of the light source. Even if one of the plurality of short-time sampling operations is performed when the light is turned off, another short-time sampling operation of the same frame is likely to be performed when the light is turned on. This increases the possibility of preventing non-detection of the lighting state even with a short exposure time.

  For example, in a frame in which a short-time sampling operation is performed, in the case of single exposure in which there is no sampling operation in the period T215 and only the sampling operation in the period T216, the period T213A in which the waveform 202 is on and the period T216 overlap. The lighting state can be detected. On the other hand, in the case of the waveform 203 having a different blinking phase from the waveform 202, the lighting state cannot be detected and the lighting of the light source may not be recognized because the sampling operation in the period T214B to be turned off and the sampling operation in the period T216 overlap. For example, when the red signal of the traffic light is lit during a bright daytime, there is a possibility that it may be erroneously detected that the signal is not lit in imaging with a short exposure time. However, even when the blinking phase of the light source is the waveform 203, such an erroneous detection is performed by performing the short-time sampling operation in the period T215 and the period T216 and adding the charges generated by the photoelectric conversion unit 107 in the charge holding unit 110. It becomes possible to prevent it.

  In this embodiment, the number of short-time sampling operations in one frame period is set to 2 and sampling is performed at an interval of 1/2 of the blinking cycle of the light source. However, for example, a larger number of times is used. Sampling may be performed. By increasing the number of short-time sampling operations, the detection accuracy of the blinking light source can be improved, and the fluctuation of the detected image can be suppressed.

  In addition, the period T217 is at least ½ of at least one frame period. For example, the period T217 may be 1/2 or more of the blinking cycle of the light source. Further, for example, the period T217 may be equal to the period in which the light source is on in the blinking cycle of the light source. In this embodiment, the period T217 is set to 1/2 of the blinking cycle of the light source. This is because, in a light source that performs full-wave rectification using a commercial power source such as a traffic signal and performs lighting control, the period during which many light sources are on is longer than the period during which they are off. Therefore, by performing sampling a plurality of times at least during a period of ½ or more of the blinking cycle of the light source, it is possible to sample a period in which the light source is turned on once. However, in a light source such as an illumination that controls the lighting ratio and dimming, for example, when the brightness is reduced, the lighting ratio of the light source blinking cycle may be smaller than 1/2. The driving method of the image pickup apparatus in this case will be described in the second embodiment.

  In the present embodiment, the period T219 in which the first and second transfer operations are performed is the same in both the frame in which the short-time sampling operation and the frame in which the long-time sampling operation is performed. Here, the period T219 is shorter than 1/2 of one frame period. Further, the period T219 is shorter than 1/2 of the period T212, which is the blinking cycle of the light source. When the period in which the charges generated in the photoelectric conversion unit 107 are transferred to the charge holding unit 110 and the period T219 in which the charges are sequentially transferred from the charge holding unit 110 to the output unit 112 overlap, the amount of charges in the charge holding unit 110 Changes during the vertical scanning of the image output. The period T219 is set to prevent this. The setting of the period T219 should be noted in the frame in which the short-time sampling operation is performed. In the present embodiment, in the frame in which the long-time sampling operation is performed, similarly to the frame in which the short-time sampling operation is performed, the charge holding unit 110 outputs to the output unit in a time shorter than 1/2 of the period T212 which is the blinking cycle of the light source. Set to transfer to. By adjusting the timing of transferring the charges from the charge holding unit 110 to the output unit 112 in all frames, it is possible to simplify the image input circuit configuration of the image processing unit 104 in the subsequent stage of the image sensor 102. This makes it possible to realize an inexpensive imaging device.

  FIG. 3 shows a configuration example of an equivalent circuit of a pixel portion of the image pickup element 102 of the image pickup apparatus 101 according to this embodiment. Although four pixels 300 are shown in FIG. 3, a large number of these pixels 300 are two-dimensionally arranged in an array in the image sensor 102. The pixel 300 corresponds to the pixel 120 in FIG. The pixel 300 includes a photoelectric conversion unit 301 corresponding to the photoelectric conversion unit 107, a charge holding unit 302 corresponding to the charge holding unit 110, a first transfer switch 304 corresponding to the first transfer unit 109, and a second transfer unit 111. And a second transfer switch 305 corresponding to Further, the pixel 300 includes an amplification unit 310 corresponding to the output unit 112, a reset transistor 309, a selection transistor 307, and an input node 303.

  The photoelectric conversion unit 301 accumulates electric charges generated by incident light. As the photoelectric conversion unit 301, for example, a photodiode is used. The first transfer switch 304 transfers the charges generated by the photoelectric conversion unit 301 to the charge holding unit 302. The charge holding unit 302 temporarily holds the charges generated by the photoelectric conversion unit 301 by the incident light. The second transfer switch 305 transfers the charge temporarily held in the charge holding unit 302 to the input node 303 of the amplification unit 310. The selection transistor 307 selects the pixel 300 that outputs a signal to the output line 308. The amplification unit 310 of the selected pixel 300 outputs a signal based on the charge generated by the incident light to the output line 308. The reset transistor 309 resets the voltage of the input node 303 of the amplification unit 310 to the potential of the reset line 326 by the signal of the control line 325. The amplification unit 310 constitutes, for example, a source follower circuit. Moreover, for example, MOS transistors are used as the first transfer switch 304 and the second transfer switch 305.

  The operation of the first transfer switch 304 is controlled by the control signal supplied through the control line 321. The drive pulse indicated by the signal waveform 205 in FIG. 2 is applied to the first transfer switch 304 from the control line 321. The operation of the second transfer switch 305 is controlled by the control signal supplied through the control line 322. In the present embodiment, the pixels 300 are arranged in a matrix, but the pixels 300 included in one row are connected to a common control line. The drive pulses of the signal waveform 206 to the signal waveform 208 shown in FIG. 2 are given from the control line 322 according to each row.

  Further, the pixel 300 has an ejection switch 318 corresponding to the reset unit 108 shown in FIG. The discharge switch 318 discharges the charge of the photoelectric conversion unit 301 to the power supply node 324 such as an overflow drain, and resets the potential due to the charge of the photoelectric conversion unit 301. The operation of the ejection switch 318 is controlled by the control signal supplied through the control line 323. The drive pulse shown in the signal waveform 204 shown in FIG. 2 is applied to the ejection switch 318 from the control line 323. With these configurations, the photoelectric conversion unit 301 can accumulate charges generated while the charge holding unit 302 temporarily holds the charges. Therefore, the imaging device 101 has a function of matching the photoelectric conversion periods in the plurality of pixels 300, that is, a so-called global electronic shutter function. The charges accumulated in the photoelectric conversion unit 301 are simultaneously discharged to a plurality of pixels 300 by the discharge switch 318 at the same time. The above operation may be performed on all pixels at the same time. It is preferable that the plurality of pixels are arrayed in at least two dimensions at the same time. In the following description, a case where all pixels are operated simultaneously will be described as an example.

  In this embodiment, the start of exposure is controlled by controlling the ejection switch 318. Specifically, by turning the discharge switch 318 from ON to OFF, the photoelectric conversion unit 301 starts to accumulate charges. This makes it possible to freely set the exposure time. By thus having the overflow drain structure and the charge holding portion 302, the charge held for a short time is transferred to the charge holding portion 302 a plurality of times during one frame to form a multiple-exposure short-time exposure image. It becomes possible to do.

  In addition, in this embodiment, the first transfer unit 109 and the reset unit 108 preferably have a function of transferring or resetting the charge generated in the photoelectric conversion unit 107 to the charge holding unit 110 without being left over. Further, the second transfer portion 111 preferably has a function of completely transferring the charge from the charge holding portion 110 to the output portion 112. This can be realized by appropriately setting the potential distribution of each element forming the circuit of the pixel 300 shown in FIG. By transferring and resetting all charges without exhaustion, it is possible to suppress transfer unevenness and reset unevenness caused by heat. As a result, the influence of thermal noise is suppressed, and good image quality can be obtained even if sampling is performed a plurality of times.

  Next, a method of synthesizing images captured by the image sensor 102 and having different exposure times will be described with reference to FIGS. 4 and 5. FIG. 4 shows a configuration example of the image processing unit 104 of the image pickup apparatus 101 according to this embodiment. The image processing unit 104 receives the image signal of each frame from the image sensor 102, synthesizes two or more images having different exposure times, and outputs a display signal to the image display unit 105 and a recording signal to the image recording unit 106. Is output. The pre-image processing unit 401 performs image processing such as black level correction on the image data input from the image sensor 102. The first image temporary storage unit 402 temporarily holds, in a frame memory or the like, image data of multiple-exposure short-time exposure obtained by a plurality of short-time sampling operations during one frame. The second image temporary storage unit 403 temporarily holds a single long-time exposure image data obtained by the long-time sampling operation during one frame in a frame memory or the like. The image combining unit 404 combines the multiple-exposure short-time exposure image input from the first image temporary storage unit 402 and the single long-exposure image input from the second image temporary storage unit 403. Then, create a composite image with an expanded dynamic range. The post-image processing unit 405 converts a pixel array signal of an image sensor such as a Bayer array into a video signal such as a general RGB signal or YCbCr signal. The post-image processing unit 405 may also perform image quality correction of the combined image, such as white balance and gamma correction. The image display IF unit 406 is an IF unit that transmits an image display signal to the image display unit 105 and transmits/receives a control signal. The image recording IF unit 407 is an IF unit that transmits an image recording signal to the image recording unit 106 and transmits and receives a control signal.

  FIG. 5 shows a method of creating a dynamic range expanded image performed by the image composition unit 404 of the image pickup apparatus 101 according to the present embodiment. Here, the short-time exposure image and the long-time exposure image that have been subjected to multiple exposure are combined, and the number of gradations of the output level is increased from 4096 gradations to 65536 gradations, and the dynamic range is changed from 72 dB to 96 dB. The case of enlarging is shown.

  FIG. 5A shows the illuminance and output level of the short-time exposure image obtained by a plurality of short-time sampling operations and the long-exposure image obtained by one long-time sampling operation before image combination. It is a figure which shows the relationship with. The characteristic 501A is an output level with respect to illuminance at the time of capturing a long-exposure image, which is mainly used for capturing a dark area of a subject. Here, the output of the image pickup device is 12 bits, and the number of output gradations is 4096 gradations. The characteristic 502A is an output level characteristic with respect to the illuminance at the time of capturing a short-time exposure image, which is mainly used for capturing the bright area of the subject. Compared with the characteristic of the long-time exposure image, the inclination of the characteristic of the short-time exposure image having a short exposure time becomes gentle. In order to simplify the description, for example, it is assumed that the exposure time of the long-time exposure is 16 times the total exposure time of the short-time exposures performed a plurality of times.

  In the present embodiment, the output level at the time of capturing a long-exposure image reaches 4095 gradations when the illuminance is 1000 lx, whereas the output level at the time of capturing a short-exposure image is 16000 lx, which is 16 times the illuminance. It is set so as to reach 4095 gradations. First, in order to combine the image captured by the long-time exposure and the image captured by the short-time exposure, an image in which the image data captured by the short-time exposure is digitally shifted to the upper side by 4 bits is created. As a result, the image data picked up by the short-time exposure is shifted to the upper side by 4 bits and becomes 16-bit data with the number of gradations of 65,536. Next, the long exposure image and the short exposure image are combined. By using the image data captured by the long-time exposure below the gradation of the output level of a certain threshold, and the image data captured by the short-time exposure at the gradation of the output level higher than the output level of the threshold, the 16-bit Generate an image. For example, the threshold value is 4095 gradations. FIG. 5B shows the output level of the gradation of the combined image with respect to the illuminance. The characteristic 501A that has been exposed for a long time is an area in which the data of the output level equal to or lower than the threshold of 4095 gradations is used as it is. A characteristic 502B is an area having an output level higher than the threshold of 4095 gradations in the shifted data by shifting the characteristic 502A of the short-time exposure image data by 4 bits to the upper side. As a result, the combined image becomes an image with an expanded dynamic range in which the saturation level is 16 times larger than that in shooting in a dark area while having a gradation accuracy of 16 bits in the dark area. In the present embodiment, the exposure time of the long-time exposure is 16 times as long as the total exposure time of the short-time exposures performed a plurality of times. However, by setting the ratio of the exposure time to a larger value, 100 dB or more can be obtained. It is possible to obtain an image having a high dynamic range.

  By using the configuration and the driving method described above, the imaging device 101, for example, lights up a bright traffic light that is a short-time exposure when capturing the lighting state of the traffic light outside the bright tunnel with an in-vehicle camera of a car running in a dark tunnel. The possibility of falsely detecting the state is suppressed. Further, the image pickup apparatus 101 can obtain a clear image having a wide dynamic range from a dark portion inside the tunnel to a bright portion outside the tunnel by combining the short-exposure image and the long-exposure image.

  As described above, in the imaging of an environment in which a dark part and a bright part coexist, which requires a wide dynamic range, when there is a blinking light source with a short cycle, an imaging device with improved detection accuracy of a light source on/off state is realized. .. Further, it is possible to realize an image pickup apparatus that suppresses flicker due to fluctuations in exposure amount due to blinking of a light source and acquires a moving image with good image quality. Further, the image pickup apparatus 101 according to the present embodiment does not require a lighting detection unit that detects blinking of the light source as disclosed in Patent Document 1. Further, the blinking phase of the light source and the exposure operation of the sensor do not have to match. Therefore, the circuit configuration becomes simple. As a result, an inexpensive imaging device is realized.

Second Embodiment With reference to FIG. 6, a method for driving an image sensor according to a second embodiment of the present invention will be described. FIG. 6 is a timing diagram illustrating the operation of the image pickup apparatus 101 according to this embodiment. In the present embodiment, as compared with the timing chart of the first embodiment of FIG. 2, instead of the signal waveform 204 from the control unit 103 to the image sensor 102 to the signal waveform 208, the signal waveform 204 and the signal waveform 205 are changed. The difference is that the signal waveform 604 and the signal waveform 605 are used. Other points may be similar to those of the first embodiment. The configuration of the image capturing apparatus according to this embodiment may be the same as that of the image capturing apparatus 101 according to the first embodiment. Therefore, the duplicated description of the similar components will be omitted.

  In the above-described first embodiment, an example in which the lighting time of the light source is long, such as a traffic signal using an LED light source that is displayed by blinking at a frequency twice that of the commercial power source, is shown. On the other hand, in a light source such as a light source that is controlled by blinking drive or a display body using such a light source, when the brightness is reduced, the ratio of lighting of the light source in the blinking cycle is less than 1/2. May be smaller. When such a light source is imaged, the light source and the display body may flicker like a flicker due to the sampling operation of a short exposure time for expanding the dynamic range. Therefore, a driving method of the image pickup apparatus according to the present embodiment that suppresses the flicker-like flicker will be described.

  In the timing chart of FIG. 6, a waveform 201 schematically shows a waveform obtained by full-wave rectifying an AC power supply. A waveform 202 shows a waveform of a lighting state of the LED light source that is dimmed by blinking drive using the power source of the waveform 201. A waveform 603 shows a waveform when the luminance is lower than that of the waveform 202 in the light source that performs the light control by the blinking drive.

  The signal waveform 604 of the pulse signal supplied from the control unit 103 to the image sensor 102 is for resetting the potential of the photoelectric conversion unit 107, which is sent from the sampling operation switching unit 113 to the reset unit 108, like the signal waveform 204 in FIG. The waveform of the reset signal of is shown. The signal waveform 605 of the pulse signal supplied from the control unit 103 to the image sensor 102 is the waveform of the transfer signal sent from the sampling operation switching unit 113 to the first transfer unit 109, similarly to the signal waveform 205 of FIG. Show.

  A period T215B shows a period of the first sampling operation in the frame in the short-time sampling operation. A period T216B shows a period of the last sampling operation in the frame among the short-time sampling operations. A period T217B shows a period from the start of the first sampling operation of the short-time sampling operation performed a plurality of times in one frame to the end of the last sampling operation. In this embodiment, in order to form an image of the same frame, a plurality of short-time sampling operations from the period T215B to the period T216B are performed during the period T217B. The charges generated during the plurality of sampling operations from the period T215B to the period T216B are sampled, transferred to the charge holding unit 110, and added to form a multiple-exposure image in a short exposure time.

  A period T218 shows a period of a long time sampling operation. The charges generated during the period T218 are sampled, and a single exposure image is formed in the charge holding unit 110 for a long exposure time.

  The operations of the signal waveform 206 to the signal waveform 208 for transferring the captured image to the output unit 112 by the second transfer unit 111 are the same as those in the first embodiment described above. The charges sampled in the nth frame are sequentially transferred to the output unit 112 row by row in the (n+1)th frame to form a frame image for one screen. An image with a different dynamic range is generated for each frame transferred from the output unit 112 to the image processing unit 104 to generate an image with an expanded dynamic range, which is displayed on the image display unit 105 and recorded in the image recording unit 106. The operation may be similar to that of the first embodiment.

  In the present embodiment, the interval between the plurality of short-time sampling operations performed in the period T215B to the period T216B is less than or equal to 1/2 of the period T212 which is the blinking cycle of the light source. In other words, the short-time sampling operation interval is ¼ or less of one frame period. In this embodiment, since sampling is performed m times during the period T212 which is the period of the light source, sampling is performed at intervals of 1/m (m≧2) of the period T212 which is the blinking period of the light source. Further, the period T217B is set equal to 1/2 of the period T211, that is, the period T212 which is a blinking cycle of the light source. That is, the period T217B is set to 1/2 of one frame period.

  When the lighting time of the light source is short as in the period T213B of the waveform 603, the lighting can be detected if the sampling interval is 1/2 or less of the period T213B that is the lighting time of the light source. As a result, it is possible to suppress a difference in brightness variation in the short-time exposure image, suppress flicker-like flicker due to the blinking operation of the light source, and obtain a moving image with high image quality and a wide dynamic range.

  Also in the present embodiment, since it is not necessary to match the relationship between the blinking phase of the light source and the phase of short-time exposure, the imaging device does not need a lighting detection unit that detects blinking of the light source. Further, since the operation for making the blinking phase of the light source and the exposure phase coincide with each other, the circuit configuration is simplified. As a result, an inexpensive imaging device is realized.

  Here, the relationship between the period T212, which is the blinking cycle of the light source, and the period of the frame of the imaging device needs to be matched to some extent. However, for example, consider the case where the blinking cycle of the light source is half the cycle of the commercial power source, such as a display device such as a traffic signal, an electronic bulletin board, and a display. The frequency of the commercial power supply is divided into 50 Hz or 60 Hz in each area. Therefore, for example, if the area is determined by GPS information or map information that has been widely used in recent years, it is not necessary to detect the blinking cycle of the light source. In the present embodiment, the lighting cycle of the light source and the period of the frame of the image pickup apparatus are shown in synchronization with each other. However, if the cycle is roughly the same, there will be no large fluctuation in brightness, so it is not necessary to strictly match the cycle. Good.

  Further, when the present embodiment is used, even if the light source is not a digital waveform such as the waveforms 202 and 603 but is a light source whose luminance periodically changes in an analog manner, sampling is performed a plurality of times. A moving image in which fluctuations and unevenness are suppressed can be acquired. This is because an average image obtained by sampling a plurality of times can be obtained even with a light source whose brightness changes greatly. Furthermore, it is possible to realize an imaging device that acquires a moving image with good image quality even with a light source that does not change periodically.

Third Embodiment With reference to FIG. 7, a method for driving an image sensor according to a third embodiment of the present invention will be described. FIG. 7 is a timing diagram illustrating the operation of the image pickup apparatus 101 according to this embodiment. In the present embodiment, as compared with the timing chart of the first embodiment of FIG. 2, instead of the signal waveform 204 from the control unit 103 to the image sensor 102 to the signal waveform 208, the signal waveform 204 and the signal waveform 205 are changed. The difference is that the signal waveform 704 and the signal waveform 705 are used. Other points may be similar to those of the first embodiment. The configuration of the image capturing apparatus according to this embodiment may be the same as that of the image capturing apparatus 101 according to the first embodiment. Therefore, the duplicated description of the similar components will be omitted.

  In the first embodiment described above, two images with different exposure conditions, that is, a multiple-exposure image for short-time exposure and a single-exposure image for long-time exposure are alternately output for each frame, and these images are combined. An example of outputting an image with expanded dynamic range has been shown. On the other hand, the present invention is not limited to image combination by combining one short-exposure image subjected to multiple exposure and one long-exposure image subjected to single exposure. In actual expansion of the dynamic range, the exposure time ratio may be a large value of 100 times or more. In such a case, the difference in captured luminance range between the short-time exposure image and the long-time exposure image becomes large, and thus the linearity of the combined image may deteriorate. In addition, there is a case where there is a large error in the luminance of the boundary at a luminance level near the threshold for combining, and the image quality may be deteriorated due to, for example, pseudo contour. In order to suppress such deterioration of image quality, it is conceivable that three or more types of images having different exposure times are acquired and combined, so that the image quality is improved as compared with the image captured and combined under two conditions. Therefore, the driving method of the image pickup apparatus according to the present embodiment will be described below.

  In the timing chart of FIG. 7, a waveform 201 schematically shows a waveform obtained by full-wave rectifying an AC power supply. A waveform 202 indicates a lighting state of an LED light source such as a traffic signal using the power source of the waveform 201.

  The signal waveform 704 of the pulse signal supplied from the control unit 103 to the image sensor 102 is for resetting the electric potential of the photoelectric conversion unit 107 sent from the sampling operation switching unit 113 to the reset unit 108, similarly to the signal waveform 204 in FIG. The waveform of the reset signal of is shown. The signal waveform 705 of the pulse signal supplied from the control unit 103 to the image sensor 102 is the waveform of the transfer signal sent from the sampling operation switching unit 113 to the first transfer unit 109, similarly to the signal waveform 205 of FIG. Show.

  A period T215 and a period T216 each indicate a period in which the short-time sampling operation is performed. A period T215C and a period T216C each indicate a period in which the third sampling operation is performed. The third sampling operation in the period T215C and the period T216C has a third exposure time. The third exposure time is set to be longer than the exposure time of the short-time sampling operation in the periods T215 and T216. Further, the third exposure time is set shorter than the exposure time of the long-time sampling operation in the period T218.

  A period T217 and a period T217C show a period from the start of the first sampling operation of the short-time sampling operation and the third sampling operation performed a plurality of times in one frame to the end of the last sampling operation. In this embodiment, two short-time sampling operations of a period T215 and a period T216 are performed during the period T217. Further, during the period T217C, the third sampling operation is performed twice, the period T215C and the period T216C. By sampling the charges generated during these sampling operations, transferring the charges to the charge holding unit 110, and adding the charges, two types of short-time exposure, that is, the exposure by the short-time sampling operation and the exposure by the third sampling operation are performed. Form a multiple exposure image. A period T218 shows a period of a long time sampling operation. The charges generated during the period T218 are sampled, and a long-exposure single-exposure image is formed in the charge holding unit 110.

  Here, the first pulse generator 114 may generate a pulse signal that causes the image sensor 102 to perform a third holding operation in which the third sampling operation is performed a plurality of times. Further, the first pulse generation unit 114 may sequentially switch generation of pulse signals for performing the first holding operation and the third holding operation for each frame. In addition, the control unit 103 further includes a third pulse generation unit that generates a pulse signal for performing the third holding operation, and sequentially outputs the pulse signals to be transmitted to the image sensor 102 by the sampling operation switching unit 113 for each frame. , May be switched.

  The charges held by the short-time sampling operations a plurality of times in the period T217 of the nth frame shown in the period T210(n) are exposed for one frame during the period T219 of the period T210(n+1) of the n+1th frame. It is output as an image. Similarly, the charge held in the third sampling operation for a plurality of times in the period T217C of the (n+1)th frame shown in the period T210(n+1) is output during the period T219 of the period T210(n+2) of the (n+2)th frame. It Further, the charge held by one long time sampling operation during the period T218 of the n+2th frame indicated by the period T210(n+2) is 1 frame during the period T219 of the period T210(n+3) of the n+3th frame. Minute exposure image is output. The image sensor 102 displays a multiple-exposure image that has been exposed for a short time by a short-time sampling operation, a multiple-exposure image that has been exposed by a third sampling operation, and a single-exposure image that has been exposed for a long time by a long-time sampling operation. In the present embodiment, the output is performed in a 3-frame cycle.

  The image processing unit 104 combines these three types of images with different exposure times into one image with an expanded dynamic range. For example, the ratio of the total exposure time of a plurality of short-time sampling operations, the total exposure time of a plurality of third sampling operations, and the exposure time of a single long-time sampling operation is 1:10:100. To do. In the case of synthesizing two types of images, an image obtained by the short-time sampling operation and an image obtained by the long-time sampling operation, the images having different ranges of 100 times are synthesized. However, in the present embodiment, by using the third sampling operation, it is possible to suppress the range difference for every 10 times, and it is possible to suppress the deterioration of the image quality. As a result, it is possible to obtain a high-quality image with an expanded dynamic range as compared with the case where images of two types of exposure times are combined. Further, when the image quality is suppressed to the same level, an image pickup apparatus having a larger dynamic range expansion amount is realized.

  Even when an image pickup apparatus having a dynamic range expansion function is realized as in the present embodiment, there is a problem that a blinking light source cannot be detected because the number of sampling cycles to be picked up increases when an image having a short exposure time is acquired. Can be suppressed. Further, with such a configuration, since it is less necessary to match the phase relationship between the blinking of the light source and the exposure for a short time, the lighting detection unit for detecting the blinking of the light source is not required. Further, since the blinking phase of the light source and the exposure operation phase of the imaging device do not have to match, the circuit configuration is simplified. As a result, an inexpensive imaging device is realized.

Fourth Embodiment With reference to FIGS. 8 to 10, a structure of an image pickup apparatus and a driving method of an image pickup element according to some embodiments of the present invention will be described. FIG. 8: is a block diagram which shows typically the structural example of the imaging device 801 in the 4th Embodiment of this invention. Compared to the imaging device 101 shown in FIG. 1, the image sensor 102 is provided with a pixel 820 including two sets including the charge holding unit 110, the first transfer unit 109, and the second transfer unit 111. .. Further, the sampling operation switching unit 113 is changed to a sampling operation control unit 813 that supplies a pulse signal for driving the image sensor 102. The configuration of the imaging device 801 other than this may be the same as the configuration of the imaging device 101 described above. In this specification, when it is necessary to distinguish each configuration included in each set, the reference numerals of each configuration are suffixed with S and L (for example, the charge holding unit 110S).

  FIG. 9 shows a configuration example of an equivalent circuit of the pixel portion of the image pickup element 102 of the image pickup apparatus 801 according to this embodiment. The pixel 800 corresponds to the pixel 820 in FIG. Compared to the pixel 300 shown in FIG. 3, the pixel 800 includes two sets including the charge holding unit 302, the first transfer switch 304, and the second transfer switch 305. Further, the control lines 321S and 321L for controlling the first transfer switches 304S and 304L, and the control lines 322S and 322L for controlling the second transfer switches 305S and 305L are included. The configuration of the pixel 800 other than this may be the same as the configuration of the pixel 300 described above. With such a configuration, while one of the charge holding units holds the charge, the charge is transferred from the photoelectric conversion unit 301 to the other charge holding unit and held (first step). Then, while the charges are held in the two charge holding units in the first step, the charges are transferred from one of the charge holding sections to the output section (second step). By sequentially performing such an operation, it is possible to hold signals with high simultaneity in the two charge holding units. The details will be described below.

  In the pixel 800, the charge generated by the photoelectric conversion unit 301 is generated by the charge holding unit 302S and the charge holding unit 302L according to the time selected by the first transfer switch 304S and the first transfer switch 304L, respectively. Hold on to each. In the above-described first to third embodiments, the control unit 103 causes the image sensor 102 to perform sampling operations with different exposure times for each frame, and sequentially generate images with different exposure times for each frame. On the other hand, in the present embodiment, the pixel 800 includes the plurality of charge holding units 110, so that the control unit 103 causes the image sensor 102 to perform sampling operations of different exposure times in parallel for each frame, and obtains at different exposure times. Generated multiple signals.

  FIG. 10 is a timing chart for explaining the operation of the image pickup apparatus 801 of this embodiment. A period T910(n), a period T910(n+1), a period T910(n+2), and a period T910(n+3) show the period of the nth, n+1, n+2, and n+3th frame, respectively. In this embodiment, in each frame, since each pixel 820 has two charge holding units, two signals having different exposure times are output from the image sensor 102 to the image processing unit 104. The image processing unit 104 synthesizes two signals output for each frame with different exposure times to generate an image. In the present embodiment, each pixel 820 includes two sets including the charge holding unit 110, the first transfer unit 109, and the second transfer unit 111. However, each pixel 820 includes, for example, three or more sets, and for each frame. Three or more images with different exposure times may be generated.

  A waveform 901 shows a lighting waveform of an analog modulated LED light source used for display of, for example, an electronic bulletin board, various lighting devices, and automobiles. A signal waveform 904 shows a waveform of a pulse signal output from the sampling operation control unit 813 to the reset unit 108 for resetting the potential of the photoelectric conversion unit 107. Similar to the above-described embodiment, the "H" level is reset and the "L" level is not reset. Further, the photoelectric conversion units 107 of all the pixels 820 arranged in an array on the image sensor 102 are collectively reset at the same time. Therefore, the “H” level reset signal of the signal waveform 904 is input to the reset units 108 of all the pixels 820 at the same timing.

  The signal waveform 905S is a pulse signal output from the sampling operation control unit 813 to the first transfer unit 109S, and the signal waveform 905L is a pulse signal waveform output from the sampling operation control unit 813 to the first transfer unit 109L. Shown respectively. In this embodiment, the charges generated by the photoelectric conversion unit 107 are simultaneously transferred to the charge holding unit 110S or the charge holding unit 110L in all the pixels 820 arranged in an array. When the signal waveform 905S is at “H” level, the charges generated by the photoelectric conversion unit 107 are transferred to the charge holding unit 110S. At this time, the “H” level transfer signal of the signal waveform 905S is simultaneously input to the first transfer units 109S of all the pixels 820. Similarly, when the signal waveform 905L is at “H” level, the charges generated by the photoelectric conversion unit 107 are transferred to the charge holding unit 110L. At this time, the “H” level transfer signals of the signal waveform 905L are simultaneously input to the first transfer units 109L of all the pixels 820. As a result, the charges generated in the photoelectric conversion unit 107 are supplied to the two charge holding units 110S and 110L at different exposure times defined by the signal waveform 904 and the signal waveform 905S, and the signal waveform 904 and the signal waveform 905L. Sampled. Alternatively, the charges may be sampled with the same exposure time but different sampling times.

  Here, in the periods T915S, T916S, and T917S, after being reset by the drive pulse of the signal waveform 904, the charge accumulated in the photoelectric conversion portion 107 is transferred to the charge holding portion 110S by the drive pulse of the signal waveform 905S for a short time. The respective periods of the sampling operation are shown. Similarly, in the periods T915L, T916L, and T917L, after being reset by the driving pulse of the signal waveform 904, the charge accumulated in the photoelectric conversion portion 107 is transferred to the charge holding portion 110L by the driving pulse of the signal waveform 905L for a long time. The respective periods of the sampling operation are shown. The periods T915S, T916S, and T917S in which the short-time sampling operation is performed and the periods T915L, T916L, and T917L in which the long-time sampling operation is performed are sequentially performed without being performed at the same time. By causing the image pickup device 102 to sequentially perform the short-time sampling operation and the long-time sampling operation, the control unit 103 can generate a plurality of images with different exposure times in parallel in the same frame period. Become. Alternatively, it is possible to generate images with high simultaneity in parallel although the sampling timings are different.

  The signal waveforms 906S and 906L are pulses output from the sampling operation control unit 813 to the second transfer units 111S and 111L in order to sequentially transfer the charges temporarily held in the charge holding units 110S and 110L to the output unit 112. The waveform of a signal is shown. When the signal waveforms 906S and 906L are at “H” level, the second transfer units 111S and 111L in the corresponding pixels 820 are turned on, and the charges temporarily held in the charge holding units 110S and 110L are sequentially output. 112 is transferred. Each of the signal waveforms 906S-1, 2 and 3 shows a waveform of a pulse signal when transferring the charges accumulated in the charge holding unit 110S during the periods T917S, 915S and 916S, respectively. Similarly, each of the signal waveforms 906L-1, 2 and 3 shows a waveform of a pulse signal when transferring the charges accumulated in the charge holding unit 110L during the periods T917L, 915L and 916L, respectively. In the period T910, the image of one frame from the first row to the last row is sequentially output to the output unit 112 by the signal waveforms 906S-1 and 906L-1 to the signal waveforms 906S-3 and 906L-3. It is a transfer period for one screen to be output. The period T910 can be referred to as one frame period.

  In this embodiment, the short-time sampling operation shown in periods T915S, T916S, and T917S is performed three times during the period T910 of one frame to generate an image subjected to multiple exposure in a short exposure time. In addition, during a period T910 of one frame, a long time sampling operation longer than the short time sampling operation shown in the periods T915L, T916L, and T917L is performed three times, and an image subjected to multiple exposure with a long exposure time is generated. The total time of a plurality of short-time sampling operations and long-time sampling operations in one frame period is the exposure time of an image obtained in each sampling operation in one frame period.

  In the above-described first to third embodiments, images are sequentially generated for each frame at different exposure times, and images of a plurality of frames are combined to obtain a combined image with an expanded dynamic range. On the other hand, in the present embodiment, two charge holding units, that is, the charge holding unit 110S and the charge holding unit 110L are provided, and appropriate pulses are input to the respective transfer units, so that two different exposure times are obtained in the same frame. It becomes possible to generate an image. When the number of sampling operations of the sampling operations having different exposure times is sufficiently increased, the time from the start to the end of each sampling operation becomes short. Since the amount of change in the information obtained at this time is small between adjacent sampling operations of different exposure times, the two pieces of information obtained have different exposure periods, but information accumulated continuously from the start of exposure to the end of exposure. Can be the same information as. For example, the first row obtains information in the period T921, the second row obtains information in the period T922, and the last row obtains information in the period T923. That is, according to the present embodiment, it is possible to obtain image information at different exposure times and at substantially the same time.

  In this way, the charges generated by the photoelectric conversion units 107 of the pixels 820 arranged in an array on the image sensor 102 are simultaneously held in the two charge holding units 110 for each pixel at different sampling times. After that, the charges are output from the charge holding units 110 to the output unit 112 for each row, so that a multiple-exposure image in which charges sampled at different exposure times are added in a frame period for one screen is output. In the present embodiment, the image obtained from the charges held in the charge holding unit 110S is a multiple exposure image with a short exposure time, and the image obtained from the charges held in the charge holding unit 110L is a multiple exposure image with a long exposure time. It becomes an exposure image.

  Two multiple-exposure images with different exposure times are images obtained by sampling multiple times at intervals of time, and therefore they are not fixed values of a certain momentary change of the lighting state of the light source, but of one frame period. It becomes information. Therefore, it is possible to suppress flicker or flicker-like flicker that occurs when the light source is photographed. Even if the flicker cannot be completely suppressed, the flicker occurrence status in the screen is similar between the short exposure time image and the long exposure time image. It is possible to suppress the deterioration of image quality due to the difference in the occurrence status of. Further, since each multiple-exposure image is obtained by sampling the same frame period a plurality of times, it becomes an image obtained by photographing at substantially the same time with two types of exposure times. Therefore, an image with an expanded dynamic range can be obtained by combining the read multiple exposure images. Further, since the multiple-exposure images have simultaneity, it is possible to suppress the occurrence of image shift caused by the movement of the subject or the camera, as compared with the case where the images of different frames are combined. ..

  In this embodiment, the control unit 103 causes each pixel 820 to perform the short-time sampling operation and the long-time sampling operation three times for the information of the photoelectric conversion unit 107 in one frame period. Therefore, in one frame period, the time from the start of one short-time or long-time sampling operation to the start of the next short-time or long-time sampling operation is 1/one of the period of one frame. It is 3. However, the number of sampling operations is not limited to this. Since the flicker suppressing effect can be obtained by performing the sampling operation twice or more, the sampling operation may be performed twice, for example. Further, the sampling operation may be performed four times or more. Further, for example, the number of short-time samplings and the number of long-time samplings performed in one frame period may be different from each other.

  When the number of sampling operations in one frame period is increased, the acquired information may have the same effect as in the case where the information is continuously accumulated in each frame period. Thereby, for example, the detection accuracy of the blinking light source is improved, and the fluctuation of the image due to the blinking of the detected light source can be suppressed, and the flicker can be suppressed. Moreover, when a moving image is captured, the blurring amount becomes the same between the image having a short exposure time and the image having a long exposure time because the images are captured in the same frame, so that a sense of discomfort in the combined image is suppressed. For example, when a vehicle running at night is photographed, it is possible to prevent problems such as the light not blurring even if the body is blurred, and the blurring of the light becoming dotted lines.

  As described above, by increasing the number of sampling operations, in other words, shortening the pulse interval of the signal waveforms 905S and 905L, the effect of suppressing flicker is increased. When the blinking frequency of the light source is N (Hz), the effect of suppressing flicker appears from the sampling cycle (Hz)>N (Hz). Further, a sufficient effect is exhibited under the condition that the sampling cycle (Hz)>2N. For example, flicker is likely to appear when imaging at 100 Hz under the illumination of a fluorescent lamp using a commercial power supply of 50 Hz. In this case, the effect appears when the sampling cycle (Hz)>100 Hz, and the sufficient effect can be obtained by setting the sampling cycle (Hz)>200 Hz. A flicker suppressing effect can be obtained by setting the pulse interval of the signal waveforms 905S and 905L to 10 ms or less for a light source using a commercial power supply of 50 Hz. Further, in order to further enhance the effect of suppressing flicker, the pulse intervals of the signal waveforms 905S and 905L may be shorter than 5 ms, for example.

  Further, the total time of the exposure times in which the charges are multiplexed may be increased in order to enhance the effect of suppressing flicker. For example, the total time of the total exposure time of the short-time sampling operation and the total exposure time of the long-time sampling operation may be 1/2 or more of the blinking cycle of the light source.

  Also in the present embodiment, when there is a blinking light source with a short cycle in imaging in an environment where a wide dynamic range is required, an imaging device with improved detection accuracy of the on/off state of the light source is realized. Further, it is possible to realize an image pickup apparatus that suppresses flicker due to fluctuations in exposure amount due to blinking of a light source and acquires a moving image with good image quality. Further, in the configuration of the present embodiment, a wide dynamic range can be created by acquiring a plurality of images with different exposure times for each frame and combining the obtained images. It is possible to suppress temporal image shift due to camera movement or subject movement and flicker difference between light sources due to light sources, as compared to the case where a plurality of images with different exposure amounts are obtained and combined for each frame. . Further, similarly to each of the above-described embodiments, the lighting detection unit that detects the blinking of the light source is not required, and the blinking phase of the light source and the exposure operation of the sensor do not have to match. Therefore, the circuit configuration is simplified and an inexpensive imaging device is realized.

  Although the four embodiments of the present invention have been described above, the present invention is not limited to these embodiments. The above-described respective embodiments can be appropriately modified and combined. For example, in the first to third embodiments, a frame in which sampling operations of two or more types of short-time exposure are performed three times or more and a frame in which a long-time sampling operation of single exposure is performed are switched for each frame for imaging. You may. In addition to sequentially switching the holding operation for each frame, the sampling operation switching unit 113 causes the sampling operation switching unit 113 to hold the first holding operation twice and then the second holding operation once in a predetermined cycle. You may switch so that it may repeat and image. Furthermore, the length of the “H” level or “L” level of the signal waveform of each embodiment can be changed appropriately. For example, in the second and subsequent embodiments, the length of the “H” level of the transfer unit when obtaining a signal with a long exposure time is longer than the “H” level when obtaining a signal with a short exposure time. However, this may have the same length or the magnitude relationship may be reversed. Further, in the transfer from the photoelectric conversion unit to the charge holding unit and/or the transfer from the charge holding unit to the output unit, the charges may be completely transferred. In order to realize such charge transfer, it is preferable that the potential of the photoelectric conversion unit at equilibrium is higher than the potential of the charge holding unit, and the potential of the charge holding unit at equilibrium is higher than the potential of the output unit. The potential here is a potential for the signal charge. Therefore, when electrons are used as signal charges, the lower the potential, the higher the potential.

101 image pickup device, 102 image pickup element, 103 control unit, 107 photoelectric conversion unit, 109 first transfer unit, 110 charge holding unit, 111 second transfer unit, 112 output unit, 120 pixels

Claims (24)

An image pickup apparatus comprising: an image pickup element; and a control section for controlling the image pickup element,
The image sensor has a plurality of pixels,
Each pixel of the plurality of pixels includes a photoelectric conversion unit, a charge holding unit, an output unit, a first transfer unit that transfers the charges generated by the photoelectric conversion unit to the charge holding unit, and the charge holding unit. A second transfer unit that transfers the electric charge held in the unit to the output unit,
The control unit transfers a charge generated by the photoelectric conversion unit to the charge holding unit by the first transfer unit and holds the charge, and a charge operation held in the charge holding unit by the second holding unit. A transfer operation of transferring to the output section by the transfer section, and causing the image sensor to repeatedly perform,
The control unit, as the holding operation,
A first sampling operation of transferring the charges generated by the photoelectric conversion unit in the first exposure time to the charge holding unit by the first transfer unit and holding the charges is performed a plurality of times in a period of one frame. Holding action,
The second sampling operation of transferring the charges generated by the photoelectric conversion unit to the charge holding unit by the first transfer unit and holding the charges in the second exposure time longer than the first exposure time is performed as described above. A second holding operation that is performed once or more during a frame period;
A plurality of holding operation including, to perform the image pickup device,
Of the period of the one frame, a period from the start of the first first sampling operation to the end of the final first sampling operation is the first of the second of the periods of the one frame. An imaging device, which is shorter than a period from the start of a sampling operation to the end of the last second sampling operation .   The image pickup apparatus according to claim 1, wherein the first sampling operation and the second sampling operation are performed simultaneously in the plurality of pixels.   The image pickup apparatus according to claim 1, wherein a period from the start of the transfer operation to the end of the one frame is the same period in each frame.   In the period of one frame, the period from the start of one first sampling operation to the start of the next first sampling operation is ½ or less of the period of one frame. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus.   In the period of one frame, the period from the start of one first sampling operation to the start of the next first sampling operation is ¼ or less of the period of one frame. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus.   In the period of one frame, the period from the start of the first first sampling operation to the end of the final first sampling operation is ½ or more of the period of the one frame. The image pickup apparatus according to claim 4 or 5, wherein: Wherein one of the frame period, the period of the transfer operation, the imaging apparatus according to any one of claims 1 to 6, wherein the at 1/2 of 1 frame period follows. In the plurality of holding operations, charges generated by the photoelectric conversion unit during a third exposure time, which is longer than the first exposure time and shorter than the second exposure time, are transferred to the first transfer. a third sampling operation of holding transferred to the charge holding portion by parts, any one of claims 1 to 7, further comprising a third holding operation a plurality of times during the period of one frame The imaging device according to. The imaging device further includes an image combining unit,
The image synthesizing section, exposed synthesized image obtained in at least two frames of different times, the image pickup apparatus according to any one of claims 1 to 8, characterized in that output. Each pixel of the plurality of pixels includes two sets including the charge holding unit, the first transfer unit, and the second transfer unit,
The control unit is
In the same frame period,
The first holding operation is applied to one of the sets,
The second holding operation is applied to the other of the sets.
The imaging apparatus according to any one of claims 1 to 9, characterized in that to perform respectively. The total time of the first exposure time and the second exposure time in the one frame period is 1/2 or more of the one frame period. Item 10. The imaging device according to item 10 . The plurality of pixels further includes a reset unit that resets the potential of the photoelectric conversion unit,
The reset of the charges of the photoelectric conversion unit by the reset unit is performed simultaneously in the plurality of pixels before the first sampling operation and the second sampling operation. the imaging apparatus according to any one of 11. The imaging device according to claim 12 , wherein the reset unit has an overflow drain structure. The period of one frame, the image pickup apparatus according to any one of claims 1 to 13, characterized in that equal to 1/2 of the period or cycle of the commercial power supply of the commercial power source. A method for driving an image sensor, which has a plurality of pixels and generates one image in one frame period,
Each pixel of the plurality of pixels includes a photoelectric conversion unit, a first charge holding unit, a second charge holding unit, and an output unit,
The driving method is
Holding a first charge generated by the photoelectric conversion unit in the first exposure time in the first charge holding unit;
Holding a second charge generated by the photoelectric conversion unit in the second charge holding unit for a second exposure time longer than the first exposure time;
Including,
At least a part of a period in which the first charge is held in the first charge holding unit, and at least a part of a period in which the second charge is held in the second charge holding unit; Ri is Do not heavy,
The first charge is generated by a plurality of first sampling operations,
The second charge is generated by one or more second sampling operations,
The driving method in the period of one frame, the first number of the sampling operation, characterized that you have differs from the second number of sampling operations.   The driving method according to claim 15, wherein a sampling cycle of the first sampling operation and a sampling cycle of the second sampling operation are earlier than a cycle of a commercial power supply.   A method for driving an image sensor, which has a plurality of pixels and generates one image in one frame period,
  Each pixel of the plurality of pixels includes a photoelectric conversion unit, a first charge holding unit, a second charge holding unit, and an output unit,
  The driving method is
    Holding a first charge generated by the photoelectric conversion unit in the first exposure time in the first charge holding unit;
    Holding a second charge generated by the photoelectric conversion unit in the second charge holding unit for a second exposure time longer than the first exposure time;
Including,
  At least a part of a period in which the first charge is held in the first charge holding unit, and at least a part of a period in which the second charge is held in the second charge holding unit; Overlap,
  The first charge is generated by a plurality of first sampling operations,
  The second charge is generated by one or more second sampling operations,
  A driving method, wherein a sampling cycle of the first sampling operation and a sampling cycle of the second sampling operation are earlier than a cycle of a commercial power supply.   18. The driving method according to claim 17, wherein the number of times of the first sampling operation and the number of times of the second sampling operation are the same in the period of the one frame. Each of the sampling operations of the first sampling operation multiple times, the
Transferring charges generated by the photoelectric conversion unit during the third exposure time from the photoelectric conversion unit to the first charge holding unit;
In the period of one frame, the total of the third exposure time is the first exposure time,
The second charge is generated by a plurality of second sampling operations,
Each sampling operation of the plurality of second sampling operations is
Transferring charges generated by the photoelectric conversion unit during the fourth exposure time from the photoelectric conversion unit to the second charge holding unit;
19. The driving method according to claim 15 , wherein a total of the fourth exposure times is the second exposure time in the one frame period. The driving method according to any one of claims 15 to 19 , wherein the first sampling operation and the second sampling operation are alternately performed in the period of the one frame. Period and the period of the second sampling sampling operation of the first sampling of the sampling operation, in any one of claims 15 to 20, characterized in that less than half of the period of the commercial power supply The described driving method. Wherein in a period of one frame, the time which is the sum of said first exposure time and said second exposure time, of claims 15 to 21, characterized in that said at one frame 1/2 or more periods of The driving method according to any one of items. An image pickup apparatus comprising: an image pickup element; and a control section for controlling the image pickup element,
Each pixel of the plurality of pixels includes a photoelectric conversion unit, a first charge holding unit, a second charge holding unit, and an output unit,
The control unit is
Causing the image sensor to generate one image in one frame period,
Causing the image sensor to perform a holding operation including a first holding operation and a second holding operation,
The first holding operation causes the first charge holding unit to hold the first charge generated by the photoelectric conversion unit during a first exposure time,
In the second holding operation, the second electric charge holding unit holds the second electric charge generated in the photoelectric conversion unit in a second exposure time longer than the first exposure time,
At least a part of a period in which the first charge is held in the first charge holding unit, and at least a part of a period in which the second charge is held in the second charge holding unit; Ri is Do not heavy,
The first charge is generated by a plurality of first sampling operations,
The second charge is generated by one or more second sampling operations,
In the period of one frame, the first number of sampling operations, the imaging apparatus characterized that you have differs from the second number of sampling operations.   An image pickup apparatus comprising: an image pickup element; and a control section for controlling the image pickup element,
  Each pixel of the plurality of pixels includes a photoelectric conversion unit, a first charge holding unit, a second charge holding unit, and an output unit,
  The control unit is
    Causing the image sensor to generate one image in one frame period,
    Causing the image sensor to perform a holding operation including a first holding operation and a second holding operation,
      The first holding operation causes the first charge holding unit to hold the first charge generated by the photoelectric conversion unit during a first exposure time,
      In the second holding operation, the second electric charge holding unit holds the second electric charge generated in the photoelectric conversion unit in a second exposure time longer than the first exposure time,
  At least a part of a period in which the first charge is held in the first charge holding unit, and at least a part of a period in which the second charge is held in the second charge holding unit; Overlap,
  The first charge is generated by a plurality of first sampling operations,
  The second charge is generated by one or more second sampling operations,
  An imaging apparatus, wherein a sampling cycle of the first sampling operation and a sampling cycle of the second sampling operation are earlier than a cycle of a commercial power source.

Images