JP6041593B2 - Solid-state imaging device - Google Patents

Description

The present invention relates to a solid-state imaging equipment such as an electronic camera.

  An image pickup apparatus including an XY address scanning type image pickup element uses a rolling shutter system that controls the charge accumulation time of a photoelectric conversion element for each line by address designation. When moving pictures are taken under a fluorescent lamp using this rolling shutter method, horizontal stripes appear in the image due to the influence of periodic luminance changes due to the flicker frequency of the fluorescent lamp light source (for example, the commercial power supply frequency of 50 Hz or 60 Hz). Unevenness (hereinafter referred to as flicker) may occur. In view of this, there has been proposed an imaging apparatus that calculates the flicker frequency of a light source from a photographed image and suppresses flicker based on the flicker frequency (see, for example, Patent Document 1).

JP 2011-176622 A

  In the imaging device of Patent Document 1, in the case of a moving image with a high frame rate (for example, a moving image of 1000 fps) in which the readout time of one screen is shorter than the blinking cycle of the light source, the charge accumulation time is set to an integral multiple of a half cycle of the blinking frequency. You may not be able to. In that case, flicker cannot be suppressed.

An object of the present invention is to provide a solid-state imaging equipment that can also reduce flicker when the frame rate is high video.

The solid-state imaging device of the present invention includes a pixel unit that captures a plurality of frames by photoelectric conversion, an exposure amount calculation unit that calculates an exposure amount with a plurality of frames captured within the flicker cycle of the light source as one unit, on the basis of the exposure amount that is arithmetic, for a plurality of frames to be captured after the plurality of frames used for calculation, including at least a plurality of charge accumulation period of two mutually different, and / or different from each other And a control unit that controls an exposure amount of each frame imaged by the pixel unit by setting a plurality of amplification gains including at least two .

  Even in the case of a moving image with a high frame rate, automatic exposure control is possible, and flicker can be reduced.

It is a block diagram which shows the structural example of a solid-state imaging device. It is a flowchart which shows the operation example of the solid-state imaging device of 1st Embodiment. It is a figure explaining the flicker frequency acquisition of 1st Embodiment. It is a figure explaining an example of exposure amount difference acquisition. It is a figure explaining an example of gain calculation and gain setting selection of each frame. It is a figure explaining an example of exposure amount control. It is a block diagram which shows the structural example of the solid-state imaging device of 2nd Embodiment. It is a flowchart which shows the operation example of the solid-state imaging device of 2nd Embodiment. It is a figure explaining an example of each line gain correction value acquisition and exposure amount control. It is a flowchart which shows the operation example of the solid-state imaging device of 3rd Embodiment. It is a figure explaining an example of exposure amount control.

(First embodiment)
In the first embodiment of the present invention, a plurality of frames within a flicker cycle are used as one unit for exposure amount calculation and exposure amount correction. From the exposure amount of each frame in one unit of exposure amount calculation, the accumulation time or gain of each frame in one unit to be imaged next is calculated. Then, imaging is performed using the calculation result, gain correction is performed, and the influence of flicker between frames is reduced. The exposure amount is determined by the external light intensity (or luminance), the pixel accumulation time, and the gain, and can be expressed as external light intensity × accumulation time × gain. One aspect of the exposure amount calculation is to calculate the external light intensity from the captured image data using the exposure time and gain at the time of imaging. As an example of the exposure adjustment method, in the present embodiment, the average value of all the pixel data in one frame is defined as one frame average exposure, and the one frame average exposure is a constant exposure (that is, an appropriate exposure). A method for adjusting the gain with the accumulation time fixed is shown. When the frame rate is high, in order to obtain a good image, it is desirable to set the accumulation time to the maximum accumulation time possible at the frame rate, so the accumulation time is fixed, but means for fine adjustment of the exposure amount The accumulation time may be adjusted as follows.

  FIG. 1 is a block diagram showing an example of the configuration of the solid-state imaging device according to the first embodiment of the present invention. In the pixel unit 1, a plurality of pixels each including a photoelectric conversion element are arranged in a matrix, and a plurality of frames are imaged by photoelectric conversion. The row scanning unit 2 is disposed adjacent to the pixel unit 1 and scans a row of pixels of the pixel unit 1. In the row scanning, an electronic shutter operation is performed to read out the charges accumulated in the pixels of the pixel unit 1 and control the charge accumulation time in the pixel unit 1. The CDS unit 3 performs correlated double sampling (CDS) on the pixel output signal from the pixel unit 1 and subtracts pixel reset noise superimposed on the pixel output signal. The gain adjustment unit 4 performs gain adjustment on the analog signal after CDS. The gain adjusting unit 4 may cause the CDS unit 3 to have a gain adjusting function. In addition, a gain adjusting function may be provided in the AD converter 5 described later. The AD converter 5 converts the analog signal after CDS into a digital signal. The horizontal scanning unit 6 sequentially transfers the digital signal of each column to the image processing unit 11 and the exposure amount calculation unit 9. The timing generation unit 7 generates a timing signal for controlling the row scanning unit 2, the CDS unit 3, the AD conversion unit 5, and the horizontal scanning unit 6. In addition, the timing generation unit 7 has a pixel accumulation time control function by controlling the time interval between the start of electronic shutter scanning and the start of readout scanning of the row scanning unit 2. The setting unit 8 holds the timing setting of the timing generation unit 7 and the gain setting of the gain adjustment unit 4. The exposure amount calculation unit 9 calculates the exposure amounts of a plurality of frames based on the image data of a plurality of frames serving as one unit of the exposure amount calculation output from the horizontal scanning unit 6. Based on the exposure information acquired by the exposure amount calculation unit 9, the frame exposure amount calculation unit 10 calculates the accumulation time and gain of each frame in a plurality of frames that are one unit of the exposure amount calculation to be imaged next. . The image processing unit 11 performs various types of image processing (gradation conversion processing, contour enhancement processing, etc.). The control unit 12 controls the setting unit 8, the exposure amount calculation unit 9, the frame exposure amount calculation unit 10, and the image processing unit 11.

  Next, an example of the operation of the solid-state imaging device according to the present embodiment will be described using the flowchart of FIG. 2 and FIGS. 3 to 6. First, the operation for obtaining the flicker frequency will be described. FIG. 2 is a flowchart illustrating a method for driving the solid-state imaging device.

  When imaging is started in step st100, the solid-state imaging device acquires an image for determining the presence or absence of solid flicker in step st101, and acquires a flicker frequency in step st102. An example of the operation of steps st101 and st102 will be described in more detail with reference to FIG. 3A shows a change in luminance of the light source, FIG. 3B shows a vertical synchronization signal input to the timing generation unit 7, and FIG. 3C shows an image output from the horizontal scanning unit 6. FIG. 3D shows the average luminance of each row of the output image, and FIG. 3E shows the exposure amount of each row of the output image calculated from FIG.

  In step st <b> 101, the control unit 12 performs imaging for four cycles or more at a frame rate of 3 × Ff (fps) or higher with respect to the frequency change frequency Ff (Hz) of the light source assumed in advance. 3A to 3E, photographing is performed in four flicker cycles at a frame rate of 5 × Ff (fps) with respect to the frequency Ff (Hz) of the luminance change of the light source. The determination of the frame rate is preferably 3 frames or more in order to determine the presence or absence of flicker. The number of frames used to determine the presence / absence of flicker is determined according to the number of pixels and the driving frequency of the solid-state imaging device. As the number of frames increases, the accuracy of exposure correction increases in each imaging screen. To do.

  Next, in step st102, the exposure amount calculation unit 9 obtains the average luminance of each row of the image data output from the horizontal scanning unit 6, and further, the spatial frequency component of the average luminance of each row in frame units from the average luminance of each row. Is subtracted to obtain the exposure amount of each row. An exposure threshold value is provided, and when the exposure amount of each row falls below the exposure threshold value three times or more and the interval is constant, the interval is set as a flicker cycle. FIG. 3D shows the average luminance of each row of the output image, but the luminance change within one frame includes a subject spatial frequency component that depends on the luminance of the photographed object. The frame fr103 having a high frame average luminance is less affected by the luminance change of the light source, and the luminance change in the frame is almost a subject spatial frequency component. That is, by subtracting the relative luminance change of the frame fr103 having a high frame average luminance from the luminance change of each frame, an exposure amount change component obtained by removing the subject spatial frequency component of each frame shown in FIG. 3E is obtained. When the exposure amount threshold e100 is provided for the exposure amount of each frame, in FIG. 3E, the exposure amount of the frame falls below the exposure amount threshold value e100 at times t101, t102, and t103. When the interval p101 between the times t101 and t102 is equal to the interval p102 between the times t102 and t103, it is determined that there is a periodic luminance change of the light source, and the flicker frequency is obtained based on the time interval p101.

  After step st102, in step st103, the exposure amount calculation unit 9 determines the presence or absence of flicker. That is, the exposure amount calculation unit 9 determines the presence or absence of a periodic luminance change (flicker) of the light source based on the exposure amounts of a plurality of frames. If flicker is absent, the control unit 12 starts main imaging in step st112. If flicker is present, the control unit 12 determines the number of frames for which the exposure amount calculation unit 9 performs exposure amount calculation processing in step st104. .

  In step st104, when N is an integer of 3 or more, the exposure amount calculation unit 9 sets the frame rate to N × Ff (fps) for the flicker frequency Ff (Hz), and the exposure amount calculation processing is performed for N frames. Decide to do it once.

  Next, the exposure amount calculation imaging performed before the main imaging and the operation of the exposure amount calculation will be described. In step st105, the control unit 12 performs exposure amount imaging. Next, in step st106, the control unit 12 determines whether or not imaging of N frames has been completed. If not, the control unit 12 returns to step st105. After the imaging of N frames is completed, in step st107, the exposure amount calculation unit 9 determines an exposure amount peak frame. Next, in step st108, the exposure amount calculation unit 9 calculates the difference between the exposure amount of each frame and the exposure amount of the peak frame. Next, in step st109, the exposure amount calculation unit 9 acquires the definite exposure amount of the exposure amount peak frame. An example of the operations of steps st105, st106, st107, st108, and st109 will be described in more detail with reference to FIG.

  4A shows a change in luminance of the light source, FIG. 4B shows a vertical synchronization signal input to the timing generation unit 7, and FIG. 4C shows an image output from the horizontal scanning unit 6. 4D shows the average luminance of each row of the output image, and FIG. 4E shows the exposure amount of each row of the output image calculated from FIG. 4D.

  In steps st105 and st106, the control unit 12 starts imaging at the frame rate determined in step st104. 4A to 4E, imaging is performed at a frame rate of 5 × Ff (fps) with respect to the flicker frequency Ff (Hz). After imaging of 5 frames within the flicker cycle of the light source, that is, the number of exposure calculation frames is completed, in step st107, the exposure amount calculation unit 9 determines an exposure amount peak frame. The exposure amount calculation unit 9 obtains the average luminance of each row of the image data output from the horizontal scanning unit 6 shown in FIG. 4D, and subtracts the spatial frequency component of the average luminance of each row in the frame unit from the average luminance of each row. Then, the exposure amount of each row shown in FIG. Then, the exposure amount calculation unit 9 obtains the average exposure amount of each frame from the exposure amount of each row, and determines the frame having the highest average exposure amount as the exposure amount peak frame. In FIG. 4 (e), the frame average exposure amount of the frame fr203 is a peak, and the exposure amount peak frame is determined to be the frame fr203.

  Next, in step st108, the exposure amount calculation unit 9 calculates the difference in exposure amount between each frame in the flicker cycle of the light source and the peak frame of each frame. In FIG. 4E, the exposure amount difference of the frame fr201 with respect to the exposure amount peak frame fr203 is er201. Similarly, the exposure amount calculation unit 9 calculates and stores the exposure amount difference of other frames.

  In step st109, the exposure amount calculation unit 9 acquires the exposure amount of the exposure amount peak frame. In FIG. 4E, the exposure amount ea200 of the frame fr203 becomes the exposure amount of the exposure amount peak frame.

  Subsequently, each frame exposure amount calculation in the number of exposure amount calculation frames in the main imaging using the exposure amount of the exposure amount peak frame and the exposure amount difference of each frame in the number of exposure amount calculation frames, and the main imaging Will be described.

  In step st110, the frame exposure amount calculation unit 10 calculates the accumulation time and gain of each frame in the exposure amount calculation processing unit. In step st111, the control unit 12 selects the accumulation time and gain setting of each frame calculated in step st110. In step st112, the control unit 12 performs main imaging. An example of the operation of steps st110, st111, and st112 will be described in more detail with reference to FIG.

  5A shows a change in luminance of the light source, FIG. 5B shows a vertical synchronization signal input to the timing generation unit 7, and FIG. 5C shows each frame calculated by the frame exposure amount calculation unit 10. Indicates the gain value. FIG. 5D shows an image output from the horizontal scanning unit 6 captured with the gain shown in FIG. 5C, and FIG. 5E shows the average luminance of each row of the output image.

  In step st110, the frame exposure amount calculation unit 10 calculates the relative gain shown in FIG. 5C from the exposure amount differences er201 to er205 (FIG. 4E) with respect to the exposure amount peak frame calculated by the exposure amount calculation unit 9. gr301 to gr305 are calculated. Next, the frame exposure amount calculation unit 10 calculates an absolute gain ga300 shown in FIG. 5C from the exposure amount ea200 of the exposure amount peak frame calculated by the exposure amount calculation unit 9. The frame exposure amount calculation unit 10 adds the relative gains gr301 to gr305 and the absolute gain ga300 to obtain the gain of each frame.

  Then, before imaging the frame fr301 in FIG. 5D, the control unit 12 selects gr301 + ga300 as the gain setting in step st111, and performs imaging with the gain setting in step st112. Similarly, the controller 12 selects the gain setting shown in FIG. 5C and performs imaging before imaging the frames fr302 to fr305.

  Finally, the operation of automatic exposure control for each number of exposure calculation frames will be described. In step st113, when imaging of the number of exposure calculation frames is completed, in step st114, the exposure amount calculation unit 9 acquires the exposure amount of the exposure amount peak frame from the image data for which imaging has been completed. In step st110, the frame exposure amount calculation unit 10 calculates the accumulation time and gain of each frame in the exposure amount calculation processing unit. In step st111, the control unit 12 selects the accumulation time and gain setting of each frame calculated in step st110. In step st112, the control unit 12 performs main imaging. An example of the operation of steps st113, st114, st110, st111, st112 will be described in more detail with reference to FIG.

  6A shows a change in luminance of the light source, FIG. 6B shows a vertical synchronization signal input to the timing generation unit 7, and FIG. 6C shows each frame calculated by the frame exposure amount calculation unit 10. Indicates the gain value. 6D shows an image output from the horizontal scanning unit 6 captured with the gain shown in FIG. 6C, FIG. 6E shows the average luminance of each row of the output image, and FIG. ) Represents the exposure amount of each row of the output image.

  In FIGS. 6A to 6F, the number of exposure calculation frames is five. When the control unit 12 completes the imaging of the frames fr306 to fr310 in FIG. 6D in step st113, the exposure amount calculation unit 9 acquires the exposure amount peak frame exposure amount in step st114. 6A to 6F, the exposure amount peak frame is the frame fr308, and the exposure amount calculation unit 9 obtains the exposure amount ea301 in FIG. 6F as the exposure amount. At this time, if the appropriate exposure amount is e300, the exposure amount ea301 is too large. Therefore, in step st110, the frame exposure amount calculation unit 10 decreases the gain from the absolute gain setting ga301 to ga302 before imaging the frame fr311. In steps st111 and st112, the control unit 12 performs imaging of the frame fr311 using a gain obtained by adding the relative gain gr301 and the absolute gain ga302. Thereafter, similarly, the control unit 12 performs imaging of the frames fr312 to fr315 with respective gain settings in steps st111 and st112. When the control unit 12 completes imaging up to the frame fr315, the exposure amount calculation unit 9 acquires the exposure amount peak frame exposure amount again in step st114. The exposure amount calculation unit 9 acquires the exposure amount ea302 of the frame fr313. Since the exposure amount ea302 is too large for the appropriate exposure amount e300, the frame exposure amount calculation unit 10 reduces the gain from the absolute gain setting from ga302 to ga303 in step st110. Then, in steps st111 and st112, the control unit 12 picks up the frames fr316 to fr320 while selecting the gain setting of each frame calculated by the frame exposure amount calculation unit 10. In step st115, the image processing unit 11 performs various types of image processing (gradation conversion processing, contour enhancement processing, etc.) on the actually captured image.

  Although the operation has been described using an example in which exposure adjustment is performed only by gain control, the exposure adjustment means may be performed only by controlling exposure time (charge accumulation time), or gain control and exposure time control. May be used in combination.

  As described above, the solid-state imaging device according to the present embodiment includes the periodic exposure adjustment unit for the periodic luminance change (flicker) of the light source and the automatic exposure unit for each cycle. The exposure amount calculation unit 9 calculates the exposure amounts of a plurality of frames captured within the flicker cycle of the light source. The control unit 12 controls the exposure amount of each frame (photoelectric conversion charge accumulation time and / or amplification gain of the frame) captured by the pixel unit 1 based on the calculated exposure amounts of a plurality of frames. This makes it easy to suppress the influence of flicker even in moving image capturing at a high frame rate that reads out a plurality of images within a flicker cycle. Further, since exposure adjustment is performed in units of a plurality of sheets, the amount of exposure amount calculation for automatic exposure is reduced, and the power consumption of the solid-state imaging device can be suppressed.

(Second Embodiment)
The solid-state imaging device according to the second embodiment of the present invention further adjusts the gain of each row before image processing to the first embodiment, thereby further reducing the flicker effect in the frame plane.

  FIG. 7 is a block diagram illustrating an example of a configuration of a solid-state imaging apparatus according to the second embodiment of the present invention. Below, it demonstrates centering on the difference between this embodiment and 1st Embodiment. In the first embodiment, the exposure amount calculation unit 9 performs one frame exposure amount calculation from image data for a plurality of imaging frames output from the horizontal scanning unit 6, and from the exposure information, the frame exposure amount calculation unit 10 Calculates the accumulation time and gain of each frame in a plurality of frames. The image processing unit 11 performs various types of image processing using the image data output from the horizontal scanning unit 6 as an input. On the other hand, in the solid-state imaging device according to the present embodiment shown in FIG. 7, the line-unit gain correction unit 21 performs line-unit gain correction on the image data output from the horizontal scanning unit 6, and the image processing unit 11 Perform various image processing. The row unit gain correction unit 21 calculates a gain correction value for each row based on the exposure information calculated by the exposure amount calculation unit 9.

  Next, an example of the operation of the solid-state imaging device according to the present embodiment will be described with reference to the flowchart of FIG. 8 and FIG. 9, focusing on the differences between the present embodiment and the first embodiment. As in the first embodiment, in step st102, the exposure amount calculation unit 9 acquires the flicker frequency. Next, in step st105, the controller 12 performs exposure amount imaging. Next, in step st107, the exposure amount calculation unit 9 determines an exposure amount peak frame. Next, in step st108, the exposure amount calculation unit 9 acquires each frame exposure amount difference. Next, in step st109, the exposure amount calculation unit 9 acquires the peak frame absolute exposure amount. In step st110, the frame exposure amount calculation unit 10 performs each imaging frame accumulation time and gain calculation of the exposure amount calculation processing unit.

  In the second embodiment, gain correction is further performed in units of rows. Hereinafter, the gain correction value acquisition imaging of each row and the calculation of the row unit gain correction value using the same will be described prior to the main imaging. In the second embodiment, after step st110, in order to obtain each row gain correction value, in step st201, the control unit 12 selects the storage time and gain setting of each frame calculated in step st110, and in step st202. Then, each row gain correction value acquisition imaging is performed. Then, when the number of imaging frames reaches the number of exposure calculation frames in step st203, the exposure amount calculation unit 9 acquires each row exposure amount difference in step st204, and acquires the peak frame exposure amount in step st205. In step st206, the line unit gain correction unit 21 calculates the gain correction value of each line in each frame from the exposure amount difference of each line acquired in step st204. An example of the operation of steps st110 and st201 to st206 will be described in more detail with reference to FIG.

  9A shows a change in luminance of the light source, FIG. 9B shows a vertical synchronization signal input to the timing generation unit 7, and FIG. 9C shows each frame calculated by the frame exposure amount calculation unit 10. Indicates the gain value. 9D shows an image output from the horizontal scanning unit 6 imaged with the gain shown in FIG. 9C, FIG. 9E shows the average luminance of each row of the output image, and FIG. ) Represents the exposure amount of each row of the output image. FIG. 9G shows the gain correction value of each row calculated by the row unit gain correction unit 21.

  In step st110, the frame exposure amount calculation unit 10 calculates relative gains gr401 to gr405 shown in FIG. 9C from the exposure amount difference with respect to the exposure amount peak frame calculated by the exposure amount calculation unit 9. Next, the frame exposure amount calculation unit 10 calculates the absolute gain ga401 shown in FIG. 9C from the exposure amount of the exposure amount peak frame calculated by the exposure amount calculation unit 9. The frame exposure amount calculation unit 10 adds the relative gains gr401 to gr405 and the absolute gain ga401 to obtain the gain of each frame.

  Then, the controller 12 selects gr401 + ga401 as a gain setting in step st201 before imaging the frame fr401 in FIG. 9D, and performs imaging with the gain setting in step st202. Thereafter, similarly, in steps st201 and 202, the control unit 12 selects the gain setting shown in FIG. 9C before imaging the frames fr402 to fr405, and performs imaging for acquiring each row gain correction value. 9A to 9J, the number of exposure calculation frames is five. When the control unit 12 completes imaging of the frames fr401 to fr405 in FIG. 9D in step st203, the exposure amount calculation is performed in step st204. The unit 9 obtains each row exposure amount difference. Specifically, the exposure amount calculation unit 9 subtracts the spatial frequency component of the average luminance of each row in frame units from the average luminance of each row shown in FIG. 9 (e), and exposes each row shown in FIG. 9 (f). Find the difference in quantity. Subsequently, in step st205, the exposure amount calculator 9 acquires an exposure amount ea401 shown in FIG. 9F as the exposure amount of the exposure amount peak frame fr403. In step st206, the line unit gain correction unit 21 calculates the gain correction value of each line in each frame shown in FIG. 9G from the difference between the line exposure amounts acquired in step st204. The exposure amount ea401 of the peak frame fr403 corresponds to a gain of 1.

  Next, the operation of the main imaging and the row unit gain correction operation after the main imaging using the gain correction value of each row will be described. In step st207, the frame exposure amount calculation unit 10 calculates the accumulation time and gain of each imaging frame in the exposure amount calculation processing unit from each frame exposure amount difference acquired in step st108 and the peak frame exposure amount acquired in step st205. Is calculated. In step st208, the control unit 12 selects the accumulation time and gain setting of each frame calculated in step st207, and performs main imaging in step st209.

  The horizontal scanning unit 6 transfers the actually captured image data to the row unit gain correction unit 21. In step st210, the row unit gain correction unit 21 performs gain correction in units of rows using the gain correction value of each row in each frame calculated in step st206. In step st <b> 211, the image processing unit 11 performs various types of image processing on the image on which the line-unit gain correction is performed.

  An example of the operation of steps st207 to st211 will be described in more detail with reference to FIG. FIG. 9H shows an image obtained as a result of performing the line unit gain correction using the gain correction value of FIG. 9G, and FIG. 9I shows the average luminance of each line after the line unit gain correction. FIG. 9J shows the exposure amount of each row after the row unit gain correction obtained by subtracting the spatial frequency component from FIG. 9I.

  In step st207, as in step st110, the frame exposure amount calculation unit 10 calculates the absolute gain ga402 shown in FIG. 9C from the exposure amount of the exposure amount peak frame. The frame exposure amount calculation unit 10 adds the relative gains gr401 to gr405 and the absolute gain ga402 to obtain the gain of each frame.

  Then, before imaging the frame fr406 in FIG. 9D, the control unit 12 selects gr401 + ga402 as the gain setting in step st208, and performs imaging with the gain setting in step st209. The horizontal scanning unit 6 transfers the image data of the frame fr406 to the row unit gain correction unit 21. In step st210, the line-unit gain correction unit 21 performs gain correction on a line-by-line basis using the gain correction value of each line in each frame calculated based on the exposure data of the frames fr401 to fr405 shown in FIG. Thereafter, similarly, the control unit 12 performs imaging of the frames fr407 to fr410, and obtains frame fr406 'to fr410' of the image after gain correction in units of rows shown in FIG. In the frames fr406 'to fr410', the exposure amount of each frame and each row is constant as shown in FIGS. 9 (i) and (j).

  Finally, the operation of automatic exposure control for each number of exposure calculation frames will be described. In step st212, when imaging of the number of exposure amount calculation frames is completed, in step st213, the exposure amount calculation unit 9 acquires the exposure amount of the exposure amount peak frame from the image data for which imaging has been completed. In step st207, the frame exposure amount calculation unit 10 calculates the accumulation time and gain of each frame in the exposure amount calculation processing unit. In step st208, the control unit 12 selects the accumulation time and gain setting of each frame calculated in step st110, and performs main imaging in step st209. In step st210, the row unit gain correction unit 21 performs gain correction of each row in each frame. In step st211, the image processing unit 11 performs various types of image processing. This will be described in more detail with reference to FIG.

  In step st212, when imaging of 5 frames, which is the number of exposure amount calculation frames of the frames fr406 to fr410 shown in FIG. 9D, is completed, in step st213, the exposure amount calculation unit 9 performs the exposure amount ea402 of the light amount peak frame fr408. To get. In step st207, the frame exposure amount calculation unit 10 calculates the accumulation time and gain of each frame to obtain an absolute gain ga403 shown in FIG. 9C. In steps st208 and 209, the control unit 12 images the frames fr411 to fr415 with the gain setting obtained by adding the relative gains gr401 to gr405. In step st210, the row unit gain correction unit 21 performs the above-described row unit gain correction to obtain frames fr411 'to fr415'.

  As described above, the solid-state imaging device according to the second embodiment of the present invention has the periodic (exposure) gain adjustment means for the periodic luminance change (flicker) of the light source, and the period unit thereof. With automatic exposure means. Furthermore, the solid-state imaging device has a line-by-row exposure adjustment unit. This makes it easy to suppress the influence of flicker in the frame plane even in moving image capturing at a high frame rate that reads out a plurality of images within a flicker cycle.

(Third embodiment)
The solid-state imaging device according to the third embodiment of the present invention performs stepwise exposure adjustment with respect to the luminance change of the non-periodic light source with respect to the first embodiment, thereby changing the exposure amount between frames. Reduce. With reference to the drawings, a third embodiment of the present invention will be described focusing on differences from the first embodiment.

  The configuration of the solid-state imaging device according to the third embodiment of the present invention is the same as the configuration of the solid-state imaging device of the first embodiment shown in FIG. The present embodiment is different from the first embodiment in operation, and an example thereof will be described with reference to the flowchart of FIG. 10 and FIG.

  As in the first embodiment, in step st101, the control unit 12 acquires a flicker presence / absence determination image. Next, in step st102, the exposure amount calculation unit 9 acquires a flicker frequency. Next, in step st103, the exposure amount calculation unit 9 determines the presence or absence of flicker. If there is flicker, the operation after step st104 is performed. Since steps st104 to st115 are the same as those in the first embodiment, description thereof is omitted.

  When there is no flicker, the operation is different from that of the first embodiment. First, in step st301, the exposure amount calculation unit 9 determines the number of frames for which the exposure amount calculation process is performed. The number of frames for which the exposure amount calculation processing is performed is preferably set to the minimum time that can be processed by the exposure amount calculation unit 9, that is, the minimum number of frames in order to reduce a time lag in automatic exposure control described later.

  Next, imaging for initial exposure amount calculation performed prior to main imaging, acquisition of exposure amount, and calculation of exposure correction amount will be described. In steps st302 and st303, the control unit 12 repeats imaging for calculating the initial exposure amount, and performs imaging for the number of frames determined in step st301. In step st304, the exposure amount calculation unit 9 acquires the exposure amount of a part of the imaging frames in the image acquired in step st302. In step st305, the exposure amount calculation unit 9 calculates an exposure correction amount from the difference between the exposure amount and the target exposure amount. An example of the operations of steps st301 to st305 will be described in more detail with reference to FIG.

  11A shows a change in luminance of the light source, FIG. 11B shows a vertical synchronization signal input to the timing generation unit 7, FIG. 11C shows a gain used at the time of imaging, and FIG. Represents an image output from the horizontal scanning unit 6. FIG. 11E shows the surface average exposure amount of the output image.

  First, in step st301, the exposure amount calculator 9 determines the number of frames for the exposure amount calculation process from the time required for the exposure calculation and the frame rate. In FIG. 11, four frames are used as the number of frames for the exposure calculation processing. Next, in step st302, the control unit 12 performs imaging for calculating the initial exposure amount. The accumulation time not shown in FIG. 11 is the maximum value at the imaging frame rate, and the gain is imaged with the gain ga501 shown in FIG. 9C. And the control part 12 images four frames, frames fr501 to fr504, with the accumulation time and gain fixed.

  Thereafter, in step st304, the exposure calculation unit 9 acquires the absolute surface average exposure amount of the final frame fr504 of four frames, which is a unit of exposure amount calculation processing. In step st305, in FIG. 11E, the exposure calculation unit 9 obtains an insufficient exposure amount, that is, an exposure correction amount ec504 when the target exposure amount is set to e500.

  In order to reduce the time lag of exposure control, which will be described later, it is desirable that the frame used for the exposure calculation is the last frame or a frame close in time to the last frame among a plurality of frames that are units of the exposure calculation process.

  Subsequently, the exposure amount calculation of each frame in the number of exposure amount calculation frames applied to the main imaging and the operation of the main imaging will be described. In step st306, the frame exposure amount calculation unit 10 calculates the accumulation time and gain of each frame in the number of exposure amount calculation frames to be imaged next from the exposure correction amount and the accumulation time and gain when it is acquired. . In step st307, the control unit 12 selects the accumulation time and gain setting of each frame calculated in step st306, and performs main imaging in step st308. An example of the operations of steps st306 to st308 will be described in more detail with reference to FIG.

  In step st306, the frame exposure amount calculation unit 10 calculates from the exposure correction amount ec504 and the gain ga504 obtained when the exposure correction amount ec504 is obtained, and obtains the target gain gt508 in the frame fr508. Further, the frame exposure amount calculation unit 10 obtains gains of the frames fr505 to fr508 from the gains ga504 and gt508 by linear interpolation, and obtains gains ga505 to ga508. Then, the control unit 12 selects the gain ga505 in step st307 and performs imaging of the frame fr505 in step st308. Thereafter, similarly, the control unit 12 selects the gains ga506 to ga508, and performs imaging of the frames fr506 to fr508.

  Finally, the operation of automatic exposure control for each number of exposure calculation frames will be described. In step st309, when imaging of the number of exposure calculation frames is completed, in step st304, the exposure amount calculation unit 9 acquires the exposure amount from a part of the imaging frames of the image that has been imaged. Further, in step st305, the exposure amount calculation unit 9 calculates an exposure correction amount, and in step st306, calculation of the accumulation time and gain of each frame from the exposure correction amount. Then, the control unit 12 selects an accumulation time and gain setting for each frame in step st307, and performs main imaging in step st308. Automatic exposure control is performed by repeating these for each number of exposure calculation frames. An example of the automatic exposure control operation will be described with reference to FIG.

  In step st309, when the control unit 12 determines that imaging of 4 frames, which is the number of exposure amount calculation frames, has been completed, in step st304, the exposure amount calculation unit 9 calculates the surface average exposure amount of the final frame fr508 in the four frames. get. If the luminance of the light source is constant, gain correction is performed from the exposure amount of the frame fr 504, which matches the target exposure amount. However, as shown in FIG. 11A, the luminance of the light source greatly increases during imaging of the frame fr 507. Yes. As a result, as shown in FIG. 11E, the exposure amount of the frame fr508 greatly exceeds the target exposure amount e500, and the output image becomes an overexposed image. Therefore, in step st305, the exposure amount calculator 9 calculates an exposure correction amount ec508 for setting the exposure amount to the target exposure amount e500. In step st306, the frame exposure amount calculation unit 10 calculates the gains of the frames fr509 to fr512 to be captured next. If the luminance of the light source of the frame fr512 is the same as that of the frame fr508, in step st307, the control unit 12 gradually increases the gains ga509 to ga512 in the frames fr509 to fr512 so that the frame fr512 reaches the target exposure amount. Reduce. In step st308, the control unit 12 images the frames fr509 to fr512 with the gains ga509 to ga512.

  However, as shown in FIG. 11A, the luminance of the light source continues to increase during the imaging of the frames fr509 to fr512. As a result, the surface average exposure amount of the frame fr 512 obtained in step st304 still exceeds the target exposure amount e500. Therefore, the exposure calculation unit 9 and the frame exposure amount calculation unit 10 further decrease the gains of the frames fr513 to fr516 to be imaged next in steps st305 and st306.

  As shown in FIG. 11A, the luminance of the light source becomes substantially constant after the frame fr512, and as a result, the surface average exposure amount of the images of the frames fr513 to fr516 taken in the steps st307 and st308 becomes the target exposure amount e500. Converge.

  In the above description, in order to simplify the description, an example of performing exposure amount control only by gain control has been shown. However, exposure amount control may be performed only by accumulation time control. Further, both accumulation time control and gain control may be used.

  As described above, the solid-state imaging device according to the third embodiment of the present invention has the stepwise exposure adjustment means for the luminance change of the non-periodic light source, and the automatic exposure means in units of a plurality of frames. Have As a result, automatic exposure control at high frame rate shooting can be performed without depending on the capability of exposure calculation processing. Even if a video shot at a high frame rate is played back slowly, there is little change in the exposure amount between frames, so that a good video is obtained.

  The solid-state imaging devices according to the first to third embodiments can be applied to various applications such as video cameras, mobile terminal cameras, in-vehicle cameras, security cameras, and industrial cameras.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

1 pixel unit, 9 exposure amount calculation unit, 10 frame exposure amount calculation unit, 12 control unit

Claims (9)

A pixel unit that captures a plurality of frames by photoelectric conversion;
An exposure amount calculation unit for calculating an exposure amount with a plurality of frames captured within the flicker cycle of the light source as one unit;
On the basis of the exposure amount that is arithmetic, for a plurality of frames to be captured after the plurality of frames used for calculation, including at least a plurality of charge accumulation period of two mutually different, and / or different from each other A solid-state imaging device comprising: a control unit that controls an exposure amount of each frame imaged by the pixel unit by setting a plurality of amplification gains including at least two . The exposure amount calculation unit obtains an exposure amount of a frame having a peak exposure amount from the plurality of frames, and a difference between the exposure amount of the plurality of frames and an exposure amount of the frame having the peak exposure amount. The solid-state imaging device according to claim 1, wherein And a frame exposure amount calculation unit that calculates an exposure amount of each frame imaged by the pixel unit based on a difference between the exposure amount of the frame having the peak exposure amount and the exposure amounts of the plurality of frames. The solid-state imaging device according to claim 2 . The exposure amount calculation unit calculates the exposure amount of each row of the plurality of frames,
Wherein, the solid-state imaging device according to any one of claims 1 to 3, characterized in that to control the exposure amount of each row of each frame. The exposure amount calculation unit determines the presence or absence of a periodic luminance change of the light source based on the exposure amounts of the plurality of frames,
The control unit controls the exposure amount of each frame imaged by the pixel unit based on the exposure amounts of the plurality of frames with respect to the appropriate exposure amount when there is no periodic luminance change of the light source. The solid-state imaging device according to any one of claims 1 to 4 , wherein the solid-state imaging device is characterized. The calculation of the exposure amount by the exposure amount calculation unit includes calculation of the light intensity of the light source using the exposure time and the amplification gain used at the time of imaging from the data of a plurality of frames imaged by the pixel unit. The solid-state imaging device according to any one of claims 1 to 5 .   The control unit controls the exposure amount by controlling the charge accumulation time of the photoelectric conversion and / or the amplification gain of the frame so that an average value of a plurality of pixel data in one frame is included in a predetermined range. The solid-state imaging device according to claim 1, wherein the solid-state imaging device is performed. A pixel unit that captures a plurality of frames by photoelectric conversion;
An exposure amount calculation unit for calculating an exposure amount with a plurality of frames captured within the flicker cycle of the light source as one unit;
A control unit configured to set a plurality of charge accumulation periods including at least two different from each other for a plurality of frames imaged after the plurality of frames used in the calculation based on the calculated exposure amount. A solid-state imaging device. A pixel unit that captures a plurality of frames by photoelectric conversion;
An exposure amount calculation unit for calculating an exposure amount with a plurality of frames captured within the flicker cycle of the light source as one unit;
A control unit configured to set a plurality of amplification gains including at least two different from each other for a plurality of frames imaged after the plurality of frames used for the calculation based on the calculated exposure amount. A solid-state imaging device.

Images