JP7234015B2 - Imaging device and its control method - Google Patents

Description

The present invention relates to an imaging device and its control method.

2. Description of the Related Art There is a well-known shooting method that reduces the amount of light incident on an image sensor by reducing the transmittance of an imaging optical system when shooting in a very bright environment using an imaging device such as a digital camera. there is This shooting method is to realize a subject expression with a shallow depth of field by opening the aperture while preventing the saturation of the image sensor even in a bright environment, or by performing a long exposure to create a scene such as a waterfall. The purpose is to express the trajectory of the subject's movement. Conventionally, an optical filter such as an ND filter has been used in this imaging method.

Japanese Patent Application Laid-Open No. 2002-200002 discloses an imaging device that transfers charges converted by a photoelectric conversion unit to a storage unit multiple times and accumulates the transferred charges collectively, thereby changing conditions such as exposure time and exposure amount. disclosed.

JP 2010-157893 A

By applying the imaging apparatus disclosed in Patent Document 1 and performing short-time accumulation intermittently by dividing the exposure, it becomes possible to adjust the light amount (electronic ND) without using an optical filter. In addition, in video shooting, short accumulation periods are evenly distributed in one frame period, and accumulation is performed collectively a plurality of times in one frame period. You can get a natural moving image.

In electronic ND photography using the apparatus disclosed in Japanese Patent Application Laid-Open No. 2002-200003, one exposure is divided, so the carrier disappears when information is transferred to the memory for each exposure. Therefore, an error occurs with respect to the ideal luminance information obtained from the exposure ratio. The luminance information is information obtained by converting luminance into signal information. The error in the above luminance information increases when the subject luminance is low or when the number of exposure divisions is large. Also, in the photographing by the electronic ND, since only an image after the amount of light is reduced can be obtained, it is impossible to distinguish between the change in the brightness information of the object and the above-described error in the brightness information, and it is difficult to detect the error in the brightness information. is difficult. SUMMARY OF THE INVENTION An object of the present invention is to provide an imaging apparatus capable of highly accurately correcting luminance information of an image obtained by photographing with divided exposure.

An imaging apparatus according to an embodiment of the present invention acquires a first image by imaging by time-division exposure with a first division number, and acquires a second image by imaging by time-division exposure with a second division number. imaging means for acquiring; control means for performing control for correcting the first image obtained by imaging by the imaging means according to the difference in luminance information between the first image and the second image; Prepare.

According to the image pickup apparatus of the present invention, it is possible to correct the luminance information of an image obtained by photographing with divided exposures with high accuracy.

It is a figure which shows the external appearance of an imaging device. 1 is an example of a functional block diagram of an imaging device; FIG. It is a figure which shows an example of a structure of an image pick-up element. FIG. 4 is a diagram for explaining accumulation and readout in an image sensor; It is a figure explaining the thinning|decimation operation|movement of the number of pixels from an image pick-up element. FIG. 4 is a diagram showing a first driving sequence of the imaging device; FIG. 10 is a diagram showing a second driving sequence of the imaging element; It is a figure explaining an example of the structure which detects the difference of luminance information. 4 is a flowchart for explaining an example of operation processing of an imaging device; FIG. 10 is a diagram showing the relationship between the number of divisions of exposure at the time of photographing and luminance information; It is a figure explaining the effect by control of the division|segmentation number of exposure. FIG. 10 is a diagram showing a specific example of the relationship between the number of divisions of exposure at the time of photographing and luminance information; FIG. 4 is a diagram showing a driving sequence of an imaging element; It is a figure explaining an example of the structure which detects the difference of luminance information. It is a figure explaining the effect of 2nd Embodiment. FIG. 4 is a diagram showing an example of luminance information of pixel n and pixel m; FIG. 4 is a diagram showing image sizes according to shooting modes; FIG. 4 is a diagram showing an example of an in-screen area and an out-of-screen area in a moving image shooting mode; FIG. 10 is a diagram showing the number of exposure divisions and luminance information according to environmental temperature; FIG. 10 is a diagram showing the number of exposure divisions and luminance information according to environmental temperature;

(First embodiment)
FIG. 1 is a diagram showing the appearance of an imaging device according to this embodiment.
The imaging device shown in FIG. 1 is a digital camera. FIG. 1A shows the front of the imaging device. FIG. 1B shows the rear surface of the imaging device. The main unit 151 is the main body of the imaging device, and accommodates therein an imaging device, a shutter device, a recording medium, and the like. The imaging optical system 152 has an aperture and a lens inside. A movable display unit 153 displays various types of information such as shooting information and images. The display unit 153 has a display luminance range that can display an image with a wide dynamic range without suppressing the luminance range.

The switch ST154 is an operation member mainly used for shooting still images. A switch MV155 is an operation member used to start and stop moving image shooting. A shooting mode selection lever 156 is an operation member used to select a shooting mode. The menu button 157 is an operation member used to shift to an operation mode (function setting mode) for setting functions of the imaging apparatus. Up-down switches 158 and 159 are operating members used to change various set values. A dial 160 is an operating member used to change various set values. The playback button 161 is an operation member used to switch to an operation mode (playback mode) in which video recorded on a recording medium housed in the main body 151 is played back on the display 153 .

FIG. 2 is an example of a functional block diagram of an imaging device.
The imaging device has an imaging optical system 152 to a wireless I/F 198 . The imaging optical system 152 forms an optical image of a subject on the imaging device 184 . The imaging device 184 photoelectrically converts the optical image of the subject formed via the imaging optical system 152 into an electrical video signal. An optical axis 180 is the optical axis of the imaging optical system 152 . A diaphragm 181 adjusts the amount of light passing through the imaging optical system 152 . The aperture control section 182 controls the aperture 181 . The optical filter 183 limits the wavelength of light incident on the imaging element 184 and the spatial frequencies transmitted to the imaging element 184 . The imaging device 184 has sufficient number of pixels, signal readout speed, color gamut, and dynamic range to meet the Ultra High Definition Television standard. The digital signal processing unit 187 performs various corrections on the digital video data output from the imaging element 184, and then compresses the video data. The timing generator 189 outputs various timing signals to the imaging element 184 and the digital signal processor 187 .

A system control CPU 178 controls various calculations and the entire imaging apparatus. The video memory 190 temporarily stores image data (video data). A display I/F 191 is an interface for displaying an image on the display unit 153 . The recording medium 193 is storage means for recording video data, additional data, and the like. The recording medium 193 is, for example, a removable semiconductor memory or the like. The recording I/F 192 is an interface for recording data on the recording medium 193 or reading data from the recording medium 193 . The external I/F 196 is an interface for communicating with the external computer 197 and the like. A print I/F 194 is an interface for outputting video to the printer 195 . A wireless I/F 198 is an interface for communicating with the network 199 . The switch input section 179 includes a plurality of operating members such as the switch ST154 and the switch MV155.

FIG. 3 is a diagram showing an example of the configuration of an imaging device.
The imaging device 184 includes a plurality of pixels (pixel units) having photoelectric conversion units (photodiodes) arranged two-dimensionally. In FIG. 3, of the plurality of pixel portions of the image sensor 184, the pixel portion of the first row and first column (1, 1) and the last row of the m row and first column (m, 1) are shown. . Since the pixel portion of the first row and first column (1, 1) and the pixel portion of the m row and first column (m, 1) have the same configuration, the constituent elements are numbered with the same numbers.

One pixel portion includes a photodiode 500, a first transfer transistor 501A, a first signal holding portion 507A, a second transfer transistor 502A, a third transfer transistor 501B, a second signal holding portion 507B, and a fourth transfer transistor. It has a transfer transistor 502B. That is, in the example shown in FIG. 3, the pixel section has a plurality of (for example, two) signal holding sections 507A and 507B for one photodiode 500 . The basic structure of the imaging device 184 having a signal holding unit is disclosed in Japanese Patent Application Laid-Open No. 2002-200013, and therefore the description thereof is omitted.

The pixel portion also includes a fifth transfer transistor 503 , a floating diffusion region 508 , a reset transistor 504 , an amplification transistor 505 and a selection transistor 506 .

The first transfer transistor 501A is controlled by a transfer pulse φTX1A. The second transfer transistor 502A is controlled by a transfer pulse φTX2A. The third transfer transistor 501B is controlled by a transfer pulse φTX1B. Also, the fourth transfer transistor 502B is controlled by a transfer pulse φTX2B. Also, the reset transistor 504 is controlled by a reset pulse φRES. The selection transistor 506 is controlled by a selection pulse φSEL. Also, the fifth transfer transistor 503 is controlled by a transfer pulse φTX3. Each control pulse is sent from a vertical scanning circuit (not shown). 520 and 521 denote power lines, and 523 denotes a signal output line.

FIG. 4 is a diagram for explaining accumulation and readout in the image sensor.
The operation in the all-pixel readout mode for reading out image signals from all the pixel units of the image sensor will be described with reference to FIG. Accumulation of the image sensor is to transfer the charge generated in the photodiode 500 to the signal holding units 507A and 507B. Further, readout of image signals is outputting the charges held in the signal holding units 507A and 507B to the outside of the image sensor 184 via the floating diffusion region 508 .

In FIG. 4, the horizontal axis indicates the passage of time. The horizontal axis indicates how all pixel signals are captured continuously. The vertical axis indicates the order of vertical scanning. Although eight rows are shown in the drawing for the sake of convenience, since the actual imaging device has several thousand rows, the last row is m rows. The direction of the vertical axis indicates that the lines are sequentially scanned line by line from the first line to the m-th line. The time Tf enclosed by the oblique solid line in the figure indicates the charge accumulation time for each row. The dashed line indicates the readout of the accumulated charge.

FIG. 5 is a diagram for explaining the operation of thinning out the number of pixels from the image sensor.
The thinning operation is an operation of thinning out the number of pixels from the image sensor and reading out an image signal. In general, if a reproduced moving image has a kind of frame-by-frame flicker, the quality of the moving image is greatly lost. In order to prevent moving images from appearing jerky, it is necessary to set an accumulation time close to one frame period in a series of shootings. That is, when the frame rate is 30 frames/second, an accumulation time of 1/30 seconds is appropriate. In an imaging device having several thousand rows of pixels, when an image is stored and all rows are read out, the number of pixels is thinned out and read out in order to reduce the time required for readout.

In FIG. 5, the horizontal axis indicates the passage of time, and the vertical axis indicates the order of vertical scanning. The number of lines of interlaced scanning is set so that the operation is completed within one imaging cycle. In the example shown in FIG. 5, unhatched rows are moving image readout rows, and hatched rows are thinning rows. The relationship between readout rows and thinned rows may be changed according to the frame rate.

For example, the image associated with the image signal acquired from the readout row is assumed to be the main image (first image). This image is an image output for display or recording. The row from which the main image is acquired is assumed to be na row. That is, the main image is acquired from the pixel portion included in the area (first area) including all the na rows. Also, for example, an image related to an image signal acquired from a thinned row is set as a reference image (second image). A reference image is an image used to detect differences in luminance information. Let the row from which the reference image is acquired be nb rows. That is, the reference image is acquired from the pixel portion included in the area (second area) including all nb rows.

FIG. 6 is an example of a timing chart showing the first driving sequence of the imaging device.
Assume that a moving image is shot with a shutter speed of 1/30 second. The imaging device divides (time-divisions) the accumulation (exposure) in one frame by a first division number (8 in FIG. 6), and each divided exposure related to the exposure time-divided by the first division number Accumulate over time. The imaging device obtains an image signal related to the main image by performing eight accumulations in one frame. In this example, the first division number is initialized to a larger value than the second division number, which will be described later.

As shown in FIG. 5, the imaging device 184 has a plurality of pixel columns in the vertical direction. FIG. 6 shows the timing of the control operation for the na-th row shown in FIG. A control operation similar to this control operation is vertically scanned by the horizontal synchronizing signal φH, so that the accumulation operation of all the pixels of the imaging device 184 is performed.

The rise time t1 of the vertical synchronization signal φV indicates the time when one shooting cycle (shooting period), that is, one frame starts. 1/30 second, which is the time from t1 at which one frame is started to t6 at which the image signal is read, corresponds to one imaging cycle. As for the shooting conditions, 8 exposures of 1/960 seconds are accumulated and added during one shooting cycle of 1/30 seconds to obtain an exposure amount equivalent to 1/120 seconds, which is equivalent to using an ND filter. is obtained. The imaging device 184 is controlled by the timing generator 189 and the system control CPU 178 during one shooting period.

At time t1, the vertical synchronizing signal φV becomes high level, and at the same time, the horizontal synchronizing signal φH
becomes high level. At time t2, when the transfer pulse φTX2A(na) of the na-th row becomes high level, the second transfer transistor 502A of the na-th row is turned on. At this time, since the reset pulse φRES(na) of all rows has already become high level and the reset transistors 504 are turned on, the floating diffusion region 508 and the first signal holding unit 507A of the na-th row are reset. be. At time t2, the selection pulse φSEL(na) for the na-th row is at low level.

At time t31, when the transfer pulse φTX3(na) of the na-th row becomes low level, the third transfer transistor 503 is turned off, the reset of the photodiode 500 of the first row is released, and the charge is transferred to the photodiode 500. Accumulation is started. In the following description, signal charges are simply referred to as charges.

At time t41, when the transfer pulse φTX1(na) of the na-th row becomes high level, the first transfer transistor 501A is turned on, and the charge accumulated in the photodiode 500 becomes a signal holding signal holding the charge of the na-th row. transferred to section 507A. At time t51, when the transfer pulse φTX1A (na) of the first row becomes low level, the third transfer transistor 501A is turned off, and transfer of the charge accumulated in the photodiode 500 to the signal holding portion 507A is completed. At the same time, the transfer pulse φTX3(na) of the na-th row becomes high level, the third transfer transistor 503 is turned on, the photodiode 500 of the na-th row is reset, and the charge accumulation in the photodiode 500 is completed. .

The period from time t31 to time t51 corresponds to one accumulation time of 1/960 seconds in one imaging cycle, and is illustrated as an accumulation time 602-1 in the shaded area extending upward to the right. The accumulation operation as described above is discretely performed eight times. Accumulation times (divided exposure times) corresponding to the respective accumulation operations are shown as accumulation times 602-1 to 602-8 in areas with diagonal lines extending upward to the right. Note that the control operation in the accumulation times 602-2 to 602-8 is the same as the control operation in the accumulation time 602-1, so the explanation is omitted.

At time t6, when the reset pulse φRES(1) of the na-th row becomes low level, the reset transistor 504 of the na-th row is turned off, and the reset state of the floating diffusion region 508 is released. At the same time, when the transfer pulse φTX2A (na) for the na-th row and the selection pulse φSEL (na) for the na-th row become high level, the transfer transistor 502A and the selection transistor 506 for the na-th row are turned on to turn on the na-th row. image signals can be read out. Further, an output corresponding to the change in the potential of the floating diffusion region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506 and supplied to a readout circuit (not shown) to produce the image of the na-th row. Output to the outside as a signal. At time t6, the vertical synchronizing signal φV becomes high level, and the next photographing cycle is started.

FIG. 7 is an example of a timing chart showing the second driving sequence of the imaging element.
The imaging device divides the exposure by the second division number (1 in FIG. 7), accumulates in the divided exposure time after division, and obtains an image signal related to the reference image by accumulation in one frame. The reference image is used to detect the difference in luminance information from the main image.

A rise time t1+Δt of φSEL(nb) in FIG. 7 indicates a time obtained by adding an arbitrary time increment to the time when the imaging cycle (one frame) starts at t1 shown in FIG. Each time shown in FIG. 7 is described with +Δt for all times to indicate the state in which this time increment is added. The reason why the time increment is added is that in moving image shooting, the images acquired from the 1st row to the m-th row in one shooting cycle are read in order. 1/30 second, which is the time from t1+.DELTA.t when the frame starts to t6+.DELTA.t when the image signal is read out, corresponds to one imaging cycle.

For example, the imaging apparatus obtains an exposure amount equivalent to 1/120 second, which is equivalent to using an ND filter, by accumulating once in 1/120 second during one shooting period of 1/30 second. The imaging device 184 is controlled by the timing generating section 189 and the system control CPU 178 during one shooting cycle. At time t1, the vertical synchronizing signal φV goes high, and at the same time the horizontal synchronizing signal φH goes high. At time t2+Δt, when the transfer pulse φTX2(nb) of the first row becomes high level, the second transfer transistor 502A of the m-th row is turned on. At this time, since the reset pulse φRES(nb) of all rows has already become high level and the reset transistors 504 are turned on, the floating diffusion region 508 and the first signal holding section 507A of the nb-th row are reset. be. At time t2+Δt, the selection pulse φSEL(nb) for the nb-th row is at low level.

At time t3+Δt, when the transfer pulse φTX3(nb) of the nb-th row becomes low level, the third transfer transistor 503 is turned off, the reset of the photodiode 500 of the nb-th row is released, and the charge is transferred to the photodiode 500. Accumulation is started.

At time t4+Δt, when the transfer pulse φTX1(nb) for the nb-th row becomes high level, the first transfer transistor 501A is turned on, and the charge accumulated in the photodiode 500 changes to the signal holding signal holding the charge for the m-th row. transferred to section 507A. At time t5+Δt, when the transfer pulse φTX1(nb) of the nb-th row becomes low level, the third transfer transistor 501A is turned off, and transfer of the charge accumulated in the photodiode 500 to the signal holding portion 507A is completed. At the same time, the transfer pulse φTX3(nb) of the nb-th row becomes high level, the third transfer transistor 503 is turned on, the photodiode 500 of the nb-th row is reset, and charge accumulation in the photodiode 500 is completed.

The period from time t3+Δt to time t5+Δt corresponds to one accumulation time of 1/120 second in one imaging period, and is illustrated as accumulation time 602 in the region of the diagonally shaded area extending upward to the right. At time t6+Δt, when the reset pulse φRES(nb) of the nb-th row becomes low level, the reset transistor 504 of the nb-th row is turned off, and the floating diffusion region 508 is released from the reset state. At the same time, when the nb-th row transfer pulse φTX2A (nb) and the nb-th row selection pulse φSEL (nb) become high level, the nb-th row transfer transistor 502A and the selection transistor 506 are turned on, and the nb-th row image signals can be read out. Further, an output according to the change in the potential of the floating diffusion region 508 is read out to the signal output line 523 through the amplification transistor 505 and the selection transistor 506, supplied to a readout circuit (not shown), and the image of the nb-th row is read out. Output to the outside as a signal. At time t6+.DELTA.t, the vertical synchronizing signal .phi.V goes high, and the next imaging cycle starts.

FIG. 8 is a diagram illustrating an example of a configuration for detecting a difference in luminance information.
The digital signal processing section 187 has a first signal processing section 187a and a second signal processing section 187b. The first signal processing unit 187a generates the main image. The second signal processing unit 187b generates a reference image. The system control CPU 178 also has an output image generation section 178 b and an image comparison processing section 178 . The output image generation unit 178b executes output processing of the main image as video. The image comparison processing unit 178 compares the main image and the reference image. The timing generator 189 has a first division number control section 189a and a second division number control section 189b. The first division number control unit 189a controls the first division number, that is, the exposure division number when acquiring the main image. The second division number control unit 189b controls the second division number, that is, the exposure division number when acquiring the reference image.

The timing generator 189 controls the number of exposure divisions, and the imaging device 184 converts the formed optical image of the subject into an electrical video signal. The converted video signal is input to the digital signal processing section 187 . An image signal obtained by exposure divided by the first division number is input to the first signal processing unit 187a that generates the main image. Also, an image signal obtained by exposure divided by the second division number is input to the first signal processing unit 187b that generates a reference image. The digital signal processing unit 187 performs various corrections on the input video signal, and converts the main image used for detecting the difference in video output and luminance information to the reference image used for detecting the difference in luminance information. An image is generated. The generated main image and reference image are input to the system control CPU 178 . An image comparison processing unit 178a in the system control CPU 178 compares the main image and the reference image, and calculates the difference in luminance information.

When the luminance information difference (luminance information difference) exceeds the reference (predetermined range), the image comparison processing unit 178a transmits control information for correcting the luminance information difference to the timing generation unit 189 . The timing generation unit 189 changes the first division number based on the control information received from the image comparison processing unit 178a, and reflects the changed first division number on the imaging device 184. FIG. Specifically, the timing generation unit 189 corrects the luminance information of the main image obtained after the next imaging cycle (after the imaging period) by reducing the first division number. Thereby, the luminance information difference can be kept within a predetermined range. The first signal processing unit 187a performs various image processing on the signal output from the imaging element 184 after the exposure division number is changed, and generates the main image. The generated main image is input to the system control CPU 178, and output processing as a video is performed by the output image generation unit 178b.

FIG. 9 is a flowchart illustrating an example of operation processing of the imaging device.
The operation processing shown in FIG. 9 is realized by the control of the system control CPU 178 and the timing generator 189. FIG.

The process starts at S101. In S102, the system control CPU 178 calculates average luminance information a of the main image. Also, in S103, the system control CPU 178 calculates the average luminance information b of the reference image. In S104, the system control CPU 178 obtains the difference |ab| Subsequently, the system control CPU 178 determines whether the luminance information difference E exceeds the threshold C in S105. If the luminance information difference E does not exceed the threshold C, the process proceeds to S107. At S107, the process ends. If the luminance information difference E exceeds the threshold C, the process proceeds to S106.

In S106, the system control CPU 178 controls the timing generator 189 to reduce the number of exposure divisions in the next imaging cycle and thereafter. As a result, the number of exposure divisions is fed back by image comparison, and the main image acquired in the next imaging cycle and thereafter is corrected so as to approach an image with a smaller number of divisions. For example, in the case of moving image shooting, image storage for the next shooting cycle has already started at the time of image comparison, so the number of exposure divisions is fed back for the next shooting cycle.

FIG. 10 is a diagram showing the relationship between the number of divisions of exposure at the time of shooting and luminance information.
10A and 10B differ in the brightness of the photographed subject. K1 and K2 indicate average luminance information when the exposure division number is one. The dashed-dotted lines K1L and K2L represent ideal lines where luminance information does not change even if exposure is divided. The range of values from +dK1 to -dK1 and from +dK2 to -dK2 indicate the allowable range of change in luminance information. The brightness of the subject corresponding to FIG. 10A is brighter than the subject corresponding to FIG. 10B, and the average brightness information K1 and K2 have the following relationship.
K1 > K2

The solid lines in FIGS. 10A and 10B indicate that when the exposure is divided, the luminance decreases as the number of divisions increases. This is because the carrier disappears when information is transferred to the memory for each divided exposure, and an error occurs with respect to the ideal luminance information obtained from the exposure ratio. Comparing FIG. 10A and FIG. 10B, it can be seen that the lower the subject brightness, the greater the change in brightness information due to exposure division. This is because the proportion of carriers lost due to transfer to the carriers accumulated due to exposure increases. Since the generated image is obtained after the brightness information has been lowered, there is a problem that the change in the brightness information of the subject cannot be distinguished from the change in the brightness information due to the division of the exposure.

FIG. 11 is a diagram for explaining the effect of controlling the number of exposure divisions in this embodiment.
FIG. 11A shows the main image obtained by exposure divided by the first division number. FIG. 11B shows a reference image acquired with exposures divided by the second division number. FIG. 11C shows the main image after correction. In FIGS. 11(A) to 11(C), the interval between diagonal lines representing the image corresponds to the magnitude of luminance information. By comparing the main image in FIG. 11A and the reference image in FIG. 11B, it is possible to detect changes in luminance information due to division of exposure while distinguishing from changes in luminance of the subject. Then, from the detection result of the luminance information, it is possible to determine an appropriate number of divisions so that the change in the luminance information falls within the permissible range, and output the image shown in FIG. 11C.

FIG. 12 is a diagram showing a specific example of the relationship between the number of divisions of exposure at the time of shooting and luminance information.
It is assumed that the first division number corresponding to FIG. 11 described above is 66, and the second division number is 1. In FIG. From the graph shown in FIG. 12, if the exposure division number is less than about 32, it falls within the allowable range. When the number of divisions is about 66, the luminance information is K3a, which is out of the allowable range. If the imaging apparatus performs control to reduce the number of exposure divisions, for example, if the first division number is set to 24, the brightness information becomes K3b, and the change in brightness information falls within the allowable range.

It is preferable that the imaging device changes the number of divisions of exposure to the largest possible number of divisions as long as the change in luminance information is within the allowable range. If the number of exposure divisions is small, the expression of moving objects becomes unnatural. can correct luminance information.

As described above, according to the imaging apparatus of the present embodiment, when the change in luminance information does not fall within the allowable range, it is possible to obtain an appropriate number of divisions and correct the image. Also, when the change in luminance information is within the allowable range, the imaging device does not perform control to decrease the number of divisions. Therefore, according to the present embodiment, it is possible to prevent the expression of a moving object from becoming unnatural without reducing the number of divisions more than necessary. In addition, in this embodiment, changes in luminance information are obtained based on information from the image sensor used for imaging, so there is no need to separately provide a luminance sensor or the like for measuring luminance information, and the camera configuration does not become complicated. . As shown in FIG. 3, one pixel unit included in the image sensor has two signal holding units, but the first embodiment can be realized with one signal holding unit. Of course, two signal holding units may be used, and the charges related to the main image and the charges related to the reference image may be accumulated in different signal holding units.

In this embodiment, the number of exposure divisions is changed to correct the luminance information of the image. The amount may be decreased to correct the luminance information of the image. The imaging apparatus changes the total exposure amount for one imaging cycle by decreasing the accumulation times 602-1 to 602-8 in FIG. 6, for example.

(Second embodiment)
Next, a second embodiment will be described. Parts similar to those in the first embodiment are denoted by the same numbers, and descriptions thereof are omitted. The image pickup apparatus of the second embodiment differs from the image pickup apparatus of the first embodiment in that the image pickup apparatus stores charges obtained by different exposure division numbers in a plurality of (for example, two) signal holding units in the pixel unit. It is a point to accumulate. As described with reference to FIG. 3, one pixel portion includes a first signal holding portion 507A and a second signal holding portion 507B. By applying these two signal holding units, the main image and the reference image can be acquired within one imaging period. The first signal holding unit 507A functions as a first storage unit that stores the main image. Also, the second signal holding unit 507B functions as a second storage unit that stores the reference image. By acquiring two images obtained by exposure divided by different division numbers using two signal holding units, it is possible to compare luminance information for each pixel.

FIG. 13 is a timing chart showing the driving sequence of the imaging device.
Assume that a moving image is shot with a shutter speed of 1/30 second. An imaging device obtains an image signal by adding accumulations in one frame.

As shown in FIG. 5, the imaging device 184 has a plurality of pixel columns in the vertical direction. FIG. 13 shows timings of control operations for the na-th row shown in FIG. A control operation similar to this control operation is vertically scanned by the horizontal synchronizing signal φH, so that the accumulation operation of all the pixels of the imaging device 184 is performed.
The rise time t1 of the vertical synchronization signal φV indicates the time when one shooting cycle (one frame) starts. Also, 1/30 second, which is the time from t1 at which one frame is started to t9 at which the image signal is read, corresponds to one imaging cycle. The exposure division number corresponding to the image generated by the first signal holding unit 507A is eight. By accumulating and adding 8 exposures of 1/960 seconds during one shooting cycle of 1/30 seconds, the imaging device obtains an exposure amount equivalent to 1/120 seconds, which is equivalent to using an ND filter. Obtainable. The number of exposure divisions for the image generated by the second signal holding unit 507B is 1, and one continuous accumulation of 1/120 second is performed in one frame.

At time t1, the vertical synchronizing signal φV goes high, and at the same time the horizontal synchronizing signal φH goes high. At time t2, when the transfer pulses φTX2A(na) and φTX2B(na) of the na-th row become high level, the second transfer transistors 502A and 502B of the na-th row are turned on. At this time, the reset pulse φRES(na) of all rows has already become high level, and the reset transistors 504 are turned on. Therefore, the na-th row floating diffusion region 508, the first signal holding section 507A and the second signal holding section 507B are reset. At time t2, the selection pulse φSEL(na) for the na-th row is at low level.

At time t31, when the transfer pulse φTX3(na) of the na-th row becomes low level, the third transfer transistor 503 is turned off, the reset of the photodiode 500 of the first row is released, and the charge is transferred to the photodiode 500. Accumulation is started. At time t41, when the transfer pulse φTX1A (na) of the na-th row becomes high level, the first transfer transistor 501A is turned on, and the charge accumulated in the photodiode 500 is transferred to the first transistor holding the charge of the na-th row. It is transferred to the signal holding unit 507A.

At time t51, when the transfer pulse φTX1A (na) of the first row becomes low level, the third transfer transistor 501A is turned off, and the transfer of the charge accumulated in the photodiode 500 to the first signal holding portion 507A is completed. do. At the same time, the transfer pulse φTX3(na) of the na-th row becomes high level, the third transfer transistor 503 is turned on, the photodiode 500 of the na-th row is reset, and the charge accumulation in the photodiode 500 is completed. .

The period from time t31 to time t51 corresponds to one accumulation time of 1/960 seconds in one imaging cycle, and is illustrated as an accumulation time 602-1 in the shaded area extending upward to the right. The accumulation operation as described above is discretely performed eight times. Accumulation times (divided exposure times) corresponding to the respective accumulation operations are shown as accumulation times 602-1 to 602-8 in areas with diagonal lines extending upward to the right. Note that the control operation in the accumulation times 602-2 to 602-8 is the same as the control operation in the accumulation time 602-1, so the explanation is omitted.

At time t6, when the reset pulse φRES(1) of the na-th row becomes low level, the reset transistor 504 of the na-th row is turned off, and the reset state of the floating diffusion region 508 is released. At the same time, when the transfer pulse φTX2A (na) of the na-th row and the selection pulse φSEL (na) of the na-th row become high level, the transfer transistor 502A of the na-th row and the selection transistor 506 of the na-th row are turned on. It becomes possible to read out the image signal of the na-th row.

In the first signal holding section 507A, a signal corresponding to the change in the potential of the floating diffusion region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506. FIG. A signal read out to the signal output line 523 is supplied to a readout circuit (not shown) and output to the outside as image information related to the main image. At time t7, when the transfer pulse φTX3(na) of the na-th row becomes low level, the third transfer transistor 503 is turned off, the reset of the photodiode 500 of the first row is released, and the charge is transferred to the photodiode 500. Accumulation is started. The period from time t7 to time t9 corresponds to one accumulation time of 1/120 second in one imaging cycle, and is illustrated as accumulation time 603-1 in the region of the diagonally shaded area extending upward to the right.

At time t8, when the transfer pulse φTX1B(na) of the na-th row becomes high level, the first transfer transistor 501B is turned on, and the charge accumulated in the photodiode 500 is transferred to the second transistor holding the charge of the na-th row. It is transferred to the signal holding unit 507B. At time t9, when the transfer pulse φTX1B (na) of the first row becomes low level, the third transfer transistor 501B is turned off, and the transfer of the charges accumulated in the photodiode 500 to the second signal holding portion 507B is completed. do. At the same time, the transfer pulse φTX3(na) of the na-th row becomes high level, the third transfer transistor 503 is turned on, the photodiode 500 of the na-th row is reset, and the charge accumulation in the photodiode 500 is completed. .

At time t10, when the reset pulse φRES(1) of the na-th row becomes low level, the reset transistor 504 of the na-th row is turned off, and the reset state of the floating diffusion region 508 is released. At the same time, when the transfer pulse φTX2B (na) for the na-th row and the selection pulse φSEL (na) for the na-th row become high level, the transfer transistor 502B for the na-th row and the selection transistor 506 for the na-th row are turned on. , na-th row image signals can be read out.

In the second signal holding section 507B, a signal corresponding to the change in the potential of the floating diffusion region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506. FIG. A signal read out to the signal output line 523 is supplied to a readout circuit (not shown) and output to the outside as image information related to the reference image. At time t10, the vertical synchronizing signal φV becomes high level, and the next photographing cycle is started.

FIG. 14 is a diagram illustrating an example of a configuration for detecting a difference in luminance information;
The description will focus on differences from the first embodiment described with reference to FIG.
The main image generated by the first signal processing section 187 and the reference image generated by the second signal processing section 187b are input to the image comparison processing section 178a of the system control CPU 178. FIG. The image comparison processing unit 178a compares the main image and the reference image, and calculates the difference in luminance information. When the luminance information difference (luminance information difference) exceeds the reference (predetermined range), the image comparison processing unit 178a transmits control information for correcting the luminance information difference to the output image generation unit 178b. The output image generator 178b corrects the luminance information of the main image based on the control information received from the image comparison processor 178a. As a result, the main image acquired in the current imaging cycle is corrected so as to approach an image with a smaller number of divisions.

FIG. 15 is a diagram for explaining the effects of the second embodiment.
FIG. 15A shows the main image generated by the first signal processing section 187a. FIG. 15B shows a reference image generated by the second signal processing section 187b. FIG. 15C shows the main image after correction. In FIGS. 15(A) to 15(C), the interval between diagonal lines representing images corresponds to the magnitude of luminance information.

(na), (ma), (nb), and (mb) in each image shown in FIGS. 15A and 15B indicate pixel positions. The positional relationship of pixels is the same for (na) and (nb), and for (ma) and (mb).

If the number of exposure divisions is large, luminance information will decrease due to carrier loss. Therefore, when the brightness information X(na), X(nb), X(ma), and X(mb) of each pixel of the image are compared, the following relationships are obtained.
X(na)<X(nb) Expression (1)
X(ma)<X(mb) Expression (2)

Let (K) be the image generated without dividing the exposure, let (nK) and (mK) be the corresponding pixels, respectively, and let X(nK) and X(mK) be the luminance information of the respective pixels. Assuming ideal luminance information, the relationships shown in the following equations (3) and (4) hold.
It can be seen that pixels X(nb) and X(mb), which are less affected by carrier loss due to exposure division, are closer to the ideal luminance information X(nK) and X(mK).
X(na)<X(nb)≈X(nK) Expression (3)
X(ma)<X(mb)≈X(mK) Equation (4)

In the second embodiment, the image comparison processing unit 178a compares the luminance information for each corresponding pixel in the main image shown in FIG. 15A and the reference image shown in FIG. Calculate the difference between The image comparison processing unit 178a generates control information for correcting the difference in luminance information based on the calculated difference in luminance information. The output image generation unit 178b changes the magnification of the luminance information of the main image based on the generated control information, and outputs an image corrected to proper luminance information. As a result, the main image can be corrected so as to approach an image with a small number of divisions.

FIG. 16 is a diagram showing an example of luminance information of pixel n and pixel m.
As shown in FIG. 16A, it is assumed that the luminance information of the (na) pixel is 100, the luminance information of the (nb) pixel is 200, and the luminance information of the (nK) pixel is 205. FIG. (na) By multiplying the luminance information of the pixels by approximately 2.0, the image can be corrected so as to approximate the proper luminance information 205 . Also, for example, as shown in FIG. Suppose. By increasing the magnification of the luminance information to approximately 1.2 times, the image can be corrected so as to approximate the proper luminance information 65 . In the example described above, the imaging device compares the luminance information of each pixel of the main image and the reference image, and corrects the magnification of the luminance information for the difference in the luminance information. The imaging device compares luminance information for each row of the main image and the reference image, calculates the difference in luminance information, and amplifies the charge in the imaging element 184 according to the difference in luminance information to correct the luminance information of the image. You may According to the second embodiment, image correction can be performed without changing the number of exposure divisions. Therefore, it is possible to prevent the moving object from being expressed unnaturally without lowering the number of divisions more than necessary.

(Third embodiment)
Next, a third embodiment will be described. The configuration regarding division of exposure for generating an image is the same as in the first embodiment. The imaging apparatus of the third embodiment uses the out-of-screen area of the imaging element as the second area for acquiring the reference image. The out-of-screen area is an area in which it is possible to change whether or not image signals are output for display or recording according to the shooting mode. Specifically, in the moving image shooting mode, an area in which an image signal is not output for display or recording is defined as an out-of-screen area.

FIG. 17 is a diagram showing the image size of an image generated in still image shooting mode and the image size of an image generated in moving image shooting mode.
A range surrounded by a solid line is an area that can be imaged by the imaging device 184 and is an area (still image area) of a still image generated in the still image shooting mode. The area indicated by the upward-sloping hatched area inside the solid line is the moving image area (moving image area) generated in the moving image shooting mode. In the example shown in FIG. 17, the still image area has a screen ratio of 4:3. In the motion picture area, the screen ratio is 16:9. Therefore, in the moving image shooting mode, the area in which the image signal is output for display or recording shrinks in the vertical direction as compared with the still image shooting mode, and an out-of-screen area is generated that is not displayed on the screen. Even in the area outside the screen, subject light is incident through the imaging optical system.

FIG. 18 is a diagram showing an example of an in-screen area and an out-of-screen area in the moving image shooting mode.
In FIG. 18, the image generated in the screen area (a) is the main image. In this example, the in-screen area (a) is an area where image signals are output for display or recording in the moving image shooting mode. An image generated in an out-of-screen area having an out-of-screen area (b1) and an out-of-screen area (b2) is used as a reference image. It is desirable to compare the in-screen area (a) and the out-of-screen area in areas that are close to each other. This is because, if the areas are close to each other, the incident subject images are almost the same. Na1 in the screen area (a) and nb1 in the outside area (b1) are adjacent areas. Therefore, the imaging device sets na1 as an area to be compared with the reference image in the main image, and sets nb1 as an area in the reference image to be compared with the main image.

19 and 20 are diagrams showing an example of the relationship between the number of exposure divisions and luminance information according to the environmental temperature at the time of shooting.
FIG. 19A shows the relationship between the number of exposure divisions and luminance information when shooting at the environmental temperature T1. Also, FIGS. 19B and 20 show the relationship between the exposure division number and the luminance information in the case of photographing at the environmental temperature T2.

K1 and K2 shown in FIGS. 19 and 20 indicate average luminance information when the number of divisions is one. The dashed-dotted lines K1L and K2L indicate ideal lines where the brightness information does not change even if exposure by division is performed. The range from +dK1 to -dK1 and the range from +dK2 to -dK2 indicate the permissible range of change in luminance information.

The solid line indicates that the luminance decreases as the number of divisions increases when exposure is performed by division. This is because the carrier disappears when information is transferred to the memory for each divided exposure, resulting in a difference with respect to the ideal luminance information reduction obtained from the exposure ratio.

Comparing FIG. 19(A) and FIG. 19(B), it can be seen that the amount of decrease in luminance information changes due to changes in the environmental temperature. It can be seen that the greater the number of exposure divisions, the greater the decrease in luminance information due to changes in the temperature environment. When the number of exposure divisions is small, luminance information is less susceptible to changes in environmental temperature. Therefore, by reducing the number of exposure divisions, it is possible to correct the decrease in luminance information within an allowable range.

Assume that the first number of divisions for obtaining the main image is 54, and the second number of divisions for obtaining the reference image is one. From the graph shown in FIG. 20, when the exposure division number is less than about 23, it falls within the allowable range, and when the exposure division number is about 54, the luminance information is K3a, which is out of the allowable range. When the luminance information deviates from the allowable range, the imaging device performs control to reduce the number of exposure divisions. When the imaging device changes the first division number to 12, for example, the brightness information becomes K3b, which is within the allowable range.

According to the imaging apparatus of this embodiment, by controlling the number of exposure divisions according to the environmental temperature, even when the environmental temperature changes, it is possible to obtain an appropriate number of divisions and perform image correction. . For example, the imaging device may control the number of divisions of exposure according to the exposure time such as long exposure, or may control the number of divisions of exposure according to changes in electromagnetic noise.

It should be noted that table data relating to the amount of error in the luminance information and the predetermined number of divisions may be stored in the recording medium 193, and the imaging apparatus may refer to the table data to correct the image during normal operation. Then, when an environmental change occurs or is expected to occur, the imaging device may compare luminance information between the main image and the reference image to correct the main image.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

178 system control CPU
189 timing generator

Claims (11)

an imaging means for acquiring a first image by imaging by time-division exposure with a first division number and acquiring a second image by imaging by time-division exposure with a second division number;
and a control means for performing control for correcting the first image obtained by imaging by the imaging means according to the difference in luminance information between the first image and the second image. imaging device. 2. The imaging apparatus according to claim 1, wherein the control means performs control to correct the first image when the difference in luminance information exceeds a predetermined range. 3. The imaging according to claim 1, wherein the control means corrects the first image acquired after the next imaging period by changing the first division number. Device. 4. The imaging apparatus according to claim 3, wherein said control means reduces said first division number. The control means corrects the first image acquired after the next imaging period by changing the divided exposure time related to the exposure time-divided by the first division number. The imaging device according to claim 4. The imaging means acquires the first image and the second image within one imaging period,
3. The imaging apparatus according to claim 1, wherein the control means corrects the first image acquired during a current imaging period according to the difference in luminance information. The imaging means includes a plurality of pixel units having photoelectric conversion units,
7. The pixel unit according to any one of claims 1 to 6, wherein each pixel unit has a first storage unit that stores the first image and a second storage unit that stores the second image. 1. The imaging device according to claim 1. The imaging apparatus according to any one of claims 1 to 7, wherein the control means corrects luminance information of the first image. The imaging means is
comprising a plurality of pixel units each having a photoelectric conversion unit,
obtaining the first image from the pixel portion of the first region;
The imaging device according to any one of claims 1 to 5, wherein the second image is obtained from the pixel portion in the second area. 10. The imaging apparatus according to claim 9, wherein the second area is an area in which whether or not an image signal is output for display or recording can be changed according to a shooting mode. a step of acquiring a first image by imaging by exposure time-divided by a first division number and acquiring a second image by imaging by exposure time-divided by a second division number;
and performing control for correcting the first image obtained by imaging by the imaging means according to the difference in luminance information between the first image and the second image. How to control the device.

Images