JP7271155B2 - Imaging device and its control method - Google Patents

Description

The present invention relates to an imaging device and its control method.

2. Description of the Related Art In recent years, an imaging device that simultaneously captures two images has been developed. Patent Literature 1 discloses an imaging apparatus that removes crosstalk components between pixels from a first video signal when two images are shot simultaneously using one imaging element. However, in Patent Document 1, the shutter speed or frame rate of the two images is set with respect to the BV value of the subject within the range of the difference in sensitivity between the two types of photodiodes provided in the pixels of the imaging device. Therefore, it is not possible to simultaneously shoot a plurality of moving images with different frame rates while freely selecting the shutter speed. Furthermore, if the shutter speed of a low frame rate, which is a normal frame rate, is matched to the shutter speed of a high frame rate image in order to match the exposure, the image may become a jerky, low-quality moving image. Therefore, while freely selecting the shutter speed, a high-quality image is obtained that can be reproduced in normal reproduction with no jerky feeling.

Further, in recent years, image blur correction and autofocus have been performed in order to obtain higher quality images. In image blur correction, a motion vector that serves as an index indicating the amount and direction of camera shake is detected, blur correction is performed using a camera shake correction mechanism, and electronic image stabilization is performed by cutting out an image from a captured image. Patent Literature 2 discloses a method of detecting a motion vector that indicates the motion of an object between multiple images. In autofocus, a method is known in which the focus state of a subject is detected by contrast evaluation by comparing the contrast between a plurality of images.

JP 2017-022445 A JP-A-2-170681

However, although the detection and use of motion vectors is effective when capturing multiple frames continuously, such as when shooting a moving image, motion vectors cannot be obtained within the still image exposure time when shooting still images. . Therefore, in order to correct blurring during still image exposure, a separate blurring detection sensor must be used. Similarly, autofocusing requires the use of a separate distance measuring sensor, which may complicate the apparatus. Therefore, a still image and a detection image are photographed in parallel without providing a separate detection sensor, and still image photographing is controlled according to the detected photographing state to acquire a high-quality still image.

SUMMARY OF THE INVENTION An object of the present invention is to provide an imaging apparatus capable of acquiring high-quality images at different frame rates from a single imaging element.

In order to solve the above problems, the imaging device of the present invention provides a first charge holding unit that accumulates charges for generating a first image in one photoelectric conversion unit; an image pickup device having pixels arranged two-dimensionally, the image pickup device comprising: a second charge holding unit for accumulating charges for generating a second image having a frame rate higher than that of the image ; While transferring the charge from the one photoelectric conversion unit to the first charge holding unit in a plurality of divided times, and transferring the charge from the one photoelectric conversion unit to the second charge holding unit in a plurality of times. The second image is acquired by transferring the electric charge, dividing the one imaging period into a plurality of times, and reading a signal corresponding to the electric charge of the second charge holding unit, and obtaining the second image by reading out the signal according to the electric charge of the second charge holding unit. an acquisition unit configured to acquire the first image by reading a signal corresponding to the charge of the first charge holding unit after reading of the signal corresponding to the charge of the second charge holding unit is completed ; detection means for detecting a motion vector from the first image; and correction means for correcting image blur in the first image based on the motion vector detected by the detection means .

ADVANTAGE OF THE INVENTION According to this invention, the imaging device which can acquire the high quality image of a different frame rate from one imaging device can be provided.

It is a figure which shows the external appearance of an imaging device. It is a block diagram which shows the structural example of an imaging device. 3 is a circuit diagram showing a configuration example of a pixel of an imaging device; FIG. 3 is a circuit diagram showing a configuration example of a readout circuit; FIG. 4 is a timing chart showing a driving sequence of the imaging element; 4 is a flowchart showing still image shooting processing including blur correction. 4 is a flow chart showing blur correction processing. 4 is a timing chart showing a driving sequence of the imaging element; 4 is a flowchart showing still image shooting processing including blur correction. 4 is a flow chart showing blur correction processing. 7 is a flowchart showing still image shooting processing including exposure amount detection. 5 is a flowchart showing processing for detecting the amount of exposure; 3 is a circuit diagram showing a configuration example of a pixel of an imaging device; FIG. It is a figure explaining the setting screen of the imaging conditions of a 1st moving image and a 2nd moving image. It is a figure explaining the setting of the imaging conditions of a 1st moving image and a 2nd moving image. FIG. 4 is a diagram for explaining accumulation and readout timings in an image pickup device; FIG. 4 is a diagram for explaining accumulation and readout timings in an image pickup device; It is a figure showing the screen displayed on a display part. It is a figure explaining the utilization example of a 1st moving image and a 2nd moving image. It is a figure explaining the utilization example of a 1st moving image and a 2nd moving image.

(First embodiment)
FIG. 1 is an external view of an imaging device 100. FIG. 1A is a front view of the imaging device 100, and FIG. 1B is a rear view of the imaging device 100. FIG. The imaging device 10 is, for example, a digital motion camera capable of capturing still images and moving images. In this embodiment, an example of an imaging device in which an imaging device main body and a lens are integrated will be described, but the imaging device is not limited to this, and may be, for example, a lens-interchangeable digital single-lens camera. Also, the image processing unit that processes the image signal output from the image pickup device 184 does not necessarily have to be configured as part of the image pickup device, and is configured by hardware separate from the image pickup device 184 and the imaging optical system. may Also, all or part of the functions of the image processing section may be mounted on the imaging device 184 .

The image pickup apparatus 100 includes an image pickup apparatus main body 151 , an image pickup optical system 152 on the front surface of the image pickup apparatus main body 151 , and a switch ST<b>154 on the top surface of the image pickup apparatus main body 151 . The imaging device 100 also includes a display unit 153 , a switch MV 155 , a selection lever 156 , a menu button 157 , an up switch 158 , a down switch 159 , a dial 160 and a playback button 161 on the back of the imaging device main body 151 .

The imaging device main body 151 is a main body portion of the imaging device 100 in which the imaging element 184 and the shutter device are accommodated. The imaging optical system 152 is an optical system for forming an optical image of a subject on the imaging device 184, and has a lens and a diaphragm inside. The display unit 153 is a display unit for displaying shooting information and images. The display unit 153 has a display luminance range that can display an image with a wide dynamic range without suppressing the luminance range. Moreover, the display unit 153 may be provided with a movable mechanism for changing the orientation of the screen as necessary.

The switch ST154 is a shutter button mainly used for taking still images. The switch MV155 is a button for starting and stopping moving image shooting. A selection lever 156 is a changeover switch for selecting a shooting mode. A menu button 157 is a menu button for shifting to a function setting mode for setting the functions of the imaging apparatus 100 . Up switch 158 and down switch 159 are up/down switches for changing various set values. A dial 160 is a dial for changing various set values. The playback button 161 is a button for switching to a playback mode in which the video recorded on the recording medium 193 in the imaging device 100 is played back on the display unit 153 .

FIG. 2 is a block diagram showing a configuration example of the imaging device 100. As shown in FIG. The imaging device 100 includes a focus lens 181 , a focus lens driver 182 , an optical filter 183 , an image sensor 184 , a blur correction lens 185 and a blur correction driver 186 . The imaging apparatus 100 also includes a digital signal processing section 187 , an analog front end 188 , a timing generation section 189 , a system control CPU 178 , a switch input means 179 and an image memory 190 . The imaging device 100 also includes a display I/F 191 , a display unit 153 , a recording I/F 192 , a recording medium 193 , a print I/F 194 , an external I/F 196 and a wireless I/F 198 .

An optical axis 180 is the optical axis of the imaging optical system 152 . The focus lens 181 is an optical lens for adjusting the focus state of the imaging optical system 152 . The focus lens driving section 182 drives the focus lens 181 . The blur correction lens 185 is an optical lens for correcting image blur. The blur correction drive section 186 drives the blur correction lens 185 . The optical filter 183 limits the wavelength of light incident on the imaging device 184 and the spatial frequencies transmitted to the imaging device 184 .

The imaging device 184 converts the optical image of the subject imaged via the imaging optical system 152 into an electrical image signal. An analog front end 188 performs analog processing and analog-to-digital conversion of the image signal output from the imaging device 184 . The analog front end 188 has a readout circuit, an A/D converter for digitizing analog signals, and the like. Details of the readout circuit will be described later.

The digital signal processing unit 187 performs various corrections on the digital video data output from the imaging element 184, and then compresses the video data. Further, the digital signal processing unit 187 has functions of image generating means and image synthesizing means. The timing generator 189 outputs various timing signals to the imaging element 184 and the digital signal processor 187 to control various timings. The system control CPU 178 is a CPU (Central Processing Unit) that performs various calculations and controls the entire imaging apparatus 100 . That is, the timing generator 189 and the system control CPU 178 have the function of control means for controlling the transfer, accumulation, and readout timings of signal charges in the imaging device 184 .

Further, in the first to third embodiments, the system control CPU 178 compares a plurality of image signals for detection acquired by the digital signal processing unit 187 to determine the state of the imaging device 100 and the imaging optical system 152 (imaging state). The system control CPU 178 also has the function of a blur correction calculation means, generates a blur correction signal based on the motion vector detection result, and sends the blur correction signal to the blur correction driving section 186 .

The video memory 190 temporarily stores video data. A display I/F 191 is an interface for displaying a captured image on the display unit 153 . The display unit 153 is a display unit such as a liquid crystal display. A recording I/F 192 is an interface for recording on or reading from the recording medium 193 . A recording medium 193 is a recording medium such as a memory for recording video data, additional data, and the like. The recording medium 193 may be attached to the imaging device 100 or may be detachable.

A print I/F 194 is an interface for outputting a photographed image to an external printer 195 for printing. Printer 195 is a printer such as a small inkjet printer. The external I/F 196 is an interface for communicating with the external device 197 and the like. The external device 197 is a device capable of displaying images, such as a computer or television. A wireless I/F 198 is an interface for communicating with an external network 199 . Network 199 is a computer network such as the Internet. The switch input unit 179 includes a switch ST154, a switch MV155, and a plurality of switches for switching between various modes, and receives operations from the user.

FIG. 3 is a circuit diagram showing the pixel configuration of the imaging device 184. As shown in FIG. The imaging device 184 has a large number of pixel elements (pixel units) arranged two-dimensionally. In FIG. 3, among the many pixel elements of the image sensor 184, the pixel element 50 in the first row and first column (1, 1) and the pixel element 51 in the last row, m row and first column (m, 1) are showing. Since the pixel element 50 and the pixel element 51 have the same configuration, each component of each pixel is given the same reference numeral.

The pixel element 50 has a photodiode 500, which is a photoelectric conversion portion, a first charge holding portion 507A and a second charge holding portion 507B. The pixel element 50 also has a first transfer transistor 501 A, a second transfer transistor 502 A, a third transfer transistor 501 B, a fourth transfer transistor 502 B and a fifth transfer transistor 503 . The pixel element 50 also has a reset transistor 504 , an amplification transistor 505 , a selection transistor 506 and a floating diffusion region (hereinafter referred to as FD region) 508 . Further, the pixel element 50 includes power lines 520 , power lines 521 and signal output lines 523 .

One pixel element of the imaging element 184 has two charge holding portions (first charge holding portion 507A and second charge holding portion 507B) for one photodiode 500 .
Thereby, it is possible to acquire the still image signal, which is the first image signal, and the detection image signal, which is the second image signal, in parallel. In this embodiment, the first charge holding unit 507A accumulates the charge for the still image, and the second charge holding unit 507B accumulates the charge for the detection image. The basic structure of the imaging device 184 having the charge holding portion is disclosed in Japanese Patent Application Laid-Open No. 2013-172210 by the applicant of the present application, so description thereof is omitted. Although an example in which there are two charge holding units is described in this embodiment, the present invention is not limited to this, and a plurality of charge holding units may be provided.

The anode of photodiode 500 is connected to the ground line. The cathode of the photodiode 500 is connected to the source of the first transfer transistor 501A, the source of the third transfer transistor 501B and the source of the fifth transfer transistor 503, respectively. The drain of the first transfer transistor 501A is connected to the source of the second transfer transistor 502A. A connection node between the drain of the first transfer transistor 501A and the source of the second transfer transistor 502A constitutes a first charge holding portion 507A. The drain of the third transfer transistor 501B is connected to the source of the fourth transfer transistor 502B. A connection node between the drain of the third transfer transistor 501B and the source of the fourth transfer transistor 502B constitutes a second charge holding portion 507B.

The drain of the first transfer transistor 502A and the drain of the third transfer transistor 502B are connected to the source of the reset transistor 504 and the gate of the amplification transistor 505, respectively. A connection node of the drain of the second transfer transistor 502A, the drain of the fourth transfer transistor 502B, the source of the reset transistor 504, and the gate of the amplification transistor 505 constitutes an FD region 508. FIG. A source of the amplification transistor 505 is connected to a drain of the selection transistor 506 . A drain of the reset transistor 504 and a drain of the amplification transistor 505 are connected to the power supply line 520 . A drain of the fifth transfer transistor 503 is connected to the power supply line 521 . The source of select transistor 506 is connected to signal output line 523 . The other end of the signal output line 523 is connected to the readout circuit 308 .

Each pixel element is connected in row units to control lines arranged in the row direction from the vertical scanning circuit 307 . Each row of control lines includes a plurality of control lines connected to the gates of transfer transistors 501A, 502A, 501B, 502B, 503, reset transistor 504 and select transistor 506, respectively. Each control pulse is sent from the vertical scanning circuit 307 to each transistor based on the control signal from the system control CPU 178 . Each transistor is turned on when the control pulse is at high level and turned off when the control pulse is at low level.

Specifically, the first transfer transistor 501A is controlled by a transfer pulse φTX1A. The second transfer transistor 502A is controlled by a transfer pulse φTX2A. The third transfer transistor 501B is controlled by a transfer pulse φTX1B. The fourth transfer transistor 502B is controlled by a transfer pulse φTX2B. The fifth transfer transistor 503 is controlled by a transfer pulse φTX3. A reset transistor 504 is controlled by a reset pulse φRES. The selection transistor 506 is controlled by a selection pulse φSEL.

The photodiode 500 is reset by a transfer pulse φTX3 that controls the fifth transfer transistor 503, and the timing of starting accumulation is determined. Also, the transfer pulse φTX1A that controls the first transfer transistor 501A determines the timing at which the charge accumulated in the photodiode 500 is transferred to the first charge holding portion 507A. A transfer pulse φTX1B that controls the third transfer transistor 501B determines the timing at which the charge accumulated in the photodiode 500 is transferred to the second charge holding portion 507B.

FIG. 4 is a circuit diagram showing a configuration example of the readout circuit 308 of the image sensor 184. As shown in FIG. The reading circuit 308 includes a first switch 414, a second switch 415, a third switch 418, a fourth switch 419, a first capacitor 410, a second capacitor 411, a first horizontal output line 424, a 2 horizontal output lines 425 and an output amplifier 421 .

A first switch 414 is a switch that controls writing of a pixel signal to the capacitor 410 . The first switch 414 is a switch controlled by a signal Ts, turns on when the signal Ts is at high level, and connects the output terminal of the signal output line 523 and the capacitor 410 . A second switch 415 is a switch that controls writing of a pixel signal to the capacitor 411 . The second switch 415 is a switch controlled by a signal Tn, is turned on when the signal Tn is at high level, and connects the output terminal of the signal output line 523 and the capacitor 411 .

A third switch 418 is a switch that controls the output of the pixel signal held in the capacitor 410 to the output amplifier 421 . A fourth switch 419 is a switch that controls the output of the pixel signal held in the capacitor 411 to the output amplifier 421 . The third switch 418 and the fourth switch 419 are turned on according to the control signal from the horizontal shift register 431 . As a result, the signal written to the capacitor 410 is output to the output amplifier 421 via the third switch 418 and horizontal output line 424 . Also, the signal written in the capacitor 411 is output to the output amplifier 421 via the fourth switch 419 and the horizontal output line 425 .

The output amplifier 421 outputs a differential signal of the signals from the horizontal output lines 424 and 425 . The output of the output amplifier 421 is converted into a digital signal by the A/D converter of the analog front end 188 and sent to the digital signal processing section 187 . The signal Ts, the signal Tn, and the signal from the horizontal shift register 431 are signals supplied from the timing generator 189 under the control of the system control CPU 178 .

FIG. 5 is a timing chart showing the driving sequence of the imaging device 184 in the first embodiment. In the following description, the timing of control of the image sensor will be described using the pixel element 50 of (1, 1) in the circuit diagram shown in FIG. 3 as an example. Note that the timing chart of FIG. 5 schematically shows the time interval, which differs from the actual control time interval.

First, at time t1, the transfer pulse φTX3 supplied from the vertical scanning circuit 307 transitions from high level to low level. This turns off the transfer transistor 503 of the pixel element 50 and releases the reset of the photodiode 500 of the pixel element 50 . Then, the photodiode 500 of the pixel element 50 starts to accumulate signal charges for a still image.

At time t2, the selection pulse φSEL transitions from low level to high level. As a result, the selection transistor 506 of the pixel element 50 is turned on, and image signals can be read from the pixel element 50 .
At time t3, the transfer pulse φTX1A transitions from low level to high level. As a result, the first transfer transistor 501A is turned on, and the signal charge accumulated in the photodiode 500 of the pixel element 50 is transferred to the first charge holding portion 507A.

At time t4, the transfer pulse φTX1A transitions from high level to low level. As a result, the first transfer transistor 501A is turned off, and transfer of the signal charge accumulated in the photodiode 500 to the first charge holding portion 507A is completed.
Similarly, at time t4, the transfer pulse φTX3 transitions from low level to high level. As a result, the fifth transfer transistor 503 is turned on and the photodiode 500 is reset.

At time t5, the transfer pulse φTX3 transitions from high level to low level. As a result, the fifth transfer transistor 503 is turned off, and the photodiode 500 of the pixel element 50 starts accumulating signal charges for detection.
At time t6, the reset pulse φRES transitions from high level to low level. As a result, the reset transistor 504 is turned off, and the reset state of the FD region 508 is canceled. At this time, a signal corresponding to the potential due to the residual noise component existing in the FD region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506 .

At time t7, the signal Tn transitions from low level to high level. As a result, the second switch 415 is turned on, and the output signal from the signal output line 523 is written to the capacitor 411 .
At time t8, the signal Tn transitions from high level to low level to turn off the second switch 415, and the writing to the capacitor 411 ends. A signal corresponding to residual noise in the FD region 508 read out through the signal output line 523 at time t6 is written in the capacitor 411 .

At time t9, the transfer pulse φTX1B transitions from low level to high level. As a result, the third transfer transistor 501B is turned on, and the signal charge accumulated in the photodiode 500 is transferred to the second charge holding portion 507B.
At time t10, the transfer pulse φTX1B transitions from high level to low level. As a result, the third transfer transistor 501B is turned off, and transfer of the signal charge accumulated in the photodiode 500 to the second charge holding portion 507B is completed.
Similarly, at time t10, the transfer pulse φTX3 transitions from low level to high level. As a result, the fifth transfer transistor 503 is turned on and the photodiode 500 is reset.

At time t11, the transfer pulse φTX3 transitions from high level to low level. As a result, the fifth transfer transistor 503 is turned off, and the photodiode 500 of the pixel element 50 starts accumulating the signal charges that form a still image again.
At time t12, the transfer pulse φTX2B transitions from low level to high level. As a result, the fourth transfer transistor 502B is turned on, and the signal charge accumulated in the second charge holding portion 507B is transferred to the FD region 508. FIG. A signal corresponding to the change in potential of the FD region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506 .

At time t13, the transfer pulse φTX2B transitions from high level to low level. As a result, the fourth transfer transistor 502B is turned off, and transfer of the signal charge accumulated in the second charge holding portion 507B to the FD region 508 is completed.
At time t14, the signal Ts transitions from low level to high level. As a result, the first switch 414 is turned on, and the output signal from the signal output line 523 is written to the capacitor 410 .

At time t15, the signal Ts transitions from high level to low level. As a result, the first switch 414 is turned off, and the writing to the capacitor 410 is completed. The signal of the image for detection of the FD area 508 read out through the signal output line 523 at time t12 is written in the capacitor 410 .
Also at time t15, a signal from the horizontal shift register 431 turns on the third switch 418 and the fourth switch 419 at the same time. As a result, the signal written in the capacitor 410 is output to the output amplifier 421 through the horizontal output line 424 , and the signal written in the capacitor 411 is output to the output amplifier 421 through the horizontal output line 425 . The output amplifier 421 outputs a differential signal of the signal from the horizontal output line 424 . This makes it possible to extract an image signal from which the residual noise has been removed from the detection image signal.

At time t16, the selection pulse φSEL transitions from high level to low level. As a result, the selection transistor 506 is turned off to be in a non-selected state.
Also at time t16, the reset pulse φRES transitions from low level to high level. As a result, the reset transistor 504 is turned on and the FD region 508 is reset.

At time t17, the selection pulse φSEL transitions from low level to high level. As a result, the selection transistor 506 is turned on, and image signals can be read out from the pixel element 50 .
At time t18, the transfer pulse φTX1A transitions from low level to high level. As a result, the first transfer transistor 501A is turned on, and the signal charge accumulated in the photodiode 500 is transferred to the first charge holding portion 507A.

At time t19, the transfer pulse φTX1A transitions from high level to low level. As a result, the first transfer transistor 501A is turned off, and transfer of the signal charge accumulated in the photodiode 500 to the first charge holding portion 507A is completed. As a result, in the first charge holding portion 507A, the signal charges accumulated in the photodiode 500 during the period from time t1 to time t4 and transferred at time t4 are combined with the signal charges accumulated in the photodiode 500 from time t11 to time t19. the signal charge is added.

Since the operation from time t17 to time t20 is the same as the operation from time t2 to time t16, detailed description thereof will be omitted.
Between time t17 and time t21, the operation from time t2 to time t16 is repeated multiple times. As a result, the reading operation of the detection image is performed multiple times. The image blur detection operation using the detection image will be described later. Further, the operation of transferring the signal charges accumulated in the photodiode 500 for still images to the first charge holding portion 507A is performed multiple times.

At time t21, the selection pulse φSEL transitions from low level to high level. As a result, the selection transistor 506 is turned on, and image signals can be read out from the pixel element 50 .
At time t22, the reset pulse φRES transitions from high level to low level. As a result, the reset transistor 504 is turned off, and the reset state of the FD region 508 is canceled. At this time, a signal corresponding to the potential due to the residual noise component present in the FD region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506, as at time t6.

At time t23, the signal Tn transitions from low level to high level. As a result, the second switch 415 is turned on, and the output signal from the signal output line 523 is written to the capacitor 411 .
At time t24, the signal Tn transitions from high level to low level. As a result, the second switch 415 is turned off, and writing to the capacitor 411 is completed. A signal corresponding to residual noise in the FD region 508 read out through the signal output line 523 at time t23 is written in the capacitor 411 .

At time t25, the transfer pulse φTX1A transitions from low level to high level. As a result, the first transfer transistor 501A is turned on, and the signal charge accumulated in the photodiode 500 is transferred to the first charge holding portion 507A.
At time t26, the transfer pulse φTX1A transitions from high level to low level. As a result, the first transfer transistor 501A is turned off, and transfer of the signal charge accumulated in the photodiode 500 to the first charge holding portion 507A is completed.
Similarly, at time t26, the transfer pulse φTX3 transitions from low level to high level. As a result, the fifth transfer transistor 503 is turned on and the photodiode 500 is reset.

At time t27, the transfer pulse φTX2A transitions from low level to high level. As a result, the second transfer transistor 502A is turned on, and the signal charge accumulated in the first charge holding portion 507A is transferred to the FD region 508. FIG. A signal corresponding to the change in potential of the FD region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506 .

At time t28, the transfer pulse φTX2A transitions from high level to low level. As a result, the second transfer transistor 502A is turned off, and transfer of the signal charge accumulated in the second charge holding portion 507B to the FD region 508 is completed.
At time t29, the signal Ts transitions from low level to high level. As a result, the first switch 414 is turned on, and the output signal from the signal output line 523 is written to the capacitor 410 .

At time t30, the signal Ts transitions from high level to low level. As a result, the switch 414 is turned off, and the writing to the capacitor 410 is completed. A still image signal of the FD area 508 read from the signal output line 523 at time t29 is written in the capacitor 410 .
At the same time, at time t30, a signal from the horizontal shift register 431 turns on the third switch 418 and the fourth switch 419 . As a result, the signal written in the capacitor 410 is output to the output amplifier 421 through the horizontal output line 424 , and the signal written in the capacitor 411 is output to the output amplifier 421 through the horizontal output line 425 . Then, the output amplifier 421 outputs a differential signal of the signal from the horizontal output line 424 . This makes it possible to extract an image signal from which the residual noise has been removed from the still image signal.

At time t31, the selection pulse φSEL transitions from high level to low level. As a result, the selection transistor 506 is turned off to be in a non-selected state.
At the same time, at time t31, the reset pulse φRES transitions from low level to high level. As a result, the reset transistor 504 is turned on and the FD region 508 is reset. Then, the driving of the imaging device 184 ends.

As described above, in the first embodiment, the signal charge of the still image accumulated in the photodiode 500 is time-divided and transferred to the first charge holding unit 507A in a plurality of cycles, and then transferred to the first charge holding unit 507A. The charge signal is transferred to the FD area 508 and the image signal is read out. That is, during the period from the start of exposure in the photodiode 500 to the transfer of the charge signal in the first charge holding portion 507A to the FD region 508, the first charge holding of the signal charge of the still image accumulated in the photodiode 500 is performed. Transfer to the unit 507A is performed multiple times. Then, the signal charge of the image for detection accumulated in the photodiode 500 during the above period (during one shooting period of the still image) is transferred to the second charge holding unit 507B, and the charge signal of the second charge holding unit 507B is transferred. are transferred to the FD area 508 and read out a plurality of times. As a result, the detection image can be acquired multiple times during one still image shooting operation.

Next, the flow of the still image shooting operation including the detection operation using the detection image will be described with reference to the flow charts of FIGS. 6 and 7. FIG. The imaging apparatus 100 of the present embodiment has a camera shake correction mechanism that includes a shake correction lens 185 and a shake correction drive unit 186 in the imaging optical system 152, and reads a detection image even during still image shooting to obtain an image. Image blur correction is performed by detecting the amount of blurring. It should be noted that the flowcharts of FIGS. 6 and 7 show only the parts related to the detection operation for the sake of simplification of explanation.

FIG. 6 is a flowchart showing still image shooting processing including blur correction using a detection image in the first embodiment. This process is started when the user presses the switch ST154 to give an instruction to shoot a still image. In step S601, the photodiode 500 starts the exposure accumulation operation of the still image signal. Specifically, the transfer transistor 503 is turned off by a control pulse signal from the signal control means composed of the system control CPU 178 and the timing generator 189, and the photodiode 500 starts the exposure accumulation operation of the still image signal. do. The operation of step S601 corresponds to time t1 in the timing chart of FIG. Note that the still image exposure time is determined in advance before the switch input means 179 issues a still image shooting command.

In step S602, the still image signal is transferred to the first charge holding unit 507A. Specifically, the transfer transistor 501A is turned on by a control pulse signal from the signal control means, and the still image signal exposed and accumulated by the photodiode 500 in step S601 is transferred to the first charge holding unit 507A. After that, the transfer transistor 501A is turned off to end the transfer. The operation of step S602 corresponds to the operations at time t3 and time t4 in the timing chart of FIG.

In step S603, the photodiode 500 starts the exposure accumulation operation of the image signal for detection. The operation of step S603 corresponds to time t5 in the timing chart of FIG.

In step S604, the image signal for detection is transferred to the second charge holding unit 507B. Specifically, the transfer transistor 501B is turned on by a control pulse signal from the signal control means, and the image signal for detection that has been exposed and accumulated in the photodiode 500 in step S603 is transferred to the second charge holding unit 507B. . After that, the transfer transistor 501B is turned off to end the transfer. The operation of step S604 corresponds to the operation at time t9 and time t10 in the timing chart of FIG.

In step S605, the still image signal exposure accumulation operation is restarted. Specifically, the transfer transistor 503 is turned off by a control pulse signal from the signal control means, and the exposure accumulation operation of the still image signal in the photodiode 500 is restarted. The operation of step S605 corresponds to the operation at time t11 in the timing chart of FIG.

In step S606, the image signal for detection is transferred to the FD area 508. FIG. Specifically, the transfer transistor 502B is turned on by a control pulse signal from the signal control means, and the charge signal held in the second charge holding portion 507B is transferred to the FD region 508 in step S604. After that, the transfer transistor 502B is turned off to end the transfer. The operation of step S606 corresponds to time t12 and time t13 in the timing chart of FIG.

In step S607, the detection image IMG-D(i) is read. Specifically, the charge signal transferred to the FD area 508 in step S606 is read by the reading circuit 308 as the detection image IMG-D(i). i is an integer value starting from 0, and indicates the number of detection images IMG-D counted after still image shooting is started. i=0 represents the first read detection image IMG-D. The operation of step S607 corresponds to time t15 in the timing chart of FIG.

In step S608, blur correction is performed. Details of blur correction will be described later with reference to FIG.
In step S609, the still image signal is transferred to the charge holding unit 507A. Specifically, the transfer transistor 501A is turned on by a control pulse signal from the signal control means, and the still image signal exposed and accumulated by the photodiode 500 is transferred to the first charge holding unit 507A from step S605. Then, the still image signal transferred in step S609 is added to the charges already accumulated in the charge holding unit 507A. After the charges are added, the transfer transistor 501A is turned off to complete the transfer. The operation in step S609 corresponds to the operation at time t18 and time t19 in the timing chart of FIG.

In step S610, the system control CPU 178 determines whether the detection image IMG-D(i) read out in step S607 is i=N, that is, whether it is the Nth image. If i=N, the process proceeds to step S611. On the other hand, when i is other than N, the process proceeds to step S614. In step S614, i is incremented such that i=i+1, and the process returns to step S603. Note that N is the number of times the detection image IMG-D is read out during still image shooting, and is determined based on the shooting exposure time predetermined before the start of shooting.

At step S 611 , the still image signal is transferred to the FD area 508 . Specifically, the transfer transistor 502A is turned on by a control pulse signal from the signal control means, and the charge signal intermittently transferred to and held in the first charge holding unit 507A is transferred between steps S601 to S609. Transfer to FD area 508 . Then, the transfer transistor 502A is turned off to end the transfer. The operation of step S611 corresponds to time t27 and time 28 in the timing chart of FIG.

In step S612, a still image signal is read. Specifically, the reading circuit 308 reads the charge signal transferred to the FD area 508 in step S611 as a still image signal. The operation of step S612 corresponds to time t30 in the timing chart of FIG.
In step S613, an image obtained by synthesizing the still image and the detection image is recorded. Specifically, the digital signal processing unit 187 synthesizes the detection image IMG-D(i) read out a plurality of times in step S607 and the still image signal read out in step S612 to create one composite image. output as The system control CPU 178 records the composite image in the video memory 190 and terminates this process.

FIG. 7 is a flowchart showing blur correction processing in step S608 of FIG.
In step S701, the system control CPU 178 determines whether the detection image IMG-D(i) read out in step S607 is i=0, that is, whether it is the first detection image. If i=0, the process ends. On the other hand, if i is other than 0, the process proceeds to step S702.

In step S702, the system control CPU 178 detects a motion vector using the two read detection images IMG-D(i) and IMG-D(i-1). A motion vector can be detected, for example, by comparing the feature points of two temporally consecutive images to detect the motion of the images. Since motion vector detection is a well-known technique, its description is omitted. In the first embodiment, the system control CPU 178 functions as motion vector detection means.

In step S703, the system control CPU 178 calculates the blur correction amount based on the motion vector detected in step S702.
In step S704, the blur correction driving unit 186 drives the blur correction lens 185 according to the blur correction amount calculated by the system control CPU 178, and the process ends.

As described above, the imaging apparatus 100 captures a still image according to the flow described with reference to FIGS. 6 and 7 . While the photodiode 500 intermittently exposes and accumulates the still image signal, the detection image signal is exposed and accumulated a plurality of times and read out as the detection image signal. Then, it is possible to detect motion vectors using the detection image signals obtained continuously in time, calculate the blur correction amount, and correct image blur during still image exposure by driving the blur correction lens 185. . As described above, in the present embodiment, it is possible to detect image blurring during still image exposure and correct image blurring without installing a separate image blurring detection sensor.

Note that in this embodiment, in step S613, the still image signal is added and synthesized with the detection image signal used for motion vector detection a plurality of times to generate one synthesized image. This is to obtain the desired amount of exposure. A predetermined set exposure time for a captured image corresponds to a period from step S601 to step S609. The set exposure time corresponds to the period from time t1 to time t25 in the timing chart of FIG. Since the still image signal is exposed and accumulated in the photodiode 500 intermittently, the accumulated exposure accumulation time is shorter than the set exposure time. In other words, the still image signal read out in step S612 is shorter than the set exposure time, and thus is lower than the desired exposure amount. Further, while the photodiode 500 is interrupting the exposure and accumulation of the still image signal, the detection image signal is exposed and accumulated. Therefore, by adding and synthesizing the image signal for detection to the still image signal, the exposure accumulation time of the synthesized image can be adjusted to the set exposure time.

Also, in step S613, an example of simply adding and synthesizing the still image and the image for detection has been described, but the present invention is not limited to this, and synthesizing may be performed to obtain a desired amount of exposure. For example, since the still image signal and the detection image signal have different exposure and accumulation times, they can be combined at a predetermined ratio during additive synthesis, and recorded with a higher number of recording bits to obtain an image with an expanded dynamic range. can be done. Image processing for expanding the dynamic range using a plurality of images with different exposure amounts is a known technique, so description thereof is omitted.

Note that the still image and the detection image may be recorded in the video memory 190 without generating the composite image in the imaging device 100 in step S613. In this case, after photographing is completed, addition synthesis may be performed in the imaging device 100 or the external device 197 to which the recording data was transferred, based on the recorded still image and the detection image.

In the present embodiment, in the timing chart of FIG. 5, the time from time t5 to time t10, which is the exposure accumulation time of one detection image, is the time of one exposure accumulation time in the photodiode 500 of the still image. The period from t1 to time t5 is set to be shorter. Since it is desirable that the detection image used for motion vector detection is less affected by blurring, the exposure accumulation time for one detection image is set short.

In this embodiment, image blur detection and image blur correction during still image shooting have been described, but other shooting conditions may be detected during still image shooting. For example, based on a plurality of continuous images, the contrast evaluation value of the range including the subject is calculated, and contrast detection is performed to detect whether the focus lens 181 is in focus with respect to the subject based on the change in the contrast evaluation value. may In this case, the system control CPU 178 functions as focus detection means for calculating the contrast evaluation value and detecting the focus state. In the contrast detection, the focus lens driver 186 performs a so-called wobbling operation in which the focus lens 181 is finely driven during still image exposure, and the contrast evaluation value of each detection image obtained multiple times by the focus detection means is calculated. Then, the focus lens 181 is driven so that the subject is in focus based on the movement position of the focus lens 181 by the wobbling operation and the change in the contrast evaluation value of the detection image obtained at that position.

When performing contrast detection, in the flowchart of FIG. 6, the process is the same as described above except that the focus lens 181 continues wobbling driving from the start of the flow and the blur correction operation in step S608 is AF operation. In the AF operation, the contrast is evaluated using the detection image IMG-D(i) obtained during the wobbling drive, and the focus lens 181 is superimposed on the wobbling drive so that the subject is in focus. By performing control in this way, even if the subject moves in the focus direction during still image exposure or camera shake occurs in the optical axis direction without attaching a separate focus detection sensor, the photographic optical The focus state of system 152 can be detected. By driving the focus lens 181 based on the focus detection result, the subject can be kept in focus.

As described above, according to the first embodiment, a still image is captured while performing image blur correction, focus adjustment, etc., according to the image blur detected from the detection image and the shooting state such as the in-focus state. By acquiring, a high-quality still image can be acquired.

(Second embodiment)
In the first embodiment, a still image and a detection image are acquired using the imaging element 184 having a configuration having two charge holding units for one photodiode, and the state of the imaging device 100 is determined using the detection image. detected and corrected for optical conditions. In the second embodiment, a still image and a detection image are acquired using the imaging element 184 having the same configuration as in the first embodiment, and the detection image is used to detect the state of the imaging device 100, and the state of the imaging device 100 is detected. Accordingly, it controls the transfer of the charges that have been exposed and accumulated for the still image.

In the second embodiment, the state of the imaging device 100 is detected using the detection image, and when the change in the photographing state is greater than or equal to a predetermined value, the first charge is emitted from the photodiode 500 of the charges exposed and accumulated for the still image. Transfer to the holding unit 507A is temporarily stopped. Hereinafter, the operation of temporarily stopping the transfer of the charges exposed and accumulated for the still image to the first charge holding unit 507A will be referred to as the transfer stop operation. The detection image is read out even during the transfer stop operation for the still image, and the transfer stop operation is canceled according to the result of the detection image. Transfer to the charge holding unit 507A is restarted. Details of the second embodiment will be described with reference to FIGS. Note that the overall configuration of the imaging device 100 and the circuit configuration of the imaging element 184 in the second embodiment are the same as in the first embodiment, so detailed descriptions thereof will be omitted.

FIG. 8 is a timing chart showing the driving sequence of the imaging element 184 in the second embodiment. The timing chart of FIG. 8 shows a part of the driving sequence of the imaging element 184 that is related to the transfer stop operation, and time t41 corresponds to time t16 in the timing chart shown in FIG. 5 of the first embodiment. Since the operation up to time t41 is the same as in the first embodiment, the explanation is omitted, and the transfer stop operation from time t42 will be explained. At time t42, a transfer stop FLAG, which will be described later, is ON during the still image shooting operation, and the transfer stop operation is switched to.

At time t42, the selection pulse φSEL transitions from low level to high level. As a result, the selection transistor 506 of the pixel element 50 is turned on, and image signals can be read from the pixel element 50 . Note that time t42 corresponds to time t17 in the first embodiment.
Time t43 corresponds to time t18 in the first embodiment. As indicated by the dotted line in FIG. 8, at time t18, the transfer pulse φTX1A for the first row of the still image transitions from low level to high level. On the other hand, in the second embodiment, as indicated by the solid line, the transfer pulse φTX1A remains at the low level because the transfer is being stopped.

At time t44, the transfer pulse φTX3 transitions from low level to high level. As a result, the transfer transistor 503 is turned on and the photodiode 500 is reset. That is, during the transfer stop operation, the charge exposed and accumulated in the photodiode 500 until time t44 is reset without being transferred to the first charge holding portion 507A.

At time t45, the transfer pulse φTX3 transitions from high level to low level. As a result, the transfer transistor 503 is turned off, the photodiode 500 is released from the reset state, and the photodiode 500 starts accumulating the signal charge of the image for detection.

At time t46, the reset pulse φRES transitions from high level to low level. As a result, the reset transistor 504 is turned off, and the reset state of the FD region 508 is canceled. At this time, a signal corresponding to the potential due to the residual noise component existing in the FD region 508 is read out to the signal output line 523 via the amplification transistor 505 and the selection transistor 506 .

Since the operation from time t47 to time t50 is the same as the operation from time t7 to time t10 in the timing chart of FIG. 5 of the first embodiment, description thereof will be omitted.
At time t51, the transfer pulse φTX3 transitions from high level to low level. As a result, the transfer transistor 503 is turned off, and the photodiode 500 starts accumulating signal charges of the still image again.

The operation after time t52 is the same as the operation after time t12 in the timing chart of FIG. 5 of the first embodiment, but the transfer pulse φTX1A remains at low level even after the transfer stop operation. As a result, the photodiode 500 transitions to the reset state without transferring the charges exposed and accumulated in the photodiode 500 to the first charge holding portion 507A, as at time t45. On the other hand, when the transfer stop operation is canceled, the transfer pulse φTX1A transitions from low level to high level, as at time t18 in the timing chart of FIG. 5 of the first embodiment. Therefore, the signal charges exposed and accumulated in the photodiode 500 are added to the signal charges accumulated in the first charge holding portion 507A.

Next, the still image shooting operation including the detection operation using the detection image will be described with reference to the flowchart of FIG. FIG. 9 is a flowchart showing still image shooting processing including blur correction using a detection image in the second embodiment. This process is started when the user presses the switch ST154 to give an instruction to shoot a still image. Since the processing from step S901 to step S907 is the same as that from step S601 to step S607 in FIG. 6 of the first embodiment, the description is omitted.

In step S908, blur correction is performed. Details of blur correction will be described later with reference to FIG.
In step S909, the system control CPU 178 determines whether or not the transfer stop FLAG is ON. The transfer stop FLAG is a determination flag indicating a state in which transfer is stopped during still image shooting, and the transfer stop FLAG is ON while the transfer is stopped. The transfer stop FLAG is switched to an ON/OFF state by the system control unit CPU 178 . At the start of the flow, the transfer stop FLAG is turned off by the system control unit CPU 178 . If it is determined in step S909 that the transfer stop FLAG is ON, the process proceeds to step S911. On the other hand, if it is determined that the transfer stop FLAG is OFF, the process proceeds to step S910.

In step S910, the system control CPU 178 turns on the first transfer transistor 501A by a control pulse signal, and transfers the still image signal exposed and accumulated in the photodiode 500 from step S905 to the first charge holding unit 507A. . Then, it is added to the charges already accumulated in the first charge holding unit 507A.

In step S911, the system control CPU 178 turns on the transfer transistor 503 and resets the photodiode 500 by the control pulse signal. If the transfer stop FLAG is determined to be ON in step S909 and the process proceeds to step S911, the charge accumulated in the photodiode 500 is reset without being transferred to the first charge holding unit 507A.

The processing from step S912 to step S915 and step S916 is the same as the processing from step S610 to step S613 and step S614 in the flowchart of FIG. 6 of the first embodiment, so description thereof will be omitted.

FIG. 10 is a flowchart showing blur correction processing in step S908 of FIG. The processing from step S1001 to step S1003 is the same as the processing from step S701 to step S703 in the flowchart of FIG. 7 of the first embodiment, so description thereof is omitted.

In step S1004, by repeating step S908, the system control CPU 178 integrates the blur correction amounts calculated a plurality of times in step S1003 to calculate an integrated blur correction amount. For example, when Bc is the blurring correction amount calculated in step S1003 and Bi is the integrated blurring correction amount, step S1004 performs the processing of the following formula (1).

Bi(i) = Bi(i-1) + Bc Formula (1)

As shown in equation (1), the correction amount Bc is added to the previously calculated integrated shake correction amount B(i−1) to calculate the integrated shake correction amount Bi(i). Note that Bi(0) is initialized to 0 at the start of the main flow.

The integrated blur correction amount Bi(i) calculated in step S1004 is the total amount of blurring after the start of still image shooting, and is the current position of the blur correction lens 185 moved from the start of still image shooting. can be regarded as However, since the blur correction lens 185 can only be driven within a certain stroke range, the blur correction cannot be performed by the blur correction lens 185 when the blur correction amount exceeding the stroke is calculated. Therefore, when the integrated blur correction amount Bi(i) exceeds the blur amount threshold value Bth and exceeds the stroke limit of the blur correction lens 185, the transfer stop FLAG is turned ON to temporarily stop the blur correction operation of the blur correction lens 185. to stop. The blur amount threshold Bth is set according to the stroke of the blur correction lens 185 .

Image blur occurs in an image obtained from charge signals exposed and accumulated in the photodiode 500 for a still image while the blur correction operation is not performed. Therefore, if it is determined in step S909 that the transfer stop FLAG is ON, then in step S910 the photodiode 500 is reset without transferring charges to the first charge holding unit 507A. Then, when it is determined that the integrated blur correction amount Bi(i) has fallen below the blur threshold value Bth, the transfer stop FLAG is turned off. By turning OFF the transfer stop FLAG, blur correction by driving the blur correction lens 185 is restarted.

Also, by turning off the transfer stop FLAG, the charge is transferred to the first charge holding unit 507A in step S910, and is added to the charge already held in the first charge holding unit 507A and held. . By controlling the imaging apparatus 100 in this way, even when a large blurring exceeding the stroke range of the shake correction lens 185 occurs, a still image can be obtained with reduced influence of the blurring correction.

In step S1005, the system control CPU 178 determines whether or not the integrated blur correction amount Bi(i) calculated in step S1004 is greater than or equal to the blur amount threshold Bth. If the integrated blur correction amount Bi(i) is less than the blur amount threshold Bth, the process proceeds to step S1006. On the other hand, if the integrated blur correction amount Bi(i) is equal to or greater than the blur amount threshold Bth, the process proceeds to step S1008.

In step S1006, the system control CPU 178 sets the transfer stop FLAG to OFF.
In step S1007, the blur correction driving unit 186 drives the blur correction lens 185 according to the integrated blur correction amount Bi(i) calculated by the system control CPU 178 to perform blur correction, and the process ends.
On the other hand, in step S1008, the system control CPU 178 sets the transfer stop FLAG to ON, and terminates this process.

In the second embodiment, after the transfer stop FLAG is turned ON, the transfer stop FLAG is turned OFF again when the integrated blur correction amount Bi becomes smaller than the blur threshold value Bth. , the process may be controlled to proceed to step S912 immediately. Even with such control, it is possible to obtain a still image in which the influence of blur correction is reduced. In this case, it is possible that the exposure time of the still image is shorter than the setting and the amount of exposure is insufficient, so gain-up processing may be performed to increase the overall brightness of the acquired still image. However, since the noise is also amplified when the gain-up processing is performed, it is desirable that the gain value of the gain-up processing be suppressed to a predetermined value or less.

In the second embodiment, whether or not the detection image IMG-D(i) is the N-th image is determined in step S912 to determine whether to proceed to step S913. . For example, during the transfer stop operation, the charges exposed and accumulated in the still image photodiode 500 are not transferred to the first charge holding unit 507A. That is, when the transfer stop operation is performed during still image shooting, the process proceeds to step S912 at the timing when the detection image IMG-D(i) reaches the Nth image, and the charge in the first charge holding unit 507A is transferred to the FD. When transferred to area 508, the following occurs. The still image read out in step S914 is an image with an insufficient amount of exposure because the accumulated time of exposure and accumulation of charges is shorter than the exposure time set before the start of the photographing command. Therefore, it is conceivable to have a counter j that counts up each time step S910 is passed, and control to proceed to step S913 when j=N. By controlling in this manner, the still image shooting flow is terminated after the exposure time for which the accumulated time of the exposure accumulation time in the photodiode 500 of the still image signal has been set is reached.

In addition, in the second embodiment, the blur correction lens 185 is provided, and blur correction is performed by calculating the motion vector using the detection image signal. In the case of , it is possible to control as follows. A blur amount threshold Bth used in step S1005 is set as an allowable amount of blur even if it occurs during still image shooting, and it is determined whether or not the integrated blur correction amount Bi(i) is larger than the blur amount threshold Bth. . It is desirable that the allowable blur amount is several tens of micrometers when converted to blur on the imaging plane. If the integrated blur correction amount Bi(i) is smaller than the blur amount threshold value Bth, the process proceeds to step S1006, but since there is no blur correction lens 185, the sub flow is terminated without proceeding to step S1007. By controlling in this way, when the integrated blur correction amount Bi is large, the transfer stop operation is performed, and the signal exposed and accumulated by the still image photodiode 500 is not transferred to the charge holding unit 507A and added. Therefore, it is possible to acquire a still image with reduced influence of blurring. Note that after moving to the transfer stop operation, the process may proceed to step S913 by terminating still image shooting, or the transfer stop operation may be canceled after the integrated blur correction amount Bi becomes less than the threshold as described above. good.

In the second embodiment, the still image and the detection image are acquired using the imaging device 184 having a configuration having two charge holding portions for one photodiode, and the state of image blur is detected using the detection image. Then, image blur correction and transfer stop operation were performed. The object to be detected using the detection image is not limited to image blurring, and may be, for example, an optical state such as an in-focus state. When detecting the in-focus state, for example, the contrast evaluation value of the range including the subject is calculated for a plurality of consecutive images, and it is detected whether the focus lens 181 is in focus with respect to the subject based on the change in the contrast evaluation value. do.

When the transfer stop operation is performed according to the contrast evaluation value, the contrast evaluation value is calculated instead of the blur correction amount, and the difference from the contrast evaluation value of IMG-D(0) is used instead of the integrated blur correction amount. Determine if it is greater than the evaluation threshold. If the difference value of the contrast evaluation values is larger than the contrast evaluation threshold value, it is determined that the amount of blur in the focus state of the subject has exceeded the range, and the transfer stop FLAG is turned ON. Then, control is performed so that the signal exposed and accumulated by the still image photodiode 500 is not transferred to the first charge holding unit 507A. On the other hand, if the difference value of the contrast evaluation values is smaller than the contrast evaluation threshold, it is determined that the object remains in focus, the transfer stop FLAG remains OFF, and the subflow is terminated without proceeding to step S1007. By controlling in this manner, the signal exposed and accumulated in the photodiode 500 when the focus state changes significantly from the in-focus state is not accumulated in the first charge holding unit 507A, so that the focus state of the subject is less deteriorated. A still image can be obtained.

As described above, according to the second embodiment, the charges exposed and accumulated for the still image are transferred from the photodiode 500 to the first charge holding unit 507A in accordance with the photographing state detected from the detection image. Perform control such as stopping and restarting. As a result, it is possible to generate a still image that does not contain electric charges when there is a large blurring or out-of-focus state that may lead to deterioration of the image quality, and it is possible to acquire a higher quality still image.

(Third embodiment)
In the third embodiment, a detection image is used during still image shooting to detect the state of luminance (exposure), and the detection result is used to check whether there is a portion where the exposure is saturated in the still image. to judge. If the still image is saturated if the exposure is continued as it is, the exposure is controlled to end. Since the overall configuration of the imaging device 100 and the circuit configuration of the imaging element 184 in the third embodiment are the same as those in the first embodiment, detailed description thereof will be omitted. Also, the control timing of the imaging element 184 is the same as the timing chart of the first embodiment shown using FIG. 5, so the explanation is omitted. In the third embodiment, the system control CPU 178 functions as luminance detection means (luminance value calculation means and luminance value estimation means).

FIG. 11 is a flowchart showing still image shooting processing including exposure amount detection using a detection image in the third embodiment. This process is started when the user presses the switch ST154 to give an instruction to shoot a still image. The processing from step S1101 to step S1107 is the same as the processing from step S601 to step S607 in the flowchart of FIG. 6 of the first embodiment, so description thereof is omitted.

In step S1108, the system control CPU 178 performs exposure amount detection processing to determine whether the exposure amount is saturated. Details of the exposure amount detection processing will be described later with reference to FIG. 12 .
In step S1109, the system control CPU 178 determines whether or not the saturation FLAG is ON. The saturation FLAG is a judgment flag indicating that saturation will be reached if exposure is continued in a still image during still image photography, and the saturation FLAG is ON when it is judged in the exposure amount detection operation that the saturation is about to occur. become. Also, the saturation FLAG is in an OFF state at the start of the flow. If it is determined in step S1109 that the saturation FLAG is ON, the process advances to step S1113 to transfer the charges in the first charge holding unit 507A to the FD region 508, and end exposure and accumulation of the still image. On the other hand, if it is determined that the saturation FLAG is OFF, the process proceeds to step S1110.

In step S1110, the system control unit CPU 178 determines whether or not the user has issued an instruction to stop still image shooting by turning off the switch ST154. If the switch ST154 is OFF, the process proceeds to step S1113. On the other hand, if the switch ST154 remains ON, the process proceeds to step S1111. The switch ST154 may be a tact switch that remains ON while the user is pressing it and turns OFF when the user stops pressing it, or other switch methods may be used.

In step S1111, the system control unit CPU 178 turns on the first transfer transistor 501A. As a result, the still image signal exposed and accumulated in the photodiode 500 starting from step S1105 is transferred to the first charge holding unit 507A and added to the charge already accumulated in the first charge holding unit 507A. .
The operations from step S1112 to step S1115 and step S1116 are the same as the operations from step S610 to step S613 and step S614 of the first embodiment, and thus description thereof will be omitted.

FIG. 12 is a flow chart showing the exposure amount detection processing in step S1108 of FIG.
In step S1201, the system control unit CPU 178 detects the brightness value of the read detection image IMG-D(i), gains up the detection image IMG-D(i) with a predetermined correction value Gi, Increases the luminance value of the entire image. The predetermined correction value Gi is determined by the ratio of one exposure accumulation time for the still image signal periodically performed by the photodiode 500 and one exposure accumulation time for the detection image. Specifically, the correction value Gi is obtained by the following equation (2), where Ts is the time from time t1 to t4 in the timing chart of FIG. 5, and Td is the time from t5 to t10.

Gi=Ts/Td Expression (2)

By increasing the gain with the correction value Gi, the luminance of the detection image obtained by one exposure accumulation in the photodiode 500 is corrected to be equivalent to the accumulated luminance of the still image obtained by one exposure accumulation. can be done.

In step S1202, the system control unit CPU 178 adds the gain-increased IMG-D(i) in step S1201 to the integrated image IMG-I to calculate an integrated image based on the detection image signal. Integrated image IMG-I is an image having the same number of pixels as detection image IMG-D(i). All pixels of the integrated image IMG-I are set to 0 at the start of shooting. By repeatedly integrating the gain-up detection image signal IMG-D(i), the integrated image IMG-I is a still image that can be generated from the charge signal of the still image signal accumulated in the first charge holding unit 507A. Luminance values can be estimated. That is, it is possible to obtain a virtual image with an exposure amount equivalent to that of the still image as the integrated image IMG-I.

In step S1203, the system control unit CPU 178 displays the integrated image IMG-I created in step S1202 on the display unit 153, and displays to the user a virtual image of a still image assumed to be obtained with the current exposure time. .
In step S1204, system control unit CPU 178 determines whether or not the maximum luminance value among all pixels of integrated image IMG-I is equal to or greater than threshold value Dth. If IMG-I is less than the threshold value Dth, it is determined that the still image is not yet saturated, and the flow ends. On the other hand, if IMG-I is equal to or greater than the threshold value Dth, it is determined that the still image is about to be saturated, and the process advances to step S1204. The threshold value Dth is set to the maximum value that can be set for the pixel value of the still image signal, that is, the saturation value, or a value slightly smaller than the saturation value.
In step S1204, the system control unit CPU 178 turns ON the saturation FLAG and ends the flow.

In the third embodiment, the detection image is used to generate an integrated image IMG-I, which is a virtual image of a still image, and the generated integrated image IMG-I is used to accumulate in the first charge holding unit 507A. It is determined whether the still image signal is close to saturation. Based on this determination result, it is determined whether or not to end the still image shooting, and the transfer of the still image signal to the FD area 508 is controlled. That is, when the luminance of the integrated image IMG-I is equal to or higher than a predetermined value, the first transfer transistor 501A stops transferring the charge to the charge holding portion 507A, and the second transfer transistor 502A transfers the charge to the first charge holding portion 507A. is transferred to the FD area. Therefore, in still image shooting, the still image can be obtained with an appropriate amount of exposure without saturation.

Further, by displaying a virtual image of the current still image signal on the display unit 153 during still image exposure, the user can determine the exposure amount of the virtual image and end the still image shooting at a desired timing. A high-quality still image desired by the user can be obtained. Note that in the case of shooting without setting the still image exposure time before the start of shooting, such as bulb shooting, the process should always proceed from step S1111 to step S1116 without making the determination in step S1112. By controlling the imaging apparatus 100 in this way, it is possible to detect whether or not the still image is saturated during still image exposure without providing a separate sensor for detecting the exposure state. As a result, it is possible to obtain a still image with an appropriate amount of exposure even in long-exposure photography where the subject changes during the exposure, such as exposing a number of fireworks into a single still image. can.

Note that, in the first to third embodiments, the imaging element 184 having the configuration having the first charge holding portion 507A and the second charge holding portion 507B for one photodiode 500 is used to obtain a still image. An example of acquiring an image for detection has been described. However, the present invention is not limited to this, and any imaging device capable of acquiring a still image and a detection image in parallel and controlling charge accumulation for the still image according to the detection result using the detection image can be used. good. For example, by using an imaging device having one charge holding unit for one photodiode, which will be described later with reference to FIG. You can get them in parallel.

(Fourth embodiment)
In the first to third embodiments, a still image and a detection image are acquired, an imaging state is detected using the detection image, and charge accumulation for generating a still image according to the imaging state is performed. took control. In this embodiment, an embodiment in which two images with different frame rates are acquired will be described using a first moving image with a low frame rate and a second moving image with a high frame rate as an example.

Since the overall configuration of the imaging apparatus 100 in this embodiment is the same as in the first embodiment, detailed description thereof will be omitted. FIG. 13 is a circuit diagram showing a configuration example of pixels of the imaging device 184 in this embodiment. The pixel of this embodiment has only the first charge holding portion 507A for one photodiode 500 . That is, one charge holding portion is provided for one photoelectric conversion portion. In addition, since the details of each configuration are similar to those of the first embodiment, the same reference numerals are used to omit detailed description thereof.

FIG. 14 is a diagram for explaining setting screens for shooting conditions for the first moving image (picture A) and the second moving image (picture B). The setting screen is displayed on the display unit 153 of the imaging device body 151 . By rotating the photographing mode selection lever 156 clockwise, a dual photographing mode is entered in which two moving images can be photographed simultaneously. The setting screen displays luminance, lens aperture value, ISO sensitivity, shutter speed, and playback mode for the first moving image and the second moving image. Each item is set by using the up switch 158, the down switch 159 and the dial 160 to select from among a plurality of options the one that suits the purpose of photography. Also, when some items are input, the remaining items may be automatically calculated and candidates may be displayed.

The brightness of the object at that time is displayed as the Bv value 1321 in the brightness. The set lens aperture value is displayed as the F number 1322 in the lens aperture value. The ISO sensitivity displays the ISO sensitivity 1323 of the first moving image and the ISO sensitivity 1324 of the second moving image. Shutter speed 1325 of the first moving image and 1326 of the second moving image are displayed. In the playback mode, a playback mode 1327 for the first moving image and a playback mode 1328 for the second moving image are displayed. In this embodiment, an example will be described in which the playback mode 1327 of the first moving image is set to "Normal", which is normal playback, and the playback mode 1328 of the second moving image is set to "Slow", which is slow playback. The second moving image for which “Slow” is set in the reproduction mode 1328 corresponds to slow reproduction, and therefore acquires a moving image with a higher frame rate than the first moving image.

FIG. 15 is a diagram for explaining the setting of shooting conditions for the first moving image and the second moving image. As an example, setting item 1 and setting item 2 are shown in which the shutter speed of the second moving image is changed. For the shutter speed of the second moving image with a high frame rate, for example, a shutter speed is set such that fine images whose movement stops during slow-motion reproduction can be continuously reproduced. In this embodiment, the shutter speed of the second moving image is set to 1/250 second or 1/500 second depending on the moving speed of the subject to be shot. The shutter speed for the first moving image with a low frame rate is set to 1/60 second, which allows obtaining a smooth image with no jerkyness with respect to the frame rate.

In general, a first moving image and a second moving image having different frame rates shot at the same aperture value at the same time have different shutter speeds, and therefore are controlled to have different ISO sensitivities. Therefore, if exposure control is performed so that the second moving image with a high frame rate is properly exposed, the first moving image with a low frame rate will be saturated and the ISO sensitivity cannot be controlled. Therefore, in the present embodiment, time-division accumulation is performed in which a moving image is generated by adding Np times of short accumulations at regular intervals during 1/60 second for acquiring the first moving image with a low frame rate. By performing time-division accumulation, the accumulation period is 1/60 second without any feeling of unevenness, but the actual accumulated amount of light exposed is less than 1/60 second, and the ISO sensitivity is substantially reduced. ing. By time-division accumulation, the ISO sensitivity of the first moving image and the second moving image is 100 in this embodiment.

As shown in setting item 1, when the Bv value 1321 is 7, the aperture value 1322 is F4.0, and both the ISO sensitivity 1323 of the first moving image and the ISO sensitivity 1324 of the second moving image are 100, A shutter speed 1326 of 1/250 sec provides proper exposure. Also, the number of times Np of accumulation of the first moving image is assumed to be 16 here. The shutter speed 1325 of the first moving image, which is normally reproduced, needs to be 1/60 seconds without a crisp feeling, but the total exposure amount needs to be 1/250 seconds, which is the same as the second moving image. The accumulation time of one moving image is 1/250 seconds/16=1/4000 seconds.

As described above, in this embodiment, the total accumulation time of the first moving image is controlled to be the same as the accumulation time of the second moving image. That is, the total accumulation time of the first moving image generated by adding the short accumulations Np times in the first charge holding unit 507A of the image sensor 184 is equal to the accumulation time of the second moving image. Therefore, the exposure of the first moving image shot in the same shooting cycle will be the same as the exposure of the second moving image, so that there will be no difference in image quality between the two without gain-up operation by ISO sensitivity. becomes possible.

16 and 17 are diagrams for explaining accumulation and readout timings in the imaging device 184 for simultaneously shooting the first moving image and the second moving image. 16 corresponds to setting item 1 shown in FIG. 15, and FIG. 17 corresponds to setting item 2. In FIG. Here, accumulation is performed by transferring charges generated in the photodiode 500 to the first charge holding portion 507A or the second charge holding portion 507B. Further, reading refers to outputting the charges held in the first charge holding portion 507A or the second charge holding portion 507B to the outside of the image pickup device 184 via the FD region 508 .

16 and 17 are conceptual diagrams, and the rows shown are not actual rows of pixels of the image sensor 184, but rows to be read out at the same time are collectively expressed in one row. For example, when the pixels of the imaging device 184 are arranged in a Bayer array, two rows of GB and RG are shown as one row in FIGS. 16 and 17 . Also, although FIGS. 16 and 17 show the timing of several lines for the sake of convenience, the actual imaging device 184 has thousands of lines. In FIGS. 16 and 17, the last line is line m.

16 and 17, with the horizontal axis representing time, a vertical synchronization signal 1650, a horizontal synchronization signal 1651, a first accumulation period 1663, a first transfer period 1664, a first readout period 1666, and a second accumulation period. 1661, a second transfer period 1662, and a second readout period 1665 are shown. In this embodiment, the first moving image and the second moving image are read out during one shooting period of the vertical synchronization signal 1650 .

Here, the first accumulation period 1663 indicates the accumulation period of signal charges in the photodiode 600 for the accumulation period of the first moving image. A first transfer period 1664 indicates a period during which the signal charge for the first moving image is transferred from the photodiode 600 to the first charge holding portion 607A. A first readout period 1666 is a readout period for the first moving image. A second accumulation period 1661 indicates a period during which signal charges are accumulated in the photodiode 600 for the second moving image accumulation period. A second transfer period 1662 indicates a period during which the signal charge for the second moving image is transferred from the photodiode 600 to the second charge holding portion 607B. A second readout period 1665 is a readout period for the second moving image.

Charge accumulation, transfer, and readout when the photographer sets the setting item 1 will be described with reference to FIG. First, accumulation will be explained. Here, it is assumed that the photographer selects T1=1/250 second as the shutter speed of the second moving image so that slow motion reproduction suitable for the moving speed of the subject is performed. That is, it is assumed that setting item 1 in FIG. 15 is set. Therefore, accumulation is performed Np times (Np=16) in the first accumulation period 1663 for the first moving image with a low frame rate in one imaging cycle (time Tf=1/60 second). Note that the first accumulation period 1663 is T1 (=1/250 seconds)/16, which is 1/4000 seconds. The 16 accumulations are added in the first charge holding unit 607A, and the total exposure amount becomes equivalent to the case where the accumulation of 1/250 seconds is performed once. In addition, 16 accumulations are performed, but substantially one long accumulation period is from the start time of the first accumulation period to the end time of the 16th accumulation period. Therefore, for the first moving image, it is possible to obtain a smooth image in which the jerky feeling is suppressed.

The first row shows signal charge accumulation for the first low frame rate movie. Accumulation is performed in the first accumulation period 1663 divided into Np times (Np=16) during one period (time Tf=1/60 second) of the vertical synchronizing signal 1650 . The second line shows signal charge accumulation for the high frame rate second movie. Of the two accumulations during one cycle (time Tf=1/60 sec) of the vertical synchronizing signal 1650, accumulation is performed during the second accumulation period 1661 at T1=1/250 sec.

The third line, like the first line, shows accumulation of signal charges for the low frame rate first moving image. The fourth line, like the second line, shows accumulation of signal charges for the high frame rate second moving image. However, the second line is accumulation in the latter half of one cycle (time Tf=1/60 second) of the vertical synchronizing signal 1650, while the fourth line is accumulation in the first half of one cycle of the vertical synchronizing signal 1650. . In this way, the second moving image has different accumulation rows for each shooting cycle. Thereafter, accumulation of signal charges for the first moving image is performed in the (2N+1) row, and accumulation of signal charges for the second moving image is performed in the (2N) row. Note that the number of thinned lines in the second moving image is two, which is larger than the one line in the first moving image.

Next, reading will be described. In the first row, as indicated by the first readout period 1666, the low frame rate first moving image accumulated in the first accumulation period 1663 in one cycle of the vertical synchronization signal 1650 is read out in the next vertical It starts at the beginning of the sync signal 1650 period. The readout of the first moving image in the third row is performed following the readout of the first row, and the readout of the subsequent (2N+1) rows is sequentially performed in the same manner. Therefore, it takes 1/2 of one cycle Tf of the vertical synchronizing signal 1650 to read out the last row.

In the second row, as indicated by the second readout period 1665, the readout of the high frame rate second moving image accumulated in the latter half of one cycle of the vertical synchronization signal 1650 is performed in the next cycle of the vertical synchronization signal 1650. starting from half of The readout of the second moving image in the 6th row is performed following the readout of the 2nd row, and the readout of the subsequent 2 (2N+1) rows is sequentially performed in the same manner. Therefore, it takes 1/4 of one cycle Tf of the vertical synchronizing signal 1650 to read out the last row.

In the fourth row, as indicated by the second readout period 1665, the high frame rate second moving image accumulated in the second half of one cycle of the vertical synchronization signal 1650 is read out in the next cycle of the vertical synchronization signal 1650. starting from three-quarters of the The readout of the second moving image in the 8th row is performed following the readout of the 4th row, and the readout of the subsequent 2 (2N+2) rows is sequentially performed in the same manner. Therefore, it takes 1/4 of one cycle Tf of the vertical synchronizing signal 1650 to read out the last row.

The readout time to the last line of each video is 1/2 of one cycle for the first video, 1/4 of one cycle for 2(2N+1) lines of the second video, and 2(2) for the second video. 2N+2) rows take 1/4 of one period Tf. Therefore, readout of all pixels is completed within one period.
Note that the accumulation timing of the second moving image generated in the (2N+1) row is divided into the first half and the latter half within the range of the shooting cycle every two rows. This is for prevention, and there is no change in the amount of exposure for each row.

The accumulation time of the second moving image is set to the shutter speed T1 set by the photographer. The accumulation end time of the second moving image is fixed for all rows (Tf time from the vertical synchronizing signal 1650), and is set so that the accumulation ends immediately before the second readout period 1665 starts. Since the accumulation end time of the second moving image is fixed for all lines, the accumulation start time of the second moving image with respect to the vertical synchronization signal 1650 is set according to the shutter speed T1 of the second moving image. Also, the accumulation end time of the second moving image is set to be less than half the time Tf of the vertical synchronization signal 1650 .

In addition, the accumulation start time of the moving image of each line is fixed with respect to the vertical synchronization signal 1650, and the accumulation end time of the first moving image is changed according to the shutter speed T1 of the second moving image set by the photographer. is set relative to vertical sync signal 1650 . In this embodiment, the first moving image having a long shooting cycle is accumulated Np times in a time-division manner, and the time-division accumulation time is determined according to the shutter speed of the second moving image.

Also, in the first moving image with a low frame rate, the “shooting cycle” is time Tf=1/60 second, and the same accumulation timing is repeated. The “shooting cycle” in the high frame rate second moving image is, for example, the first half and the second half within the time Tf. Therefore, in the second moving image, the accumulation rows are different for each shooting cycle.

Next, a case where the moving speed of the subject is fast will be described. In order to realize slow-motion reproduction suitable for a fast-moving subject, the photographer sets the shutter speed T2 of the second moving image to be shorter than the shutter speed T1 (for example, T2=1/500 second). Setting item 2 in FIG. 15 shows this setting. Compared with setting item 1, the aperture value 1322 of setting item 2 is F2.8, which is one step open. The Bv value 1321, the first moving image ISO sensitivity 1323, the second moving image ISO sensitivity 1324, and the first moving image shutter speed 1325 are common to the setting item 1. Therefore, one accumulation time of the first moving image is set to T2/16 (=1/8000 second). Since the total exposure of the first moving image is 1/8000 seconds×16=1/500 seconds, it is equal to the exposure of the second moving image.

Charge accumulation, transfer, and readout when the photographer sets the setting item 2 will be described with reference to FIG. First, accumulation will be explained. The first moving image with a long shooting cycle is accumulated Np times in a time-division manner, and the time-division accumulation time of the first image is determined according to the shutter speed of the second moving image. In this embodiment, one accumulation time of the first moving image is set to T2/16 (=1/8000 seconds). Also, in FIG. 17, the number of thinned lines in the second moving image is four, which is larger than the one line in the first moving image. Also, in the second moving image, the accumulation row is different for each shooting cycle.

Also, as described with reference to FIG. 16, the accumulation end time of the second moving image is fixed for all lines, and is set so that the accumulation ends immediately before the start of readout of the image of the first line. Since the accumulation end time of the second moving image is fixed for all lines, the time-division accumulation time of the first moving image with respect to the vertical synchronization signal 1650 is set according to the shutter speed of the second moving image.

As in FIG. 16, the accumulation of the first moving image is performed at equal time intervals during one cycle, and the time intervals are set so that the accumulation divided into 16 times is completed by the time immediately before the start of reading out the moving image of each row. be done. At this time, the time interval for accumulating the first moving image is set to an integral multiple of the interval Th of the horizontal synchronizing signal 1651 . As a result, the accumulation timing of moving images in each row is the same. The accumulation start time of the moving image of each row is fixed with respect to the vertical synchronization signal 1650, and the accumulation end time of one time of the first moving image is vertical synchronization according to the shutter speed of the second moving image set by the photographer. Set for signal 1650 .

Next, reading will be described. In the first moving image with a low frame rate, it takes 1/2 of one cycle Tf to read up to the last row, as in the description of FIG. 16 .
In the second movie with high frame rate: First, the 2(4N+1) row will be described. In the second line, readout of the high-frame-rate second moving image accumulated in the latter half of the period of one cycle Tf of the vertical synchronization signal 1650 is started from half of the period of the next vertical synchronization signal 1650 . In the next line 10, readout of the second moving image follows the readout of the second line. The reading of subsequent 2 (4N+1) rows also operates in the same manner. Therefore, it takes ⅛ of one cycle to read data up to the last row.

Next, 2 (4N+2) rows will be described. In the fourth line, the readout of the second moving image with the high frame rate accumulated at the end of the period of one cycle Tf of the vertical synchronization signal 1650 follows the readout of the last row of accumulation in the next period of the vertical synchronization signal 1650. is started. In the next line 12, the readout of the second moving image follows the readout of the 4th line. The reading of subsequent 2 (4N+2) rows also operates in the same manner. Therefore, it takes ⅛ of one cycle Tf to read out data up to the last row.

Next, the 2(4N+3) row will be described. In the sixth row, the readout of the high frame rate second moving image accumulated in the first half of the period of one cycle Tf of the vertical synchronization signal 1650 is the last row of 2 (4N+2) rows in the same period of the vertical synchronization signal 1650. It is started following readout. In the following line 14, readout of the second moving image follows the readout of line 6. The reading of subsequent 2 (4N+3) rows also operates in the same manner. Therefore, it takes ⅛ of one cycle Tf to read out data up to the last row.

Next, the 2(4N+4) row will be described. In the eighth row, the high frame rate second moving image stored in the first half of the period of one cycle Tf of the vertical synchronizing signal 1650 is read out in the last row of 2 (4N+3) rows in the same period of the vertical synchronizing signal 1650. It is started following readout. In the next 16 lines, the readout of the second animation follows the readout of line 8. The reading of subsequent 2 (4N+4) rows also operates in the same manner. Therefore, it takes ⅛ of one cycle to read data up to the last row.

The readout time up to the last line of the first moving image is 1/2 of one period Tf of the vertical synchronization signal 1650 . The readout time to the last row of the second moving image is 1/8 of one cycle Tf of the vertical synchronization signal 1650 for each of the 2(4N+1) to 2(4N+4) rows, so the total is one cycle of the vertical synchronization signal 1650. 1/2 of Tf. Therefore, the readout of all pixels, which is the sum of the readout of the first moving image and the readout of the second moving image, is one period Tf of the vertical synchronization signal 1650 .

In the present embodiment, the first moving image is stored row by row, and the second moving image is stored in 2 (4N+1) rows, 2 (4N+2) rows, 2 (4N+3) rows, and 2 (4N+4) rows, for a total of 4 rows. Lines are thinned out and accumulated. That is, the number of lines to be thinned out in the second moving picture is four, which is larger than the number of lines to be thinned out in the first moving picture. Note that the accumulation timing of the second moving images generated in the 2 (4N+1) row, the 2 (4N+2) row, the 2 (4N+3) row, and the 2 (4N+4) row is one cycle Tf of the vertical synchronization signal 1650, which is the shooting cycle. It is divided into four within the range of This is to prevent imbalance within the shooting cycle, and the amount of exposure for each row remains the same.

As described above, in an imaging device capable of shooting at a plurality of different frame rates, the shutter speed of the first moving image with a low frame rate is 1/30 at a shutter speed of about 30 fps, which does not give a jerky feeling in normal playback. , or 1/60 can always be selected. On the other hand, the shutter speed of the second moving image with a high frame rate can be freely selected by the photographer, depending on the moving speed of the subject, such that slow-motion reproduction suitable for it is performed. In addition, since it is not necessary to change the ISO sensitivity when selecting the shutter speed, it is possible to suppress deterioration in image quality due to a gain increase or the like.

FIG. 18 is a diagram showing a screen displayed on the display unit 153. As shown in FIG. A fencing person 163 captured through the imaging optical system 152 is displayed on the display unit 153 . FIG. 18A shows a state in which normal reproduction is selected and the first moving image is displayed. Display unit 153 displays a first moving image (picture A) as normal reproduction, and shutter speed 1791 and F number 1793 for shooting the first moving image. The first moving image is shot at a low frame rate of 1/60 second, and is played back as a smooth image without any blurriness.

FIG. 18B shows a state in which slow playback is selected and the second moving image is displayed. The display unit 153 displays a second moving image (picture B) as slow reproduction, and a shutter speed 1791 and an F number 1793 at which the second moving image was captured. The second video was filmed at a high frame rate of 1/250 second, and slow-motion playback makes it possible to see fencing strikes that were difficult to see with the naked eye. The terminal that reproduces the first moving image and the second moving image is not limited to the imaging device 100, and may be displayed on an external device 197 such as a tablet terminal, a personal computer, or a television monitor.

FIG. 19 is a diagram illustrating an example of utilization of the first moving image and the second moving image. The data files of the first moving image (picture A) and the second moving image (picture B) are stored in the recording medium 193, storage on the network, or the like. The first moving image and the second moving image are saved, for example, as MP4 files, and are associated with each other by setting the same CLIP-UMID at the time of shooting. A first frame group 1801 is a frame group of a low frame rate first moving image (picture A). A second frame group 1811 is a frame group of a high frame rate second moving image (picture B).

The second video (picture B) was shot at 1/250 sec, four times faster than the frame rate of the first video (picture A), 1/60 sec. Therefore, the frame of the second moving image corresponding to one frame 1802 of the first moving image is a group of four images of frames 1821 to 1824 . Similarly, the frames of the second moving image corresponding to frame 1803 of the first moving image are frames 1831-1834.

During normal playback, when the user performs an operation to switch the playback mode, such as tapping the screen, the playback is switched to slow playback. That is, when an operation to switch the reproduction mode is performed during reproduction of the first moving image, the corresponding second moving image is reproduced. For example, suppose you are filming a fencer and want to view the image at the moment of the strike. First, when reproduction of a moving image is started, the frames of the first frame group 1801 are sequentially reproduced at a determined frame rate from the first frame 1802 of the frame group 1801 of the first moving image.

Playback progresses to frame 1803 near the desired scene, and when the user taps the screen, frame 1831 with the same time code is retrieved from the data file of the second moving image corresponding to the first moving image. Then, while the user keeps tapping the screen, the frames of the second frame group 1811 are reproduced in slow motion from the frame 1831 in order of the frames 1832, 1833, and 1834. FIG. Slow-motion playback allows you to see things like fencing strikes that were difficult to see with the naked eye.

Then, when the user stops tapping at frame 1934, normal playback of the first moving image is resumed, and playback resumes from the next frame 1804 of the same time code of the first moving image corresponding to the second moving image. In the present embodiment, when the screen is tapped during normal playback in which the video of the first video is displayed on the same screen, the video of the second video is played in slow motion while the screen is continuously tapped. . It is not limited to this example, and as shown in FIG. 20 , when the screen is tapped during normal playback in which the video of the first video is displayed, the video is switched to the video of the second video, and the video of the second video is displayed according to the user's operation. You may enable it to reproduce|regenerate the moving image by frame advance. Also, if there are two display sections, normal playback may be displayed on one screen, and slow motion playback may be performed on the other screen when this is tapped. With this configuration, two displayed moving images can be viewed at once, but in this embodiment, the exposure of the two moving images is the same, so no problem occurs.

Alternatively, the second moving image with a high frame rate may be reproduced frame by frame, a desired frame may be selected, and an image at a desired moment may be printed. After selecting a frame, when printing is decided, the data of the selected frame is output to the printer 195 via the print I/F 194 . Images taken with a fast shutter speed can be printed, so the printed material will have a powerful stop-motion effect.

In the fourth embodiment, an imaging device having a configuration in which one charge holding unit is provided for one photodiode is used, and different rows are used for accumulating charges in the first moving image and the second moving image. An example of acquiring videos with different frame rates in parallel has been explained. However, the present invention is not limited to this, and any imaging device that can acquire moving images with different frame rates in parallel may be used. For example, as shown in FIG. 3, using the imaging element 184 having a configuration having a first charge holding portion 507A and a second charge holding portion 507B for one photodiode 500, moving images with different frame rates are displayed in parallel. may be obtained by Even when an image pickup device having a plurality of charge holding portions for one photodiode is used, the low frame rate moving image is time-divided so that the exposure of the low frame rate moving image matches the exposure of the high frame rate moving image. By controlling, it is possible to obtain a movie with uniform exposure.

As described above, according to the present embodiment, a smooth high-quality moving image for normal playback and a high-frame-rate moving image for slow-motion playback are processed in parallel with the same exposure. can be obtained.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist thereof.

100 imaging device 184 imaging device 178 system control CPU

Claims (6)

For one photoelectric conversion unit, a first charge holding unit that accumulates charges for generating a first image; an imaging device in which pixels each having a second charge holding portion for accumulating charges are arranged two-dimensionally;
The imaging device is controlled to transfer the charge from the one photoelectric conversion unit to the first charge holding unit in a plurality of divided times during one imaging period, and transfer the charge from the one photoelectric conversion unit to the second charge holding unit. The second image is obtained by transferring the charge to the charge holding unit a plurality of times and reading a signal corresponding to the charge of the second charge holding unit by dividing the image capturing period a plurality of times. After completion of readout of the signal corresponding to the charge of the second charge holding unit divided into multiple times, the signal corresponding to the charge of the first charge holding unit is read out to obtain the first image. a obtaining means for
detection means for detecting motion vectors from the plurality of second images;
and correction means for correcting image blur in the first image based on the motion vector detected by the detection means. The detection means detects luminance from a plurality of the second images,
2. The imaging apparatus according to claim 1 , wherein said obtaining means controls accumulation of electric charges corresponding to said first image according to said luminance. 2. The imaging apparatus according to claim 1, wherein said acquisition means determines a time-division accumulation time of said first image according to a shutter speed of said second image. the first image and the second image are thinned-out images;
4. The acquisition unit according to any one of claims 1 to 3 , wherein the accumulation of charges corresponding to the first image and the accumulation of charges corresponding to the second image are performed in different rows of the imaging device. 1. The imaging device according to claim 1. 5. The image pickup apparatus according to claim 4 , wherein said acquisition means changes the row of said image pickup device for accumulating electric charges corresponding to said second image according to an imaging cycle. For one photoelectric conversion unit, a first charge holding unit that accumulates charges for generating a first image; A control method for an image pickup device having an image pickup device in which pixels each having a second charge holding portion for accumulating charge are arranged two-dimensionally,
The imaging device is controlled to transfer the charge from the one photoelectric conversion unit to the first charge holding unit in a plurality of divided times during one imaging period, and transfer the charge from the one photoelectric conversion unit to the second charge holding unit. The second image is obtained by transferring the charge to the charge holding unit a plurality of times and reading a signal corresponding to the charge of the second charge holding unit by dividing the image capturing period a plurality of times. After completion of readout of the signal corresponding to the charge of the second charge holding unit divided into multiple times, the signal corresponding to the charge of the first charge holding unit is read out to obtain the first image. an obtaining step for
a detecting step of detecting motion vectors from a plurality of said second images;
and a correction step of correcting image blur in the first image based on the motion vector detected in the detection step.

Images