JP4437503B2 - Imaging device - Google Patents

Description

  The present invention relates to an image pickup apparatus using a solid-state image pickup device such as a CCD image sensor, and relates to a still image pickup apparatus having a blur correction function for a still image.

  It is a well-known phenomenon that when taking an image of a subject, even if the subject is stationary, the captured image is blurred by moving the imaging device when the photographer presses the shutter button. Is generally expressed as “camera shake”. The occurrence of “camera shake” increases as the illuminance of the environment in which the subject is imaged decreases and the shutter speed decreases (the charge accumulation period increases).

  A technique for improving the quality of a captured image by correcting the captured image due to camera shake mechanically or by image processing to reduce image degradation due to camera shake has been widely studied. For example, there is a technique that detects a displacement (angle) of an imaging apparatus using a sensor that detects angular velocity or angular acceleration, and corrects camera shake by moving an optical lens or a solid-state imaging device itself in a direction to cancel the displacement. .

  For example, an image blur detection unit using an angular velocity sensor is provided, and a unit for driving and controlling the correction optical system in a direction to reduce or eliminate the image blur according to the camera shake detection result and the position detection result of the correction optical system is provided. An imaging device including a correction device is known (see, for example, Patent Document 1).

  In another conventional imaging apparatus, camera shake correction processing by image processing is performed. For example, the charge accumulation period is controlled to give a plurality of consecutive exposures, and a plurality of obtained captured images are added to generate one image. There is known an imaging apparatus provided with an image processing means that performs an automatic motion compensation and corrects a subject shift between images (see, for example, Patent Document 2).

  In addition, the image pickup apparatus includes a camera shake correction processing unit based on the same image processing, and performs motion prediction and motion detection based on a motion vector of continuous inter-field images generated by camera shake, and extracts an image clipping position (from an image memory). There is also known one that corrects camera shake by changing the reading position (see, for example, Non-Patent Document 1).

JP-A-7-98466 (pages 1-8, FIG. 1) JP 2001-86398 A (page 1-9, FIG. 1) Hiroya Kusaka, Yasunori Kishi, “High Resolution Pure Electronic Image Stabilization System”, Journal of the Institute of Image Information and Television Engineers, 2002, Vol. 56, no. 10, p. 1663-1668

  However, the conventional imaging device described in Patent Document 1 requires a mechanical component for canceling vibration caused by camera shake, and also requires an actuator and a motor for driving the device. There is a problem that power consumption and cost increase as the size increases. Furthermore, since the mechanical parts are affected by the drop impact, it is necessary to design for impact.

  Further, in another conventional imaging device described in Patent Document 2, a plurality of captured images obtained by giving a plurality of consecutive exposures are added, and thus all of the plurality of captured images are stored. A separate frame memory is required.

  Furthermore, as described in Non-Patent Document 1, when camera shake correction is performed based on the motion vectors of continuous inter-field images, it is effective for moving images, but occurs in one image. It cannot be used to correct camera shake on a still image. Further, since an image is cut out from the image memory, a necessary memory capacity is also increased.

  Note that image blur caused by reasons other than camera shake, such as image blur caused by relative movement of the imaging device with respect to the subject when the imaging device is mounted on a moving body such as a vehicle and imaging is performed while moving, is performed. There is a similar problem.

  The present invention has been made to solve the above-described problems, and does not require a special mechanism component or a large-capacity memory, and is small and low-cost by performing blur correction in a solid-state imaging device. An object is to provide an imaging device.

The present invention
A photoelectric conversion unit that is arranged at a plurality of pixel positions aligned in the first direction on the imaging screen and includes photoelectric conversion elements that perform photoelectric conversion by receiving light from a subject, and pixels aligned in the first direction A solid-state imaging device having a transfer unit provided corresponding to each of the photoelectric conversion elements at the position, and having a plurality of transfer elements that transfer charges stored in the corresponding photoelectric conversion elements and transfer charges between them When,
Movement detecting means for detecting movement of the solid-state imaging device in the first direction;
First to Nth (N is an integer of 2 or more) in a predetermined frame period corresponding to an optical image formed on the solid-state imaging device according to the amount of motion detected by the motion detection means. Combining means for reading out the first image signal and shifting the read-out first to (N-1) th image signals by the number of pixels corresponding to the detected amount of motion. Have
The synthesis means includes
In accordance with the amount of motion per predetermined time detected by the motion detection means, electric charge is accumulated in the photoelectric conversion element in the first to Nth image generation periods that follow each other,
The charge accumulated in the photoelectric conversion element during the first image generation period is read from the photoelectric conversion element to the corresponding transfer element, and the transfer element is transferred between the transfer elements by a first movement amount related to the amount of movement. Forward with
When N is larger than 2, the electric charge accumulated in the photoelectric conversion element during the n-th (n is an integer of 2 to (N−1)) image corresponds to the charge from the photoelectric conversion element. Read to the transfer element, add the read charge to the charge transferred by the (n-1) th movement amount related to the amount of movement, and add the charge obtained as a result of the addition The process of transferring between the transfer elements by the nth movement amount related to the amount of movement is sequentially executed from 2 to (N−1),
The charge accumulated in the photoelectric conversion element during the N-th image generation period is read from the photoelectric conversion element to the corresponding transfer element, and the read charge is related to the amount of motion. N-1) is added to the transferred charge, and the resulting charge is output from the solid-state imaging device as a combined image signal,
The synthesizing unit determines the number of pixels to shift the image signal in each frame period to a value corresponding to the amount of motion detected by the motion detecting unit in a frame before the frame period. An imaging device is provided.

  According to the present invention, it is possible to provide a small-sized and low-cost imaging device by performing blur correction in a solid-state imaging device without requiring a special mechanism device or memory.

  Hereinafter, the case where the image blur is caused by camera shake will be described. However, the present invention is also applicable to image blur that occurs for reasons other than camera shake. For example, the present invention can be applied to a camera that is mounted on an automobile or other moving body and performs shooting while moving and the direction of the main movement is known in advance.

Embodiment 1 FIG.
FIG. 1 is a diagram showing a configuration of an imaging apparatus according to Embodiment 1 of the present invention. As shown in the figure, the imaging apparatus includes a lens 1, a solid-state imaging device 2, an analog signal processing unit 3, a pixel signal A / D conversion unit 4, a signal processing unit 5, a display unit 6, a CPU 7, The timing generator 8, the camera shake sensor 9, the motion signal A / D conversion unit 10, the integrating unit 11, and the lens driving unit 12 are included.

  A subject image incident through the lens 1 is photoelectrically converted by the solid-state imaging device 2. This lens 1 functions as a zoom lens having a variable focal length. By the control signal from the lens driving unit 12, the lens 1 can move back and forth on the optical axis, and the focal length can be continuously changed. This control signal is output from the CPU 7.

  As shown in FIG. 2, the solid-state imaging device 2 includes a photoelectric conversion unit 19 having a plurality of photoelectric conversion element arrays 20 and a plurality of vertical transfer units 22 provided adjacent to each other corresponding to the photoelectric conversion element arrays 20. A readout gate unit 24 positioned between the photoelectric conversion element array 20 and the vertical transfer unit 22, a horizontal transfer unit 27, and an output amplifier 28.

Each photoelectric conversion element array 20 is arranged at a plurality of pixel positions in a first direction on the imaging screen, for example, the vertical direction, and receives light from a subject (not shown) via the lens 1 to perform photoelectric conversion. It has a plurality of photoelectric conversion elements 21 for performing.
Each vertical transfer unit 22 is provided corresponding to each photoelectric conversion element 21 of the corresponding photoelectric conversion element array 20, and the electric charge accumulated in the corresponding photoelectric conversion element 21 is read by each read gate 25 of the read gate unit 24. And a plurality of transfer elements 23 that transfer charges between each other in the vertical direction.
The horizontal transfer unit 27 receives the electric charge transferred from the vertical transfer unit 22 and output from the vertical transfer unit 22, and transfers the charge in a second direction orthogonal to the first direction, for example, the horizontal direction.
The output amplifier 28 amplifies the charge transferred from the horizontal transfer unit 27 and output from the horizontal transfer unit 27 and outputs the amplified charge.

  The photoelectric conversion in the photoelectric conversion element 23 of the photoelectric conversion unit 19 accumulates an amount of charge corresponding to the intensity of light, and the read pulse TG extracts the charge to the vertical transfer unit 22 via the read gate unit 24, The signals are sequentially transferred in the direction of the horizontal transfer unit 27 by a vertical transfer pulse VDP including four-phase pulses φV1, φV2, φV3, and φV4. The horizontal transfer unit 27 transfers charges in the horizontal direction by a horizontal transfer pulse HDP composed of two-phase pulses φH1 and φH2 every time a pixel for one line enters, and converts the signal into a signal voltage by an output amplifier 28 to output a signal. To do.

  In the present application, for simplicity, the transfer in the vertical transfer unit 22 may be simply referred to as “vertical transfer”, and the transfer in the horizontal transfer unit 27 may be simply referred to as “horizontal transfer”.

With the above-described vertical transfer pulse VDP, charges in the vertical transfer unit 22 are transferred in the direction of the horizontal transfer unit 27 (the direction of the arrow FW in the vertical transfer unit 22 in FIG. 2, hereinafter referred to as “forward direction”). In addition to this, it is also possible to transfer in a direction away from the horizontal transfer unit 27 (a direction opposite to the arrow FW in the vertical transfer unit 22 in FIG. 2, hereinafter referred to as “reverse direction”). When the vertical transfer pulse VDP is not applied, the vertical transfer unit 22 functions as an image holding unit, and charges taken out to the vertical transfer unit 22 through the read gate unit 24 are transferred to the vertical transfer unit 22. Thus, it can be held in the taken-out position without being moved in either the forward direction or the reverse direction.
Each transfer element 23 of the vertical transfer unit 22 also functions as an image addition unit that adds the charge held inside and the charge transferred from the corresponding photoelectric conversion element 21.

  FIG. 3 is an enlarged view showing four pixels arranged in the vertical direction in FIG. A symbol PX indicates one pixel. Each pixel includes a photoelectric conversion element 21, a read gate 25, and a transfer element 23. In the present embodiment, a four-phase vertical transfer pulse VDP is used, and there are four electrodes 26a to 26d per pixel. The electric charge photoelectrically converted by the photoelectric conversion element 21 is read out to the transfer element 23 via the read gate 25. As described below, by changing the voltage of the four-phase vertical transfer pulse VDP in a predetermined time series, the charge in the vertical transfer unit 22 is transferred in the forward direction and the reverse direction by the desired number of pixels. can do.

  The photoelectric conversion elements 21 of the photoelectric conversion unit 19 of the solid-state imaging element are covered with color filters of different colors, and the color filters are periodically arranged in a predetermined repeating pattern. This periodic arrangement may be a Bayer type as shown in FIG. The color filters are, for example, red (R), green (G), and blue (B) color filters. From the photoelectric conversion elements covered with the R, G, and B color filters, R, G, and B pixels are used. A signal is obtained. In FIG. 4, symbol H indicates the horizontal axis on the imaging screen, that is, the horizontal axis, 1, 2,... M along the horizontal axis indicate coordinate values on the horizontal axis, and symbol V indicates the imaging screen. The upper vertical axis, that is, the vertical axis is shown, and 1, 2,..., N along the vertical axis represent coordinate values on the vertical axis.

  In addition, instead of the Bayer array shown in FIG. 4, the honeycomb array shown in FIG. 5 may be used. The honeycomb array is obtained by rotating the Bayer array by 45 degrees, and in the honeycomb array, the imaging elements each include photoelectric conversion elements of pixels that are substantially regular octagons, and the vertical transfer unit 22 extends along the pixel columns ( The shape meanders between adjacent pixel columns.

Referring to FIG. 1 again, the R, G, B pixel signals obtained from the solid-state imaging device 2 are subjected to correlated double sampling processing (CDS) and automatic gain control amplification processing (AGC) in the analog signal processing unit 3. Is done.
A readout pulse TG, a vertical transfer pulse VDP, a horizontal transfer pulse HDP, and an overflow drain pulse OFD (to be described later) supplied to the solid-state imaging device 2 (collectively referred to as “a solid-state imaging device driving pulse”). The sampling pulse DSP for CDS supplied to the analog signal processing unit 3 is supplied by the timing generator 8. The generation timing of these pulses is controlled by the CPU 7. The AGC in the analog signal processing unit 3 is also performed by a control signal from the CPU 7.

  The output signal of the analog signal processing unit 3 is converted into a digital signal by the pixel signal A / D conversion unit 4 and then processed by the signal processing unit 5 to become a video signal that can be displayed on the display means 6. . The signal processing unit 5 performs image processing such as conversion from RGB signals to luminance color difference signals (YCbCr), white balance correction, γ (gamma) correction, color interpolation processing, and contour enhancement correction. The display means 6 constituted by an LCD or the like displays the video signal output from the signal processing unit 5. When shooting a still image, the photographer determines the composition while viewing the image of the subject on the display means 6.

  The camera shake sensor 9 is used as a motion detection unit that senses the movement of the imaging apparatus body, particularly the solid-state imaging device 2 in the first direction (vertical direction) when the photographer images a subject. Here, the motion detection means for sensing the movement in the first direction may be one that senses only the movement in the first direction and senses the movement in the other direction. It may be one that generates a signal corresponding to the component. The camera shake sensor 9 is, for example, an element that detects an angular velocity, and a displacement amount and a displacement direction due to vibration can be obtained by time-integrating the obtained angular velocity. The angular velocity signal detected by the camera shake sensor 9 is converted into a digital signal by the motion signal A / D converter 10 and then output to the CPU 7. The camera shake sensor 9 is not limited to an angular velocity sensor, and instead, an angular acceleration sensor, an acceleration sensor, a geomagnetic sensor, or the like can be used, and by performing appropriate signal processing according to each sensor signal, The same effect as when the angular velocity sensor is used can be obtained.

The CPU 7 performs overall control of the imaging system and functions as means for performing various calculations. Based on the video signal data from the signal processing unit 5, automatic exposure control (AE) / automatic white balance control (AWB) is performed, amplification gain setting for the analog signal processing unit 3, and solid-state imaging generated by the timing generator 8 Control of generation of driving pulses for the element 2 and generation of control signals for the lens driving unit 12 are performed.
The integrating unit 11 integrates video signals for one screen input from the signal processing unit 5. The integration result represents the brightness of the captured image. Based on this, the CPU 7 controls the generation of the overflow drain pulse OFD, thereby controlling the charge accumulation time.

  Further, the CPU 7 integrates the angular velocity signal converted into the digital signal by the motion signal A / D conversion unit 10 to calculate the angle and direction of the shake generated in the imaging apparatus, and further, the amount of camera shake based on the calculated value. It also has a function of converting to the number of pixels of image blur.

  Further, as will be described in detail later, the waveform of the vertical transfer pulse VDP for moving charges in the vertical transfer unit 22 (see FIG. 2) and the generation timing of the vertical transfer pulse VDP are determined. A control signal is output to the timing generator 8 based on the determined waveform and generation timing of the vertical transfer pulse VDP, and the timing generator 8 generates a driving pulse signal for the solid-state imaging device 2 based on the control signal. . As described above, the timing generator 8 that has received the control signal calculated by the CPU 7 functions as a vertical transfer signal generating means for moving charges in the vertical transfer unit 22 described later.

  Further, the CPU 7 and the timing generator 8 control the timing of the readout pulse for transfer (reading) from the photoelectric conversion element to the transfer element when motion is detected by the camera shake sensor 9, and during a predetermined frame period. In addition, control for reading the image a plurality of times is also performed.

  FIG. 6 is a functional block diagram showing components for performing camera shake correction in the present embodiment. The light image incident through the lens 1 is photoelectrically converted by the charge accumulating means 31, and charges are accumulated.

In this embodiment, the charge storage unit 31 is configured by the photoelectric conversion unit 19 in FIG. The motion detection means 32 is a means for detecting the motion of the imaging apparatus, and is constituted by the hand movement sensor 9 of FIG. The synthesizing unit 33 includes first to Nth frames in a predetermined frame period corresponding to the optical image formed on the solid-state imaging device 2 according to the amount of motion (BL) detected by the motion detecting unit 32. (N is an integer of 2 or more) image signals are read, and the read first to (N-1) th image signals are shifted by the number of pixels corresponding to the detected amount of motion. Synthesize.
Note that such a synthesis process may be performed only when the amount of motion per predetermined time detected by the motion detection means 32 is greater than or equal to a predetermined value.

As shown in the figure, the synthesizing unit 33 includes an image shift unit 34, an image holding unit 35, an image adding unit 36, and a control unit 37.
In the first embodiment, the image shift unit 34, the image holding unit 35, and the image addition unit 36 are all configured by the vertical transfer unit 22.

  As will be described in detail below, the synthesizing unit 33 forms a part of the same frame period in accordance with the amount of motion per predetermined time (BL) detected by the motion detecting unit 32, and the first to the next Charges are accumulated in the photoelectric conversion element 21 during the Nth (N is an integer of 2 or more) image generation period, and the charges accumulated in the photoelectric conversion element 21 during the first image generation period are converted into the first image generation period. After the elapse of the period, data is read from the photoelectric conversion element 21 to the corresponding transfer element 23 and transferred between the transfer elements 23 by the first movement amount related to the movement amount BL. When N is larger than 2, the synthesizing unit 33 further converts the electric charge accumulated in the photoelectric conversion element 21 during the n-th (n is an integer from 2 to (N−1)) photoelectric conversion element 21. The corresponding readout from 21, the read charge is added to the transferred charge by the (n−1) th movement amount related to the amount of movement, and the charge obtained as a result of the addition is added to the movement The process of transferring between the transfer elements 23 by the n-th movement amount related to the amount of n is sequentially executed from 2 to (N−1). The synthesizing unit 33 further reads out the electric charge accumulated in the photoelectric conversion element 21 during the Nth image generation period from the photoelectric conversion element 21 to the corresponding transfer element 23, and uses the read electric charge as the amount of movement. The added (N-1) th movement amount is added to the transferred charge, and the charge obtained as a result of the addition is output from the solid-state imaging device as a combined image signal.

  For example, the first to Nth image generation periods have substantially the same length, and the amount of motion BL per predetermined time is a solid-state imaging device per total time of the first to Nth image generation periods. 2 is the amount of motion represented by the number of pixels of 2. The amount related to the amount of motion is the amount of motion per predetermined time divided by N, and the amount of motion of the pixels of the solid-state imaging device 2 It is the amount of motion represented by a number (BL / N).

  The solid-state imaging device 2 according to the present embodiment is, for example, an interline type CCD imaging device, and is of an “all pixel readout” method in which all pixel signals are read out in order in a line in the horizontal direction by line sequential scanning. is there.

  FIG. 7 is a schematic diagram showing the operation of the synthesizing means 33 over time. An image signal representing an image generated in each of the image generation periods that are part of a predetermined frame period and divided into N divided charge accumulation periods (determined by the brightness of the subject) is used for image stabilization according to the present embodiment. Therefore, an operation of combining after performing the shift is shown. The first image generation period GC1 indicates a period during which charge is accumulated in the photoelectric conversion unit 19 in order to obtain the first image (first image). Hereinafter, the second image generation period GC2 to Nth The image generation period GCN has the same meaning. After the image signal representing the first image formed by the charge accumulation in the first image generation period GC1 is read (RD1) to the vertical transfer unit 22, the photoelectric conversion unit obtains the image signal representing the second image During charge accumulation (GC2) at 19, the image signal representing the first image is shifted by the image shift means 34 by the number of pixels corresponding to 1 / N of the amount of movement (SH1), and the second image is read out. Until the pixel position after the shift is held by the image holding means 35 (HL1).

  Thereafter, the image signal held at the pixel position by the image holding means 35 is added by the image addition means 36 with the image signal representing the second image (AD1), and the added image is obtained by the image shift means 34. The pixel is shifted by the number of pixels corresponding to 1 / N of the amount of movement (SH2), and is held at the pixel position after the shift by the image holding unit 35 until the third image is read (HL2). Thereafter, the image signal held at the pixel position by the image holding means 35 is added by the image adding means 36 to the image signal representing the third image (AD2). Thereafter, the same operation is repeated until the image signals representing the Nth image are added to obtain a synthesized image signal.

  FIGS. 8 and 9 are timing charts of potential change of the vertical transfer pulse VDP, as a specific method for transferring the charge in the vertical transfer unit 22 of the solid-state imaging device 2 in the forward direction shown in FIGS. Will be described. FIG. 8 is a timing chart of the four-phase pulse φVi (i = 1, 2, 3, 4) of the vertical transfer pulse VDP of each electrode, with the horizontal axis representing time and the vertical axis representing four electrodes 26a to 26d. The applied potentials are shown side by side. FIG. 9 is a diagram showing the state of charge transfer when the vertical transfer pulse VDP shown in FIG. 8 is applied to each electrode. In FIG. 9, the horizontal axis indicates the positions of the electrodes 26 a to 26 d of the transfer element 23 of the vertical transfer unit 22, and the vertical axis indicates the time-dependent change of the potential energy φi (= −eφVi) and the charge QC distribution of each electrode. (Charge distributions at different times t = a1 to a9) are shown.

  Next, the operation will be described. In FIG. 8, since the potential φV2 of the electrode 26b and the potential φV3 of the electrode 26c are + V and the potential φV1 of the electrode 26a and the potential φV4 of the electrode 26d are 0 V at time a1, as shown in FIG. A potential well is formed in the electrode 26c, and charge QC is accumulated. Next, when a bias voltage + V is applied to the electrode 26d at time a2, the charge QC that has been in the electrodes 26b and 26c is diffused from the electrode 26b to the electrode 26d. Next, when the electrode 26b is unbiased at time a3, the charge QC converges from the electrode 26c to the electrode 26d, and as a result, from time a1 to a3, the charge QC is one electrode in the forward direction (rightward in FIG. 9). It ’s moved by the minute.

  By applying a bias voltage + V to the electrode 26a at time a4 and making the electrode 26c non-biased at time a5, the charge QC further moves forward by one electrode. By applying a bias voltage + V to the electrode 26b at time a6 and making the electrode 26d non-biased at time a7, the charge QC further moves forward by one electrode. At time a8, a bias voltage + V is applied to the electrode 26c, and at time a9, the electrode 26a is made non-biased, so that the state becomes the same as at time a1. That is, from time a1 to time a9, the charge QC for one pixel is moved in the forward direction.

  On the other hand, a case where the charge QC in the vertical transfer unit is transferred in the reverse direction will be described with reference to FIGS. 8 used for the description of the case where the charge QC is transferred in the forward direction corresponds to FIG. 10, and FIG. 9 corresponds to FIG. Times a1 to a9 correspond to b1 to b9, respectively.

  Next, the operation will be described. In FIG. 10, at time b1, the bias voltage + V is applied to the electrode 26b and the electrode 26c, and the electrode 26a and the electrode 26d are 0 V with no bias. Therefore, as shown in FIG. 11, potential wells are formed in the electrode 26b and the electrode 26c. The charge QC is accumulated. Next, when a bias voltage + V is applied to the electrode 26a at time b2, the charge QC that was in the electrodes 26b and 26c diffuses from the electrode 26a to the electrode 26c. Next, when the electrode 26c is unbiased at time b3, the charge QC converges from the electrode 26a to the electrode 26b, and as a result, the charge QC is one electrode in the reverse direction (leftward in FIG. 11) from time b1 to b3. It ’s moved by the minute.

  By applying a bias voltage + V to the electrode 26d at time b4 and making the electrode 26b non-biased at time b5, the charge QC further moves in the reverse direction by one electrode. By applying a bias voltage + V to the electrode 26c at time b6 and making the electrode 26a unbiased at time b7, the charge QC further moves in the reverse direction by one electrode. At time b8, a bias voltage + V is applied to the electrode 26b, and at time b9, the electrode 26d is made non-biased, so that the state is the same as at time b1. That is, the charge for one pixel is moved in the reverse direction from time b1 to time b9.

  In the present embodiment, as described above, the convergence step is performed after the charge diffusion step and the charge transfer is realized by repeating this. However, the charge transfer is performed even if the diffusion and convergence steps are performed simultaneously. Is possible. For example, similar charge transfer can be achieved by simultaneously performing diffusion at time a2 and convergence at time a3 in FIG. 9, that is, by simultaneously applying a bias voltage to the electrode 26d and making the electrode 26b non-biased. Alternatively, even if convergence at time a3 and diffusion at time a4 are performed at the same time in FIG. 9, charge transfer is possible.

  Furthermore, in this embodiment, charges are transferred by a four-phase vertical transfer pulse VDP. However, the transfer is not limited to four phases, and charges can be transferred by the same method as long as there are three or more phases. Similarly, the horizontal transfer pulse HDP is a two-phase one, but is not limited thereto.

  In this embodiment, a CCD image sensor is used as the image sensor. However, the present invention is not limited to this. For example, a CMOS (Complementary Metal Oxide Semiconductor) type image sensor having means equivalent to the above-described combining means 33 is provided. Also good.

  Next, the concept of “camera shake” to be corrected in the present invention will be described. FIG. 12A is a diagram illustrating a relationship between a camera shake direction and a captured image when the camera operator shakes a camera shake in the vertical (pitching) direction when imaging a subject. Here, the subject is a circular light source LS, and a captured image when captured without causing camera shake is illustrated in FIG. An image LSb holding a round subject shape is captured.

  The photographer HY first holds the imaging device PG so that the optical axis coincides with the ground GL and the line HL, and presses the shutter button SB. Camera shake occurs at this time, and when the optical axis OX of the imaging device PG moves upward ((+) direction) due to camera shake, the captured image La is represented by the symbol TR as shown in FIG. As shown in FIG. 12D, when tailing occurs in the downward direction and the optical axis OX of the imaging device PG moves in the downward direction ((−) direction), the captured image Lc has As indicated by the symbol TRc, tailing occurs in the upward direction.

  An object of the present invention is to realize a high-quality captured image in which the tails TRa and TRb of the subject images LSa and LSc reproduced in FIGS. 12B and 12D are reduced. In most cases, the image quality degradation due to camera shake is caused by rotation of the imaging device or the like caused by the operation of the photographer HY pressing the shutter button SB. Since this rotation direction substantially coincides with the pressing direction of the shutter button SB, a practically sufficient correction is possible by one-dimensional correction corresponding to the pressing direction. In the present embodiment, the direction in which charges are transferred in the vertical transfer unit 22 (vertical direction on the imaging screen) corresponds to the direction in which camera shake is likely to occur (when the imaging screen is provided vertically). In the vertical transfer unit 22, the direction in which charges are transferred (the vertical direction on the imaging screen) is made substantially parallel to the direction in which camera shake is likely to occur.

FIGS. 13A to 13I are timing charts showing the operation of the imaging apparatus in the present embodiment. FIG. 13A shows the shutter signal BP, and the transition from the high level to the low level occurs at the time Ton when the pressing of the shutter is detected.
FIG. 13B shows a vertical synchronization signal VD, which is generated once in a predetermined cycle, for example, a vertical cycle in principle. The overflow drain pulse in FIG. 13C is applied from the timing generator 8 to the overflow drain electrode of the solid-state imaging device 2 in synchronization with the horizontal synchronizing signal, and is subjected to photoelectric conversion to adjust the charge accumulation period in AE control. The electronic shutter function is realized by discharging the electric charge accumulated in the element 21 onto the substrate in the solid-state imaging element 2. Of the overflow drain pulse OFD in FIG. 13C, pulses other than the first pulse OFD1 (generated simultaneously with or immediately after the rising of the vertical synchronization signal VD (first after the rising)) Sometimes called a shutter pulse.

  FIG. 13D shows a period PD in which photoelectric conversion and charge accumulation are performed in the photoelectric conversion element 21. This period PD (indicated by symbols SS1, SS2, etc., respectively) starts after the generation of the overflow drain pulse OFD in each frame period and continues until the next rising edge of the vertical synchronization signal VD. In other words, the time point at which the generation of the column of the overflow drain pulse OFD is finished is determined so that the charge accumulation period has a given length. FIG. 13E shows a read pulse TG, which controls the read gate 25 in order to read (transfer) the charge accumulated in the photoelectric conversion element 21 to the corresponding transfer element 23 of the vertical transfer unit 22. FIGS. 13F to 13H show vertical transfer pulses VDP (φV1 to φV4), and FIG. 13I shows horizontal transfer pulses HDP (φH1, φH2).

  Prior to pressing the shutter button SB, a charge accumulation period SSi (i = 1, 2,...) Defined as one of the exposure conditions is set based on the brightness of the subject and imaging is performed. For example, the charge accumulated in the charge accumulation period SS1 in the frame (first frame FP (1)) starting in the first vertical blanking period VB (1) in which a certain vertical synchronization signal VD is generated is the first The frame is read out to the vertical transfer unit 22 by the read pulse TG10 generated in the first vertical blanking period VB (2) of the frame following the frame FP (1) (second frame FP (2)). The data is transferred toward the horizontal transfer unit 27 in the vertical transfer unit 22 in the period 100b in the second frame FP (2). Transfer to the horizontal transfer unit 27 is simultaneously performed in the plurality of vertical transfer units 22. The pixel signals transferred at the same time are a plurality of pixels aligned in the horizontal direction on the imaging surface, that is, signal charges read from the plurality of photoelectric conversion elements 21 respectively constituting the plurality of pixels on the same line, Are transferred by the vertical transfer unit 22. In other words, the signal charges transferred from the respective photoelectric conversion element arrays 20 of the photoelectric conversion unit 19 to the vertical transfer unit 22 are sequentially transferred to the horizontal transfer unit 27 line by line. Then, in the period 101b in the same second frame period FP (2), the charges transferred from the vertical transfer unit 22 to the horizontal transfer unit 27 are transferred to the output amplifier 28 in the horizontal transfer unit 27 and output. Amplified by the amplifier 28 and output.

  If the shutter button SB is pressed at a certain time Ton in the charge accumulation period SS2 in the second frame period FP (2) and the shutter signal BP transitions from a high level to a low level, the second frame FP (2 The charge accumulated in the charge accumulation period SS2 is applied with the overflow drain pulse OFD1 to the solid-state imaging device 2 at the beginning of the next vertical blanking period VB (3) (substantially simultaneously with the rise of the vertical synchronization signal VD). As a result, the sheet is discharged onto the substrate without being transferred to the vertical transfer unit 22. Note that the overflow drain pulse OFD1 is not necessarily required when an electronic shutter pulse (overflow drain pulse OFD other than OFD1) is generated by AE control within the vertical blanking period. If the generation of the readout pulse (such as TG10 in VB (2)) is prohibited, the reading of the signal charge to the vertical transfer unit 22 is prohibited, so that it is accumulated when the electronic shutter pulse is generated by the AE control. This is because the charged charges are discharged onto the substrate.

  The frame period FP (3) next to the time point Ton when the pressing of the shutter button SB is detected is a period in which charge accumulation is performed to obtain a video signal for an image to be recorded as a still image. In this frame period FP (3), the charge accumulation period is divided into a plurality of image generation periods (see FIG. 7), and a plurality of images are obtained by reading out charges to the vertical transfer unit 22 for each image generation period.

  The plurality of images are finally combined into one image and output from the solid-state imaging device. Until then, among the images constituting the plurality of images, the images read during the first image generation period ( An image read out during the second image generation period (second image) after temporarily holding in the vertical transfer unit 22 (which may be referred to as “first image” hereinafter). Until the first image is read by the vertical transfer unit 22, the first image is moved by the image position obtained by dividing the amount of motion detected by the motion detection means by the number of the plurality of images, and then the second image is moved to the image. The operation of adding the image read during the generation period (second image) and temporarily holding it is repeated until the image read last (“fourth image” in the example shown) is added.

  In order to read out four images, the number of divisions of the charge accumulation period is set to 4, and the image generation periods (charge accumulation periods after division) after dividing one charge accumulation period into equal parts are respectively S1, S2, Let S3 and S4. In this case, the charge accumulated in the image generation period S1 is read pulse TG1, the charge accumulated in the image generation period S2 is read pulse TG2, the charge accumulated in the image generation period S3 is read pulse TG3, and the charge accumulated in the image generation period S4 is read. The data is read to the vertical transfer unit 22 by the read pulse TG4. 102a is a charge read by the read pulse TG1, 102b is a charge read by the read pulse TG2, and 102c is a charge read by the read pulse TG3. This is a period in which only the number of pixels is transferred.

Further, when the number of divisions in the charge accumulation period is 4, the vertical number is proportional to 1/4 of the amount of motion detected by the motion detecting means, that is, the number of pixels corresponding to 1/4 of the number of pixels corresponding to the amount of motion. Charge is transferred in the transfer unit 22. For example, when there are three charge transfer periods represented by reference numerals 102a, 102b, and 102c as shown in the figure, the charge transfer is performed in proportion to ¼ of the total camera shake amount in each period. 3/4 of the total amount of camera shake is corrected as the total period. However, ¼ of the total camera shake amount remains as it is, and as a result, camera shake is reduced to ¼.
Generally speaking, if the number of divisions in the charge accumulation period is N, camera shake is reduced to 1 / N.

Normally, in order to reduce the image blur due to camera shake of the recorded image to 1/4 without using the camera shake correction as in the present embodiment, the exposure aperture is opened by two stages (the F value is set to a large value by two stages). In other words, this can be achieved by quadrupling the amount of light taken into the solid-state imaging device and reducing the charge accumulation period (shutter speed) to 1/4, but this embodiment does not open the aperture stop by two stages. The camera shake can be reduced to 1/4. In other words, when the number of divisions in the charge accumulation period is N, the same correction effect as when the exposure aperture is increased by log ₂ N stages by reading out N images and synthesizing N images. I can expect.

  As described above, when the number of divisions of the charge accumulation period is fixed to 4, when correcting an image with a different amount of camera shake, the number of vertical transfer pulses VDP applied to the solid-state image sensor during each image generation period is determined according to the amount of camera shake. Increase or decrease. For example, when there is no camera shake (below a predetermined threshold value), as shown in FIG. 13 (f), it is not necessary to perform vertical transfer every time the image generation periods S1, S2, and S3 are finished. Is not applied and there is no period for vertical transfer. In this case, the four read images are not moved, but are added in the read state to finally obtain one image.

  When there is no camera shake, the charge accumulation period is not divided, that is, the readout pulses TG1 to TG4 in FIG. 13E are not generated, and the vertical blanking period VB is the same as when the shutter-on is not detected. A read pulse (same as TG10) may be generated during (4).

  When camera shake is detected by the motion detection means 32, as shown in FIGS. 13G and 13H, charges are transferred within the vertical transfer unit 22 by a distance (number of pixels) corresponding to the amount of camera shake. In this case, the amount of charge transfer is proportional to the amount of camera shake. When the amount of shake is large, the amount of charge transfer during each image generation period is long. When the amount of blur is small, the amount of charge transfer is also short. For example, it is assumed that the amount of camera shake is larger in FIG. 13H than in FIG. That is, the vertical transfer periods 104a, 104b, and 104c shown in FIG. 13 (h) are longer than the vertical transfer periods 102a, 102b, and 102c shown in FIG. 13 (g).

  Reference numeral 103 denotes an extra stored in the horizontal transfer unit 27 after being pushed out to the horizontal transfer unit 27 by transferring the charges in the vertical transfer unit 22 toward the horizontal transfer unit 27 in the vertical transfer periods 102a, 102b, and 102c. This is a period for sweeping away unnecessary charges, that is, to make the accumulated charges in the horizontal transfer section 27 zero (prior to the next vertical transfer).

  In FIG. 13, by applying a read pulse TG to the read gate 25 during the vertical blanking period VB (3), all charges accumulated in the charge accumulation period SS2 are read out to the vertical transfer unit 22 and then transferred horizontally. It is possible to eliminate the overflow drain pulse OFD1 by performing high-speed transfer to the unit 27 and further sweeping away by high-speed transfer by the horizontal transfer unit 27.

  Further, if the first line for the charge transfer in the horizontal transfer period 101c in the frame period FP (4) is discarded, the horizontal transfer period 103 in the blanking period VB (4) becomes unnecessary. Can do.

  Further, in this embodiment, the charge accumulation period is adjusted by the electronic shutter pulse. However, by providing a mechanical shutter, the charge accumulation can be performed only while the shutter is open.

  In the example shown in FIG. 13, the remaining period obtained by subtracting the period during which the overflow drain pulse OFD is continuously applied to the solid-state imaging device 2 from one frame period is defined as a charge accumulation period, and the charge accumulation period is divided into four. ing. In other words, the charge accumulation period determined according to the brightness of the subject (a period having the same length as the charge accumulation period when not divided) is divided into four image generation periods.

  Instead of doing this, the overflow drain pulse OFD may not be applied, and charge accumulation may be performed over the maximum period that can be used for charge accumulation in one frame period. Further, the generation of the vertical synchronization signal VD is suppressed once, the length of the frame period is doubled, and the maximum period that can be used for charge accumulation in the frame period having the length of these two cases is divided. Thus, charge accumulation may be performed during a plurality of image generation periods. FIG. 14 (d) shows the maximum that can be used for charge accumulation in the frame period having twice the length of the frame period by suppressing the generation of the vertical synchronization signal VD once and doubling the length of the frame period. This is a case where the period is divided into four image generation periods. In this way, a relatively large amount of charge can be accumulated when the illuminance of the subject is low, and a relatively high level signal can be obtained.

  FIG. 15 is a flowchart illustrating an image stabilization operation according to the first embodiment. First, it is monitored whether or not the shutter button SB has been pressed (step St1). If the shutter button SB has been pressed, the charge storage period calculated by the AE process (that is, the charge storage period determined based on the brightness of the subject). ), The division number N of the charge accumulation period, that is, the number N of images to be read is determined (step St2).

  Next, it is determined whether or not the division number N is 1 (step St3). If N = 1, the image is read out (charges in the photoelectric conversion element array 20 are transferred to the vertical transfer unit 22 (step St4), and all charges in the vertical transfer unit 22 are transferred to the horizontal transfer unit 27 in the next frame period. (Step St14) and the process is terminated.

  If N = 1 is not satisfied in step St3, the process proceeds to step St5, and the camera shake amount BL is detected by the camera shake sensor 9. In the present embodiment, an angular velocity sensor is used as the camera shake sensor 9, and the angular velocity of vibration generated in the imaging device when the photographer presses the shutter button SB is detected, and the CPU 1 determines the zoom magnification of the lens 1. From the integral value of the position and the angular velocity, the blur amount is converted into the corresponding pixel number BL to obtain the blur amount represented by the number of pixels.

  In the above example, the camera shake amount is detected after the determination of the number of charge accumulation time divisions (St2). However, after the camera shake amount detection (St5), the camera shake amount is taken into consideration. It is also possible to determine the number of divisions of the charge accumulation time (St2) (for example, increase the number of divisions as the amount of camera shake increases). In that case, step St5 is moved before step St2 on the flowchart.

  For the purpose of reading from the first image among the divided N images, n = 1 is given (step St6). Next, the nth image is read out. In the initial state, since n = 1 in step St6, the first image is read (step St7).

  Next, the image is shifted by the image position (number of pixels) obtained by dividing the number of pixels BL representing the amount of motion by the number of divisions of the charge accumulation period, that is, the number N of image generation periods (step St8), and then the vertical transfer unit 22 (Step St9). The image held at this time is referred to as an image (A). Next, the (n + 1) -th image is read out, and the read-out image is set as an image (B) (step St10).

  In the initial state (that is, when step St10 is executed for the first time), the (n + 1) th image refers to the second image. Then, the image (A) and the image (B) are added (step St11). This addition is applied to each transfer element 23 of the vertical transfer unit 22 holding the charge corresponding to the image (A), that is, the image (A), from the respective photoelectric conversion elements 21 of the photoelectric conversion element array 20. B), that is, by reading the charge corresponding to the image (B).

  Next, it is determined whether or not n = N−1 (step St12). If n = N−1 is not satisfied, 1 is added to n (step St13), the process returns to step St8, and the processes from step St8 to step St12 are repeated. This repetition is continued until n = N−1 in Step St12. If n = N−1, all charges in the vertical transfer unit 22 are transferred to the horizontal transfer unit 27 in the next frame period (step St14), and the process ends.

  In the above example, the method of converting the blur amount from the integrated value of the position of the lens 1 and the angular velocity into the number of pixels BL may use a table with the integrated value of the position of the lens 1 and the angular velocity as a variable, or may be given in advance. It may be calculated from the previously calculated formula. For example, in FIG. 13 where the number of image generation periods is 4, when the number of blur pixels BL is 16, charges are transferred by BL / 4 = 4 pixels in the vertical transfer periods 102a, 102b, and 102c, respectively.

  FIGS. 16A to 16I show details of a captured image when camera shake occurs during imaging and a subject image on the imaging surface of the solid-state imaging device 2. In this example, the camera shake occurs in the upward direction at the moment the photographer presses the shutter button SB of the image pickup device (the image pickup device moves upward, so that the picked-up image gradually moves downward), and the image generation period This shows a case where the number is 4 and the pixel conversion number of camera shake in the vertical direction is 16. FIG. 16A shows a captured image when there is no camera shake. FIG. 16B is a captured image in the case where the charge accumulation period is not divided and the camera shake correction is not performed, as in the case indicated by the symbols SS1 and SS2 in FIG. 13, and the camera shake of 16 pixels in the vertical direction. Has occurred.

  FIGS. 16C to 16F are images obtained by dividing the image of FIG. 16B into four image generation periods, and charge accumulation is performed in the image generation periods S1, S2, S3, and S4, respectively. And shows a captured image obtained when it is assumed that all pixels are read for each image generation period (in practice, an image for each image generation period is not read out from the solid-state imaging device, After being added by the vertical adder, it is read outside). The image generation periods S1 to S4 have the same length, and the brightness of the captured images in FIGS. 16C to 16F is ¼ of the brightness of the captured image in FIG.

  In each of FIGS. 16C to 16F, the difference between the dotted line and the solid line represents a shake during each image generation period.

  The deviation BL of the subject image for 16 pixels generated in the image of FIG. 16B can be equally distributed to the deviation of 4 pixels (DBL = BL / 4) in FIGS. 16C to 16F. And If it is assumed that the angular velocity of camera shake is constant within the image generation period of the images in FIGS. 16C to 16F, such a treatment can be approximately performed. The first embodiment sequentially corrects and adds (synthesizes) a shift of 4 pixels existing between images obtained one after the other. That is, the image of FIG. Is shifted by the shift DBL and synthesized with the image of FIG. 16D to form a first composite image, and this first composite image is further shifted by the shift DBL of 4 pixels, and FIG. Is combined with the image of FIG. 16 to form a second composite image, and this second composite image is further shifted by a shift DBL of four pixels and combined with the image of FIG. As a result, a final composite image is formed. As a result, an image in which the subject image is positioned substantially at the same position as the captured image of FIG. 16F (the subject image of FIG. )-(E) subject image is shifted and combined (superimposed) same image) And outputs as a final specific captured image from the solid-state image pickup element 2.

  FIGS. 16G to 16J show images when a subject is projected on the imaging surface of the solid-state imaging device 2 through a lens. The corresponding images in FIGS. There is an upside down relationship.

  For the subject images in FIGS. 16 (g) to (j), the tail portion due to camera shake is not drawn for convenience, but actually the tail portion due to camera shake similar to the subject image in FIGS. 16 (c) to (f). Exists. If the number of image generation periods is 4, camera shake that is 1/4 of the total camera shake amount occurs for each image in each image generation period.

  Note that if the number of image generation periods is 8, camera shake of 1/8 of the total amount of motion occurs in the images in each image generation period.

  As shown in FIG. 12 (a), when the orientation of the imaging device changes upward due to camera shake, the subject on the imaging surface upside down in FIGS. 16 (g), (h), (i), and (j). The image moves upward. In FIG. 16 (g), (h), (i), and (j), “*” marks indicated by reference signs d1, d2, d3, and d4 indicate one pixel at the same location on the subject (person's face). The details of the charge transfer in the vertical transfer unit 22 will be described below with reference to FIGS. 17 and 18 while paying attention to this pixel.

  FIG. 17 is a diagram in which the period in which the charge accumulation period is divided in FIG. 13 is enlarged in the time axis direction, and codes TSa to TSk are assigned to each operation in time series. The state of charge transfer in the vertical transfer unit 22 at the time when these operations TSa to TSk are performed will be described with reference to FIGS.

  FIGS. 18A to 18H show the charges in the vertical transfer unit 22 when the subject moves upward on the imaging surface of the solid-state imaging device 2, as shown in FIGS. 16G to 16J. It shows a method for correcting camera shake by transfer. Similarly to FIGS. 2 and 3, in FIGS. 18A to 18J, each photoelectric conversion element array 20 includes a plurality of photoelectric conversion elements 21, and the vertical transfer unit 22 includes a plurality of transfer elements 23. Each transfer element 23 corresponds to a pixel and includes four transfer electrodes, but this point is not shown in FIGS. 18A to 18H.

  Each of the read gates 25 of the read gate unit 24 is provided between each photoelectric conversion element 21 and the corresponding transfer element 23.

  First, as shown in FIG. 18A, in the first image generation period S1 (TSa), the photoelectric conversion is performed by the photoelectric conversion element 21 (1) corresponding to the target pixel (d1 in FIG. 16G). Accumulate d1.

  Next, as shown in FIG. 18B, the charge d1 accumulated in the photoelectric conversion element 21 (1) during the image generation period S1 by the readout pulse TG1 is transferred to the corresponding transfer element 23 (1) of the vertical transfer unit 22. Read (TSb).

  Immediately after the end of this charge reading, as shown in FIG. 18C, the charge d2 accumulation (TSc) in the photoelectric conversion element 21 (2) of the target pixel is started in the second image generation period S2, and this charge During the accumulation (TSc), the charge read out to the vertical transfer unit 22 by the operation TSb is moved upward by four pixels from the transfer element 23 (1), that is, moved to the transfer element 23 (2) (TSd).

  Next, as shown in FIG. 18D, the charge d2 of the photoelectric conversion element 21 (2) (charge d2 accumulated in the photoelectric conversion element 21 (2) during the image generation period S2) is vertically transferred by the readout pulse TG2. The data is read to the corresponding transfer element 23 (2) of the unit 22 and added to the charge d1 moved from the transfer element 23 (1) to the transfer element 23 (2) in the vertical transfer unit 22 (TSe).

  Immediately after the completion of the charge reading and addition, as shown in FIG. 18E, in the third image generation period S3, the accumulation (TSf) of the charge d3 in the photoelectric conversion element 21 (3) of the target pixel is started. During this charge accumulation (TSf), the charge (d1 + d2) added in the transfer element 23 (2) of the vertical transfer unit 22 in the operation Tse is upward by four pixels from the transfer element 23 (2), that is, the transfer element 23. Move to (3) (TSg).

  Next, as shown in FIG. 18F, the charge d3 of the photoelectric conversion element 21 (3) (charge d3 accumulated in the photoelectric conversion element 21 (3) during the image generation period S3) is vertically transferred by the readout pulse TG3. The data is read to the corresponding transfer element 23 (3) of the unit 22 and added to the charge (d1 + d2) transferred from the transfer element 23 (2) to the transfer element 23 (3) in the vertical transfer unit 22 (TSh).

  Immediately after the completion of the reading and addition of the charge, as shown in FIG. 18G, the accumulation (TSi) of the charge d4 in the photoelectric conversion element 21 (4) of the target pixel is started in the fourth image generation period S4. During the charge accumulation (TSi), the charge (d1 + d2 + d3) added in the transfer element 23 (3) of the vertical transfer unit 22 in the operation TSh is upward by four pixels from the transfer element 23 (3), that is, the transfer element. 23 (4) (TSj).

  Next, as shown in FIG. 18 (h), the charge d4 of the photoelectric conversion element 21 (4) (charge d4 accumulated in the photoelectric conversion element 21 (4) during the image generation period S4) is vertically transferred by the readout pulse TG4. The data is read to the corresponding transfer element 23 (4) of the unit 22 and added to the charge (d1 + d2 + d3) transferred from the transfer element 23 (3) to the transfer element 23 (4) in the vertical transfer unit 22 (TSk).

Thereafter, as indicated by the arrow Ym, the data of all pixels is transferred to the horizontal transfer unit 27 (100c in FIGS. 13 and 17), and at the same time, the signal charge is transferred to the output amplifier unit 28 in the horizontal transfer unit 27. (101c in FIGS. 13 and 17).
With the above operation, image data with reduced image blur can be obtained.

  Note that, as described above, the photoelectric conversion elements 23 of the photoelectric conversion unit 19 of the solid-state imaging device 2 are covered with different color filters, and the color filters are periodically arranged in a predetermined repeating pattern. The amount of vertical transfer (number of pixels) corresponding to each image generation period is desirably an integer multiple of the number of pixels corresponding to the period of the repetitive pattern of the color filter.

  With respect to this point, a case where the periodic arrangement of the color filters is a Bayer type will be described with reference to FIG. In the case of the Bayer type arrangement, as shown in FIG. 4, green G is arranged in a checkered pattern, and red R and blue B are arranged in a line sequence in the other portions. For example, when attention is paid to the third column (H = 3) in FIG. 4, the arrangement of the pixels is R, G, R, G,..., G from the top to the bottom. Both are arranged every other pixel.

  Therefore, if the amount of movement when moving charges in the vertical transfer unit 22 in each image generation period is an odd number of pixels in the forward or backward direction, for example, G pixels are added to the moved R pixels. Or the R pixel is added to the moved G pixel, which is not appropriate. In order to add signal charges of pixels of the same color, it is necessary to make the amount of charge movement equal to even pixels. In FIG. 18, when charges are moved in the vertical transfer unit in the operations TSd, TSg, and TSj, they are moved by 4 pixels, and the same color pixels are added together. For example, when the camera shake conversion pixel number BL in the first image generation period is 5 pixels, the charge transfer amount needs to be either 4 pixels or 6 pixels. As a specific method, one method is to multiply the maximum natural number not exceeding (BL + 1) / 2 by 2 as the amount of charge transfer. For example, when BL = 5, the amount of charge transfer is 6. Further, when BL = 4.9, the charge transfer amount is 4.

  In addition, when moving charges in the vertical transfer unit 22 in each image generation period, if the charges are moved in the forward direction, that is, in the direction of the horizontal transfer unit 27, the charge is accumulated in the vertical transfer unit 22 in the lowest line. Without being discharged into the horizontal transfer unit 27. On the other hand, when the charge is moved in the reverse direction in the vertical transfer unit 22, there is no horizontal transfer unit 27 above the uppermost line due to the structure of the solid-state imaging device 2, so that the charge discharge groove (drain) as shown in FIG. 40 is provided. As a result, even when the charge is moved upward in the vertical transfer unit 22, the charge can be discharged into the charge discharge groove 40 without overflowing the uppermost line.

  Further, instead of providing the charge discharge groove 40 in the upper part of the solid-state image pickup device 2 as shown in FIG. It is also possible to prevent the electric charge from overflowing at the uppermost line when moving to. A schematic diagram in this case is shown in FIG. The shaded area is the light shielding area 42. Although the size of the light-shielding region 42 depends on the amount of charge movement required at the time of camera shake correction, (the amount of charge discharged to the charge discharge groove 40 in FIG. 19) is stored when it is assumed that photoelectric conversion is performed in the light-shielding region. It is necessary to secure an area having a size proportional to the shake amount to be corrected by the camera shake correction so as to be approximately equal to the (charge amount).

  When N images are read out, image blur in an image that is finally combined into one by performing camera shake correction in the present embodiment is 1 / compared to a case in which camera shake correction according to the present embodiment is not performed. N can be reduced. For this reason, in this embodiment, the number of divisions of the charge accumulation period (the number of images to be read) is set to 4. However, if more accurate camera shake correction is desired, the number of divisions may be increased. In an apparatus with a simple configuration, the number of divisions may be reduced.

  In the present embodiment, a zoom mechanism is used in which the lens 1 can be driven in the optical axis direction, but a similar effect can be obtained even with a single focus lens. Although the vertical transfer pulse VDP has been described for the case of four phases, the same operation is possible for three phases, and of course the same operation is possible for five phases, six phases, and the like.

  As described above, according to the first embodiment, it is possible to provide an imaging device that reduces and corrects camera shake only by charge movement inside the solid-state imaging device without requiring complicated mechanism parts such as an actuator. In addition, even if camera shake is corrected by dividing the charge accumulation period, frame memory or the like is not necessary, and high-precision camera shake correction is possible even when there are structural restrictions such as cameras for portable devices. Is possible.

  In the first embodiment, it has been described that the amount of movement can be detected immediately after the detection Ton of pressing the shutter button SB, that is, between the time Ton and the next rise of the VD signal. Hereinafter, a case will be described in which the detection of the amount of motion does not end before the rise of the first VD signal after Ton.

  FIGS. 21A to 21I show time charts in that case. This time chart corresponds to FIG. FIGS. 21 (a) to (e), (h), and (i) show the same as FIGS. 13 (a) to (e), (h), and (i), respectively. In FIG. 21A, the transition (Ton) of the signal BP from the high level to the low level due to the detection of pressing the shutter button occurs in the first frame period FP (1). FIG. 21F shows the output GY of the camera shake sensor (angular velocity sensor) 9. In the example shown in the figure, the output GY rises substantially simultaneously with the fall of the signal BP and thereafter maintains a substantially constant value. FIG. 21G shows the operation timing in the CPU 7, that is, based on the period SGb in which the CPU 7 integrates the output GY of the camera shake sensor 9 (converted by the motion signal A / D converter 10) and the integrated value. A period SKb for calculating the amount of movement is shown.

  In the illustrated example, the amount of movement is detected in the frame period FP (2) next to the frame period FP (1) in which the pressing of the shutter button SB is detected, and the charge is detected in the next frame period FP (3). The accumulation time SS3 is divided into four to perform camera shake correction imaging. This point will be described in more detail below.

  The camera shake sensor 9 outputs a signal in accordance with the movement of the imaging apparatus. For example, when the shutter button SB is pressed and the imaging apparatus moves, output is started in accordance with the movement. The start of the output of the camera shake sensor 9 does not always coincide with the detection timing Ton of pressing the shutter button, and may be a little earlier than that, or may be a little later. After the start of the output of the camera shake sensor, the movement of the imaging device usually does not stop immediately, and continues to move for several frames.

  In the illustrated example, the output GY of the camera shake sensor 9 starts from the point in time when the generation of the column of the overflow drain pulse OFD ends after the vertical synchronization signal VD rises in the frame period FP (2) following the detection Ton of the shutter button press. Integration is started. This integration is continued for a time SGb having the same length as the charge accumulation time SS2 determined based on the brightness of the subject. The charge accumulation time SS2 of the second frame FP (2) and the charge accumulation time SS3 (= S1 + S2 + S3 + S4) of the third frame are both the imaging results of the first frame FP (1) or the previous frame. It is determined based on (the result of integration by the integration means 11) and is the same as each other.

  When the integration is completed, the amount of movement is calculated by the CPU 7 based on the integration result. The period for this calculation is indicated by the symbol SKb.

  In the third frame FP (3), the amount of motion calculated in the second frame FP (2) is used to perform image stabilization imaging by dividing the charge accumulation time SS3 into four parts. Therefore, if the speed of movement of the imaging device does not change between the second frame FP (2) and the third frame FP (3), the amount of movement and the charge accumulation time obtained by integration in the integration time SGb. The amount of motion in SS3 is exactly the same, and correction can be performed accurately. The movement of the imaging device due to camera shake usually continues over several frames, and the difference between the speed of movement in the second frame and the speed of movement in the third frame is small.

  As shown in FIG. 21, the camera shake sensor is detected from the time when the generation of the column of the overflow drain pulse OFD ends after the vertical synchronizing signal VD rises in the frame period FP (2) next to the detection Ton of the shutter button press. Instead of starting the integration of the output of 9, the integration of the output of the camera shake sensor 9 may be started simultaneously with the rise of the vertical synchronization signal VD in the frame period FP (2) following the detection Ton of pressing the shutter button. . Also, the integration of the output of the camera shake sensor 9 is started simultaneously with the detection Ton of the shutter button press, and the imaging result (the subject brightness, that is, the integration means 11 of the integration unit 11) is started before the frame where the shutter button press is detected. Based on the output, the integration time SGb and the charge accumulation time for camera shake correction imaging may be determined.

  In this case, the integration time SGb and the charge accumulation time for camera shake correction imaging are determined based on the imaging result (brightness of the subject, that is, the output of the integration unit 11) in the frame before the frame where the shutter button press is detected. It is also good.

  Instead, the time is shorter than the charge accumulation time for camera shake correction imaging determined based on the imaging result (brightness of the object, that is, the output of the integrating unit 11) in the frame before the frame where the shutter button press is detected. It is good also as integrating during this time. In this case, the amount of movement obtained by integration multiplied by the ratio (SS / SG) of the charge accumulation time to the accumulation time SG is obtained as the amount of movement per charge accumulation time (estimated as the amount of movement). )be able to.

  At the same time as the detection Ton of pressing the shutter button, the integration of the output of the camera shake sensor is started, and if the integration is performed over a time shorter than the charge accumulation time, the frame (FP (1)) in which the pressing of the shutter button is detected. ) In the next frame (FP (2)).

Embodiment 2. FIG.
FIG. 22 is a diagram illustrating the configuration of the imaging apparatus according to the second embodiment of the present invention. The image pickup apparatus according to the second embodiment is generally the same as the image pickup apparatus according to the first embodiment shown in FIG. 1, but includes a frame memory 48 (hereinafter may be simply referred to as “memory”), an adder 44, and a selection unit. The difference is that a device 46 is added. 1 receives the pixel signal from the frame memory 48 instead of the pixel signal A / D conversion unit 4, and the CPU 7 also controls the adder 44 and the selector 46.

  In addition, the functional block diagram showing the components for performing camera shake correction in the second embodiment is the same as that in FIG. 6 of the first embodiment, but the synthesizing means 33 is different from the first embodiment. That is, the image shift means 34 is constituted by the CPU 7 for designating the address of the memory 48, the image holding means 35 is constituted by the frame memory 48, and the image addition means 36 is constituted by the adder 44. The control means 37 is composed of the CPU 7 as in the first embodiment.

  The synthesizing unit 33 forms a part of the same frame period according to the amount of motion per predetermined time (BL) detected by the motion detecting unit 32, and the first through Nth (N is 2 or more) The charge is accumulated in the photoelectric conversion element 21 during the image generation period, and the charge accumulated during the first image generation period is output from the solid-state imaging element 2 after the first image generation period. Then, the pixel data corresponding to the charges output from the solid-state imaging device 2 is supplied to the frame memory 48 via the analog signal processing unit 3, the pixel signal A / D conversion unit 4, and the selector 46. Are stored in an address shifted by a first amount related to the amount of movement (an address shifted by the above amount from an address corresponding to a pixel of the image sensor). When N is larger than 2, the synthesizing unit 33 further stores the charge accumulated during the nth image generation period (n is 2 to (N-1)) after the elapse of the nth image generation period. , Output from the solid-state imaging device 2 and supplied to the adder 44 via the analog signal processing unit 3 and the pixel signal A / D conversion unit 4, and at the same time, relate to the amount of motion in the frame memory 48. The address shifted by the nth amount (the pixel data corresponding to the charge from each pixel of the image sensor accumulated during the first image generation period matches the address shifted by the first amount). ) Is read out and supplied to the adder 44. The pixel data input by the adder 44 are added together, and the addition result is shifted by the nth amount in the frame memory 48. Stored at the address Processing n is executed in order from 2 to (N-1). The synthesizing unit 33 further causes the charge accumulated during the Nth image generation period to be output from the solid-state imaging device 2 after the Nth image generation period, and outputs the analog signal processing unit 3 and the pixel signal A. The address is supplied to the adder 44 via the / D conversion unit 4 and at the same time, the address in the frame memory 48 shifted by the Nth amount related to the amount of motion (during the first image generation period) The pixel data stored in the accumulated pixel data corresponding to the charges from each pixel of the image sensor is coincident with the address shifted by the first amount) and supplied to the adder 44 for addition. The pixel data input by the unit 44 are added together, and the addition result is stored in the address shifted by the Nth amount in the frame memory 48, and then output as synthesized pixel data to be a signal processing unit. Supplied to.

  Note that pixel data after adding pixel data corresponding to charges accumulated during the Nth image generation period may be supplied to the signal processing unit 5 without being temporarily stored in the frame memory 48. .

  Further, pixel data corresponding to the electric charge accumulated during the first image generation period is supplied to the frame memory 48 without going through the adder 44. Instead, it is supplied to the adder 44. At the same time, the pixel data stored at the address shifted by the first amount related to the amount of motion in the frame memory 48 is read out, supplied to the adder 44, and input by the adder 44. The added pixel data may be added, and the addition result may be stored in the address shifted by the first amount in the frame memory 48 and supplied to the signal processing unit 5. In this case, the same result can be obtained if zero is stored in the address before reading from the frame memory 48. In this case, the selector 46 becomes unnecessary, and the processing for controlling the selector 46 (operation of the CPU 7) becomes unnecessary, and the processing when the charge accumulated during the first image generation period is read out n (where n is 2 to (N−1)) can be the same as the processing when reading out the electric charge accumulated during the image generation period.

For example, the first to Nth image generation periods have substantially the same length,
The amount of motion per predetermined time (BL) is the amount of motion represented by the number of pixels of the image sensor per total time of the first to Nth image generation periods,
The first amount related to the amount of movement is
BL · (N-1) / N
And
The nth amount related to the amount of movement is
BL · (N−n) / N
It is good to be.

Further, the frame memory 48 has an address (storage position) for the pixel of the composite image,
The shifted pixel data may be stored in an address for each pixel in each composite image.

  FIG. 23 is a schematic diagram showing the operation of the synthesizing unit 33 of the second embodiment over time. An image signal representing an image generated in each of the image generation periods that are part of a predetermined frame period and divided into N divided charge accumulation periods (determined by the brightness of the subject) is used for image stabilization according to the present embodiment. For this reason, an operation of synthesizing after shifting is shown. The first image generation period GC1 indicates a period in which charge is accumulated in the photoelectric conversion unit 19 in order to obtain the first image. Hereinafter, the second image generation period GC2 to the Nth image generation period GCN are Has the same meaning.

  After the charge accumulation (GC1) of the first image is completed, the charge signal representing the first image is read out to the vertical transfer unit 22 and transferred at high speed in the direction of the horizontal transfer unit 27, and at the same time in the horizontal transfer unit 27 Thus, high-speed transfer of charges is performed in the horizontal direction, and the charge signal is converted into a signal voltage by the output amplifier 28 and output as an image signal (VE1). The image signal is supplied to the frame memory 48 (image holding means 35) after passing through the analog processing unit 3, the pixel signal A / D conversion unit 4 and the selector 46, and BL · (N− 1) It is written to an address (storage position) shifted by an image position corresponding to / N (SD1). This address (storage position) is called a composite image storage address (storage position).

  Thereafter, it is held in the frame memory 48 (image holding means 35) until the charge accumulation in the second image generation period (GC2) ends (ST1).

  In the second image, after the charge accumulation (GC2) is completed, the charge signal representing the second image is read out to the vertical transfer unit 22 and transferred at high speed in the direction of the horizontal transfer unit 27. Then, high-speed transfer of charges is performed in the horizontal direction, and the charge signal is converted into a signal voltage by the output amplifier 28 and output as an image signal (VE2). The image signal is supplied to the adder 44 via the analog processing unit 3 and the pixel signal A / D conversion unit 4. At the same time, the pixel in the frame memory 48 corresponding to the charge of each pixel accumulated during the first image generation period (GC1) shifted by the second amount related to the amount of motion. For the data, pixel data stored in the address shifted by the first amount with respect to the address corresponding to the pixel of the image sensor, that is, the address matching the composite image storage position) is read and supplied to the adder 44, The pixel data input by the adder 44 are added together (AD2). This means that the pixel data corresponding to the charge accumulated in each pixel (photoelectric conversion element 21) during the second image generation period is shifted by the number of pixels corresponding to (BL · (N−2) / N). The pixel data corresponding to the charge accumulated in each pixel (photoelectric conversion element 21) during the first image generation period is the pixel data shifted by the number of pixels corresponding to (BL · (N−1) / N). Means to be added.

The pixel data obtained as a result of the addition is written into the composite image storage address (for each pixel) (SD2), and the frame memory 48 (image holding means 35) until the charge accumulation in the third image generation period is completed. (ST2).
Similarly, pixel data corresponding to charges accumulated in each pixel (photoelectric conversion element 21) during the n-th image generation period is shifted by the number of pixels corresponding to (BL · (N−n) / N). This is added to the pixel data stored in the corresponding composite image storage address and stored in the same composite image storage address.By repeating such an operation up to the Nth image in sequence, the composite image at the composite image storage address is stored. Pixel data of each pixel is obtained.

  The details of the captured image when camera shake occurs during imaging and the image written to the frame memory 48 are shown in FIGS. In this example, the camera shake occurs in the upward direction at the moment the photographer presses the shutter button SB of the image pickup device (the image pickup device moves upward, so that the picked-up image gradually moves downward), and the image generation period This shows a case where the number is 4 and the pixel-converted number of camera shake in the vertical direction is 4. It is shown that there is no upside down on the imaging surface. FIG. 24A shows a captured image when there is no camera shake. FIG. 24B shows a captured image when the charge accumulation period is not divided and no camera shake correction is performed, and camera shake of four pixels occurs in the vertical direction.

  FIGS. 24C to 24F are images read by dividing the image of FIG. 24B into four image generation periods. The four image generation periods S1 to S4 have the same length, and the brightness of the picked-up images in FIGS. 24C to 24F is 1/4 of the brightness of the picked-up image in FIG. .

  In each of FIGS. 24C to 24F, a dotted line indicates a part of the subject image at the start of each image generation period, and a solid line indicates a subject image at the end of each image generation period. The solid line difference represents the blur during each image generation period. In each of FIGS. 24C to 24E, the chain line represents the difference SFT1 to SFT3 between the positions of the subject image generated in the last image generation period and the subject image generated in the last image generation period. .

The subject image shift BL generated in the image of FIG. 24B can be equally distributed to the shift of one pixel (DBL = BL / 4) in FIGS. 24C to 24D. And If it is assumed that the angular velocity of camera shake is constant within the image generation period of the images of FIGS. 24C to 24F, such a treatment can be approximately performed.

  In the second embodiment, the position difference existing between the first to third images shown in FIGS. 24C to 24E and the last image shown in FIG. 24F is corrected. In other words, the image is written at the same address in the frame memory 48, that is, the address (corresponding to the pixel in the image sensor) obtained by shifting the image of FIG. The first written image is formed by writing to the address shifted from the address to be addressed), and the image of FIG. 24D is shifted by the difference SFT2 of the position of two pixels in the frame memory 48. Is added (synthesized) with the first written image written in the memory, and the addition result is written to the same address in the frame memory 48 (the same address where the first written image was written). 2 written images And adding (synthesizing) the image of FIG. 24E with the second written image written at the address shifted by the difference SFT3 of the position of one pixel in the frame memory 48, and the addition result To the same address of the frame memory 48 (the same address as the second written image was written) to form a third written image, and the image of FIG. Is added (synthesized) with the third written image written at the corresponding address of the frame, and the addition result is added to the same address of the frame memory 48 (the same address as the second written image is written). A fourth written image, that is, a final written image is formed by writing, and an image in which the subject image is located at substantially the same position as the captured image of FIG. 24 (f) (FIG. 24 (f)). Subject To and forms in FIG. 24 (c) ~ frame memory 48 in the object image were synthesized on shifted respectively (superimposed) the same image as) as the final writing image (e).

  For the subject images in FIGS. 24G to 24J, the tail portion due to camera shake is not drawn for the sake of convenience, but actually the tail portion due to camera shake similar to the subject image in FIGS. Exists. If the number of image generation periods is 4, camera shake that is ¼ of the total camera shake amount occurs for each image in each image generation period.

  In order to simplify the illustration, it is assumed that the number of pixels in the vertical direction in the image on the imaging screen of the imaging device 2 is 13, and one image is obtained by reading out 13 horizontal lines. In the illustrated example, line numbers are given in order from the bottom.

  Of the four images read out at this time, the first image (FIG. 24C) is written in the frame memory 48 as shown in FIG. The pixel data of the line is discarded, and the pixel data of the fourth to thirteenth lines of the image sensor 2 are shifted by the number of lines corresponding to the position difference SFT1 (= 3 pixels) in the frame memory 48, that is, Write to addresses corresponding to the first to tenth lines. That is, writing is performed at an address shifted by 3 lines, in other words, an address shifted by 3 pixels in the vertical direction. In the 11th to 13th lines of the frame memory 48, a dummy image signal (for example, a signal having a value of zero) is written. Then, the state where the writing is completed is held in the frame memory 48 as it is.

  Subsequently, for the second image (FIG. 24D), as shown in FIG. 24H, the pixel data of the first and second lines of the image sensor 2 are discarded when writing to the frame memory 48. The pixel data of the third to thirteenth lines of the image sensor 2 are shifted by the number of lines corresponding to the position difference SFT2 (= 2 pixels) in the frame memory 48, that is, the first to eleventh lines (ie, The data is added to the data held at the address shifted by two lines, in other words, the address shifted by two pixels in the vertical direction), and then written to the same first to eleventh lines. In the 12th to 13th lines of the frame memory 48, a dummy image signal (for example, a signal having a value of zero) is written. Then, the state where the writing is completed is held in the frame memory 48 as it is.

For the third image (FIG. 24E), as shown in FIG. 24I, when writing to the frame memory 48, the pixel data of the first line of the image sensor 2 is discarded, and the image sensor 2 The pixel data of the second to thirteenth lines are shifted by the number of lines corresponding to the position difference SFT3 (= 1 pixel) in the frame memory 48, that is, the first to twelfth lines (that is, one line are shifted). After adding each of the data held at the address (in other words, the address shifted by one pixel in the vertical direction), the same first to twelfth lines are written. In the 13th line of the frame memory 48, a dummy image signal (for example, a signal having a value of zero) is written. Then, the state where the writing is completed is held in the frame memory 48 as it is.
As for the fourth image (FIG. 24F), as shown in FIG. 24J, the pixel data of the first to thirteenth lines of the image sensor 2 are stored in the frame memory 48 when writing to the frame memory 48. Are added to the corresponding addresses, that is, the data held in the first to thirteenth lines, respectively, and then written to the same first to thirteenth lines.

  As a result, a composite image is formed in the frame memory 48. The first to tenth lines in the frame memory 48 store the sum of the pixel data obtained from each of the four captured images. The eleventh line is the sum of the pixel data and dummy data of the second to fourth captured images, the twelfth line is the sum of the pixel data and dummy data of the third to fourth captured images, The thirteenth line stores the sum of the pixel data and dummy data of the fourth captured image.

  When supplying pixel data from the frame memory 48 to the signal processing unit, only pixel data of the first to tenth lines are supplied (the remaining image region RR excluding the region DM in which the dummy image signal was written at the time of writing). Or all the pixel data of the first to thirteenth lines may be supplied.

  FIG. 24 shows a case where the camera shakes upward at the moment when the photographer presses the shutter button SB of the imaging device. The image position is shifted upward by (N−n) / N and then written to the frame memory 48 or added by the adder 44 to finally obtain an image with reduced hand shake of 1 / N. be able to.

  FIG. 25 is a flowchart illustrating an image stabilization operation according to the second embodiment. The operations of steps St1, St2, and St5 to St7 are the same as those in FIG. If N = 1 in step St3, the process proceeds to step St21, where the nth image (n = 1) is read, and then in step St22, the pixel data of the nth image (the signal charge output from the solid-state imaging device 2). Corresponding data) is written in the frame memory 48. (The image read from the solid-state imaging device 2 passes through the analog signal processing unit 3 and the pixel signal A / D conversion unit 4 and does not pass through the adder 44 (the selector 46 is supplied to the frame memory 48). , Written).

After step St7, the process proceeds to step St23, where the nth image (n = 1) is read, and the pixel data of the read image is written into the frame memory 48. (The read image passes through the analog signal processing unit 3 and the pixel signal A / D conversion unit 4 and does not pass through the adder 44 (via the selector 46, supplied to the frame memory 48 and written). At this time, writing is performed at an address shifted by the number of pixels corresponding to the motion amount (N-1) / N.
Thereafter, the process proceeds to step St24, 1 is added to n in step St24, and then the process proceeds to step St25 to read the nth image. The read image is supplied to the adder 44 through the analog signal processing unit 3 and the pixel signal A / D conversion unit 4. In step St26, data is read from the shifted address in the frame memory 48 and added with the data from the A / D converter 44. Then, the pixel data obtained as a result of the addition is written to the same address (the same address from which the data was read in step St26) in the frame memory 48 (St27).

Thereafter, the process proceeds to step St28, where it is determined whether n = N. If n = N is not satisfied in step St28, the process proceeds to step St24.
The processes in steps St24, St25, St26, St27, and St28 are repeated until it is determined that n = N in step St28.
If it is determined in step St28 that n = N, the process ends.
At this time, the frame memory 48 stores pixel data corresponding to a composite image of the first to Nth images.

  As described above, according to the second embodiment, camera shake can be reduced and corrected by controlling the method of writing data to the frame memory 48 without requiring complicated mechanism parts such as actuators. An imaging device is obtained.

It is a block diagram which shows the structure of the imaging device of Embodiment 1 of this invention. It is a figure which shows the structure of the solid-state image sensor used by this invention. It is a figure which expands and shows 4 pixels located in a line in the vertical direction in FIG. It is a figure which shows a Bayer type color filter arrangement | sequence. It is a figure which shows a honeycomb type color filter arrangement | sequence. 3 is a functional block diagram illustrating components for performing camera shake correction in the first embodiment. FIG. 6 is a diagram illustrating an operation of a solid-state imaging device that constitutes a combining unit according to Embodiment 1. FIG. It is a timing chart which shows the vertical transfer pulse VDP applied when transferring an electric charge in the forward direction in a vertical transfer part. It is a figure which shows the electric potential distribution of the electrode of the transfer element of a solid-state image sensor in the case of transferring an electric charge in a forward direction within a vertical transfer part. It is a timing chart which shows the vertical transfer pulse VDP applied when an electric charge is transferred in the reverse direction in a vertical transfer part. It is a figure which shows the electric potential distribution of the electrode of the transfer element of a solid-state image sensor in the case of transferring an electric charge in the reverse direction within a vertical transfer part. (A)-(d) is a figure which shows the example of the captured image containing the direction and angle of camera shake of an imaging device, and camera shake. (A)-(i) is a timing chart which shows operation | movement of the imaging device in Embodiment 1. FIG. (A)-(i) is a timing chart which shows operation | movement of the imaging device in the modification of Embodiment 1. FIG. 6 is a flowchart illustrating an operation of correcting camera shake in the first embodiment. FIG. 3 is a diagram illustrating displacement of a captured image and a subject image on an imaging surface of a solid-state imaging device according to the first embodiment. (A)-(f) is the figure which expanded the period which divided | segmented the electric charge accumulation period in FIG. 13 in the time-axis direction. 6 is a diagram illustrating a charge transfer and charge addition method in a vertical transfer unit according to the first embodiment. FIG. It is a figure which shows the solid-state image sensor provided with the electric charge discharge groove. It is a figure which shows the solid-state image sensor provided with the light-shielding area | region. (Z)-(f) is a timing chart which shows operation | movement of the imaging device in the modification of Embodiment 1. FIG. It is a block diagram which shows the structure of the imaging device of Embodiment 2 of this invention. FIG. 10 is a diagram illustrating an operation of a solid-state imaging device that constitutes a combining unit in the second embodiment. 6 is a diagram illustrating a relationship between a captured image, a subject image on an imaging surface of a solid-state imaging device, and an image in a frame memory according to Embodiment 2. FIG. 10 is a flowchart illustrating an operation of correcting camera shake in the second embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Lens, 2 Solid-state image sensor, 3 Analog signal processing part, 4 Pixel signal A / D conversion part, 5 Signal processing part, 6 Display means, 7 CPU, 8 Timing generator, 9 Shake sensor, 10 Motion signal A / D conversion Unit, 11 integrating unit, 12 lens driving unit, 21 photoelectric conversion element (PD), 22 vertical transfer unit, 24 readout gate unit, 27 horizontal transfer unit, 28 output amplifier, 31 charge storage unit, 32 motion detection unit, 33 synthesis Means, 34 image shift means, 35 image holding means, 36 image addition means, 37 control means, 40 charge discharge groove, 42 light shielding area, 48 frame memory, PX pixel.

Claims (4)

A photoelectric conversion unit that is arranged at a plurality of pixel positions aligned in the first direction on the imaging screen and includes photoelectric conversion elements that perform photoelectric conversion by receiving light from a subject, and pixels aligned in the first direction A solid-state imaging device having a transfer unit provided corresponding to each of the photoelectric conversion elements at the position, and having a plurality of transfer elements that transfer charges stored in the corresponding photoelectric conversion elements and transfer charges between them When,
Movement detecting means for detecting movement of the solid-state imaging device in the first direction;
First to Nth (N is an integer of 2 or more) in a predetermined frame period corresponding to an optical image formed on the solid-state imaging device according to the amount of motion detected by the motion detection means. Combining means for reading out the first image signal and shifting the read-out first to (N-1) th image signals by the number of pixels corresponding to the detected amount of motion. Have
The synthesis means includes
In accordance with the amount of motion per predetermined time detected by the motion detection means, electric charge is accumulated in the photoelectric conversion element in the first to Nth image generation periods that follow each other,
The charge accumulated in the photoelectric conversion element during the first image generation period is read from the photoelectric conversion element to the corresponding transfer element, and the transfer element is transferred between the transfer elements by a first movement amount related to the amount of movement. Forward with
When N is larger than 2, the electric charge accumulated in the photoelectric conversion element during the n-th (n is an integer of 2 to (N−1)) image corresponds to the charge from the photoelectric conversion element. Read to the transfer element, add the read charge to the charge transferred by the (n-1) th movement amount related to the amount of movement, and add the charge obtained as a result of the addition The process of transferring between the transfer elements by the nth movement amount related to the amount of movement is sequentially executed from 2 to (N−1),
The charge accumulated in the photoelectric conversion element during the N-th image generation period is read from the photoelectric conversion element to the corresponding transfer element, and the read charge is related to the amount of motion. N-1) is added to the transferred charge, and the resulting charge is output from the solid-state imaging device as a combined image signal,
The synthesizing unit determines the number of pixels to shift the image signal in each frame period to a value corresponding to the amount of motion detected by the motion detecting unit in a frame before the frame period. An imaging device. The imaging apparatus according to claim 1, wherein the first to (N−1) th moving amount is proportional to the amount of movement per predetermined time. The photoelectric conversion unit includes photoelectric conversion elements covered with color filters of different colors, and the color filters are periodically arranged in a predetermined repeating pattern,
3. The image pickup apparatus according to claim 1, wherein the amount of charge transfer in the transfer unit is an integer multiple of the number of pixels corresponding to a cycle of a color filter repetition pattern.   The imaging apparatus according to claim 1, wherein a total time of the first to Nth image generation periods is determined based on brightness of the subject.

Images