JP2012161029A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correct an image blur and suppress image distortion in an imaging device equipped with an image sensor with an asynchronous accumulation period of a signal charge.SOLUTION: An imaging device comprises: an image sensor in which a charge accumulation period is different for each predetermined area on an imaging surface; imaging control means that controls the drive of the image sensor to capture a subject in plural times of continuous exposure times; blur detection means that compares the plural captured images to detect a blur between the images; blur correction means that mutually shifts the plural images so as to correct the detected blur to combine them to one image; velocity vector calculation means that calculates a velocity vector of the subject based on the detected blur; and distortion correction means that corrects image distortion due to asynchrony of the charge accumulation period between the predetermined areas based on the velocity vector.

Description

  The present invention relates to an imaging device equipped with an imaging device in which signal charge accumulation periods are not simultaneous.

  In recent years, imaging devices equipped with solid-state imaging elements have become widespread. In imaging of a subject by an imaging device, suppressing image blur is one of important issues. Various imaging devices configured to suppress image blur have been proposed.

  For example, Patent Document 1 describes an imaging apparatus that images a subject in a time-division manner with a plurality of continuous short exposure times. Each image picked up by the image pickup apparatus described in Patent Document 1 has a small amount of blurring of the subject because the exposure time per time is short, but the overall brightness is low due to insufficient exposure. For this reason, the imaging apparatus repeatedly performs short-time exposure until the amount of blurring of the apparatus main body from the start of exposure reaches an allowable amount. A plurality of images taken in this way are combined into one image to ensure proper luminance. The composition is performed in a state where the image blur amount of each image is detected and corrected.

Japanese Patent Laid-Open No. 2003-32540

  The imaging apparatus described in Patent Document 1 assumes a CCD (Charge Coupled Device) image sensor as a solid-state imaging device. However, image blur also occurs when imaging is performed using a solid-state imaging device other than the CCD image sensor. As a typical solid-state image sensor other than a CCD image sensor, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor that is inexpensive, has low power consumption, and is excellent in mass productivity is known.

  When the image blur correction invention described in Patent Document 1 is applied to an image pickup apparatus equipped with a CMOS image sensor, it is considered that image blur is corrected in the same manner as an image pickup apparatus equipped with a CCD image sensor.

  However, since the CMOS image sensor starts accumulation of the photoelectrically converted signal again immediately after signal output, the signal charge accumulation period differs for each scanning line. For this reason, image distortion due to a shift in the charge accumulation period occurs. This type of image distortion becomes more prominent because the overlapping portion of each charge accumulation period becomes shorter as the exposure time is shorter. That is, when the image blur correction invention described in Patent Document 1 is applied to an image pickup apparatus equipped with a CMOS image sensor, there is a problem that the image is greatly distorted as a price for image blur correction.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to correct image blur and suppress image distortion in an imaging apparatus equipped with an imaging element in which signal charge accumulation periods are not simultaneous. That is.

  An imaging apparatus according to an embodiment of the present invention that solves the above-described problems includes an imaging element having a different charge accumulation period for each predetermined region of the imaging surface, and a plurality of continuous exposure times of the subject by driving the imaging element. The image pickup control means for picking up the image, the shake detection means for comparing the picked up images and detecting the mutual shake of the images, and the plural images are mutually shifted so as to correct the detected shake. Blur correction means for combining the images into one image, speed vector calculation means for calculating the velocity vector of the subject based on the detected blur, and non-simultaneousness of the charge accumulation period between the predetermined areas based on the velocity vector And distortion correcting means for correcting image distortion caused by the above.

  According to the present invention, the signal charge accumulation period is not synchronized even when multiple short-time exposures are performed for image blur correction using an image sensor in which the charge accumulation period is different for each predetermined region of the imaging surface. The distortion of the image due to the image can be satisfactorily suppressed.

  The distortion correction unit defines a charge accumulation period of a predetermined reference area on the imaging surface as a reference accumulation period, and other predetermined areas other than the reference area on the imaging surface that capture a correction target image to be subjected to distortion correction. The shift time between the charge storage period of the region and the reference storage period is defined as the storage shift time, and the moving distance of the correction target image from the start of the reference storage period to the start of the charge storage period of the other predetermined region When the movement direction is defined as the exposure deviation amount, the exposure deviation amount is calculated using the velocity vector and the accumulation deviation time for each other predetermined area, and the position of the correction target image is determined using the exposure deviation amount as a reference accumulation period. It is good also as a structure converted into the position estimated that it existed at the start time of.

  The imaging apparatus according to the present invention may be configured to include a shape changing unit that changes the shape of the image after distortion correction in accordance with a predetermined display screen area.

  As the imaging device, for example, a CMOS image sensor that is inexpensive, has low power consumption, and is excellent in mass productivity is assumed.

  The blur detection unit detects a common feature point from the plurality of images, and the blur correction unit shifts the plurality of images to each other so that the feature points overlap to be combined into one image. It may be configured.

  According to the imaging apparatus of the present invention, image blur is corrected and image distortion is suppressed even when the signal charge accumulation period of the imaging element is not simultaneous.

It is a block diagram which shows the structure of the imaging device of embodiment of this invention. It is a figure for demonstrating the scanning timing of the imaging surface of the CMOS image sensor mounted in the imaging device of embodiment of this invention. It is a figure for demonstrating the image blur detection by the image blur detection circuit mounted in the imaging device of embodiment of this invention. It is a figure for demonstrating correction | amendment of the image by the image blurring correction circuit and exposure shift correction circuit mounted in the imaging device of embodiment of this invention.

  Hereinafter, an imaging apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of the imaging apparatus 100 according to the present embodiment. In the present embodiment, the imaging device 100 is an extracted imaging device portion in an endoscope system, but is a digital still camera, a digital video camera, a PDA (Personal Digital Assistant), a PND (Portable Navigation Device). It is good also as an apparatus incorporated in the apparatus of another form which has imaging functions, such as PHS (Personal Handy-phone System), a portable game machine, and PC.

  As illustrated in FIG. 1, the imaging apparatus 100 includes a control circuit 1. The control circuit 1 comprehensively controls each block. In FIG. 1, the connection between the control circuit 1 and each block is omitted for the sake of simplicity.

  The control circuit 1 supplies clock pulses to the CMOS image sensor 2 according to a predetermined synchronization signal. The control circuit 1 controls the operation of the CMOS image sensor 2 so that the subject is imaged in a time-division manner with a plurality of continuous short exposure times for image blur correction. In each imaging, the exposure time is constant and the imaging interval is also constant. The CMOS image sensor 2 is driven according to a clock pulse and amplifies and outputs photoelectrically converted signal charges at each pixel.

FIGS. 2A and 2B are diagrams for explaining the scanning timing of the imaging surface of the CMOS image sensor 2. As shown in FIG. 2A, the imaging surface of the CMOS image sensor 2 has a plurality of scanning lines in which a plurality of pixels are arranged in a line (scanning lines L ₁ , L ₂ ,..., L _{n. −1} , L _n ). The scanning timing of the imaging surface is as shown in FIG. That is, in the CMOS image sensor 2, the signal charge accumulation period is shifted by the scanning time for each scanning line, and there is no simultaneity. The scanning line L ₁ is defined as the uppermost scanning line of the imaging surface, when the scanning line L _n is defined as the lowermost scanning line of the imaging surface, the start and end of the accumulation period of the signal charge in each scan line The scanning line on the upper side of the imaging surface is earlier. As shown in FIG. 2B, the CMOS image sensor 2 outputs several frames of pixel signals that are non-simultaneous in charge accumulation in each frame during a series of photographing operations.

  The pixel signal of each frame output from the CMOS image sensor 2 is input to the A / D conversion circuit 3 and converted into a digital signal, and then input to the pre-stage image processing circuit 4. Each pixel signal input to the pre-stage image processing circuit 4 is subjected to predetermined signal processing such as clamping, knee, γ correction, interpolation processing, AGC (Auto Gain Control), etc., and then buffered in the frame memory of the image memory 5 in units of frames. Is done. The pixel memory 5 is provided with an area for storing at least the number of frames obtained by the series of photographing operations.

  The image blur detection circuit 6 is a circuit that electronically detects an image blur amount. FIG. 3 is a diagram for explaining image blur detection by the image blur detection circuit 6. As shown in FIGS. 3A to 3C, the image blur detection circuit 6 includes a plurality of (three in the present embodiment) images captured during the series of imaging operations stored in the frame memory. A common feature point f is detected from the image by edge detection or the like. The image blur detection circuit 6 compares the appearance coordinates of the feature point f of each image and shifts the mutual display position of the subject (for example, the subject O located at the center of the shooting range in FIG. 3A), that is, image blur. Detect the amount.

  The image blur correction circuit 7 corrects image blur, and the exposure shift correction circuit 8 corrects image distortion. FIG. 4 is a diagram for explaining image correction by the image blur correction circuit 7 and the exposure deviation correction circuit 8.

  The image blur correction circuit 7 simply superimposes three images stored in the frame memory (see FIG. 4A). Since each image is an image obtained by short-time exposure, image blurring is small. However, the display position of the subject O does not match because there is image blur between the images. The image blur correction circuit 7 shifts the images to each other by the amount of the image blur detected by the image blur detection circuit 6 and combines them into a single image. In the composite image after the shift process, the feature point f overlaps one point (see FIG. 4B).

  As shown in FIG. 4B, the image size of the composite image is enlarged according to the shift amount. The image blur correction circuit 7 performs trimming to cut an area exceeding a prescribed image size (for example, the same size as the image stored in the frame memory) (see FIG. 4C). When the resolution of the monitor 200 is high, trimming may not be performed to display a large image.

  The image blur correction circuit 7 synthesizes a plurality of images obtained by short-time exposure as described above, thereby obtaining an image with less blur and with ensured luminance. However, distortion resulting from the non-synchronization of the signal charge accumulation period of the CMOS image sensor 2 remains in the composite image. A subject O ′ (dotted line) in FIGS. 3A to 3C shows a subject image picked up when the signal charge accumulation period and transfer of all the pixels are performed simultaneously as in a CCD image sensor. That is, in the example of FIG. 3, the actual captured image (that is, the subject O) itself with respect to the subject O ′ expresses image distortion.

The exposure deviation correction circuit 8 corrects distortion caused by the non-simultaneity of the signal charge accumulation period of the CMOS image sensor 2. Here, the signal charge accumulation period in each of the scanning lines L ₁ , L ₂ ,..., Ln is known. Therefore, when the signal charge accumulation period in the reference scanning line is defined as the reference accumulation period, the time difference between the reference accumulation period and the signal charge accumulation period in each scanning line (hereinafter referred to as “accumulation deviation time”) is also included. Known. Reference scan line in the present embodiment is, for example, a scanning line L _1.

  The exposure shift correction circuit 8 calculates the velocity vector of the subject O from the shift of the display position of the subject O in each image. The exposure shift correction circuit 8 estimates the moving distance and moving direction of the subject O from the start of the reference accumulation period to the start of the scanning line accumulation period in units of scanning lines based on the velocity vector and the accumulation deviation time. The exposure deviation correction circuit 8 performs coordinate conversion corresponding to the estimated movement distance and movement direction of the subject O in units of scanning lines. That is, the exposure shift correction circuit 8 moves the subject O on the scanning line to be corrected to a position (position of the subject O ′) that is estimated to exist at the start of the reference accumulation period. In the example of FIG. 3, since the subject O has moved to the right side of the screen, the subject image of the scanning line on the lower side of the imaging surface appears on the right side of the screen. The exposure deviation correction circuit 8 performs coordinate conversion so that the subject O appears at the position of the subject O ′ (see FIGS. 3A to 3C and FIG. 4D), and the signal charges of the CMOS image sensor 2 are converted. Correct distortion caused by non-simultaneous accumulation period.

  The image after the coordinate conversion by the exposure deviation correction circuit 8 is input to the subsequent image processing circuit 9. The shape of the image after coordinate transformation is not rectangular but distorted as shown in FIG. The rear-stage image processing circuit 9 trims the image after coordinate conversion in accordance with the display area (for example, a rectangle) of the captured image on the monitor 200 (see FIG. 4E). In FIG. 4 (e), for convenience of explanation, many regions removed by trimming are shown, but there are few regions actually removed. In addition, since the subject to be watched is displayed at the center of the screen, no problem occurs even if the peripheral area is removed. The post-stage image processing circuit 9 may perform rectangular masking instead of trimming. Trimming and masking may be omitted to simplify the process.

The trimmed image is input to the image output circuit 10 and input to NTSC (National Television
It is converted into a video signal that conforms to a predetermined standard such as System Committee) or PAL (Phase Alternating Line). By sequentially inputting the converted video signals to the monitor 200, an image of the subject is displayed on the display screen of the monitor 200.

  As described above, according to the present embodiment, in the imaging apparatus 100 equipped with the CMOS image sensor 2, even when performing a plurality of short-time exposures for image blur correction, the signal charge accumulation period is not synchronized. The resulting image distortion can be satisfactorily suppressed. Note that it is not necessary to add a specific hardware configuration for performing the image blur correction and distortion correction of the present embodiment, and the cost is not substantially increased.

  The above is the description of the embodiment of the present invention. The present invention is not limited to the above-described configuration, and various modifications can be made within the scope of the technical idea of the present invention. For example, the CMOS image sensor 2 may have a configuration in which the signal charge accumulation period is shifted not in units of scanning lines but in units of pixels. Further, the scanning order of the accumulated signal charges is not limited to the order shown in this embodiment.

  The image blur detection circuit 6 is not limited to an electronic type, and may be replaced with a module that detects an image blur amount using sensors such as an angular velocity sensor.

DESCRIPTION OF SYMBOLS 1 Control circuit 2 CMOS image sensor 3 A / D conversion circuit 4 Pre-stage image processing circuit 5 Image memory 6 Image blur detection circuit 7 Image blur correction circuit 8 Exposure shift correction circuit 9 Post-stage image processing circuit 10 Image output circuit 100 Imaging device

Claims (5)

An image sensor having a different charge accumulation period for each predetermined region of the imaging surface;
Imaging control means for driving and controlling the imaging element to image a subject with a plurality of continuous exposure times;
Blur detection means for comparing the plurality of captured images to detect mutual blur between the images;
A blur correction unit that shifts the plurality of images to each other and combines them into a single image so as to correct the detected blur;
Speed vector calculation means for calculating a speed vector of the subject based on the detected shake;
Distortion correcting means for correcting distortion of the image due to non-simultaneity of the charge accumulation period between the predetermined regions based on the velocity vector;
An imaging device comprising: The distortion correction means includes
The charge accumulation period of a predetermined reference area on the imaging surface is defined as a reference accumulation period, and charge accumulation in a predetermined area other than the reference area on the imaging surface that captures a correction target image to be subjected to distortion correction The deviation time between the period and the reference accumulation period is defined as the accumulation deviation time, and the movement distance and movement direction of the correction target image from the start of the reference accumulation period to the start of the charge accumulation period of the other predetermined area are exposed. When defined as a deviation amount,
For each of the other predetermined areas,
Calculate the amount of exposure deviation using the velocity vector and the accumulated deviation time,
The imaging apparatus according to claim 1, wherein the position of the correction target image is converted to a position estimated to exist at the start time of the reference accumulation period using the exposure deviation amount.   The imaging apparatus according to claim 1, further comprising a shape changing unit that changes the shape of the image after distortion correction in accordance with a predetermined display screen area.   The imaging device according to any one of claims 1 to 3, wherein the imaging device is a complementary metal oxide semiconductor (CMOS) image sensor. The blur detection means detects a common feature point from the plurality of images,
5. The blur correction unit according to claim 1, wherein the blur correction unit shifts the plurality of images to be combined with each other so that the feature points overlap each other. The imaging device described.

Images