JPWO2006043315A1 - IMAGING DEVICE AND PORTABLE DEVICE HAVING IMAGING DEVICE - Google Patents

Abstract

In an imaging device mounted on a cellular phone or a digital still camera, an imaging device having a shake correction function with a simple configuration is obtained. Along with the main imaging device, the number of pixels is smaller than that of the main imaging device, and the exposure time is reduced. A sub image sensor is provided, and the sub image sensor outputs a plurality of images while the main image sensor captures one image. A blur amount calculating unit that calculates a blur amount by correlating each successive image, and a blur component removing unit that removes a blur component based on the calculated blur amount are provided. .

Description

  The present invention relates to a camera shake amount or image blur amount detection method for detecting and correcting camera shake or image blur in a portable device equipped with an electronic camera such as a digital still camera or a cellular phone.

  In recent years, the progress of solid-state image sensors and signal processing circuits has enabled electronic camera functions that achieve image quality close to that of silver halide cameras to be realized in a small and inexpensive manner, realizing a digital still camera that fits in a pocket, and a mobile phone that is always carried around It is also used in the general public. When taking a picture with these small portable devices, the device is so small and light that it can be operated with one hand, which causes problems such as camera shake and image blur.

  In conventional single-lens reflex type silver halide cameras, lens shutter type silver halide cameras, and high-end digital still cameras, exposure is mainly performed using a mechanical shutter that limits the time during which light is incident on the image sensor or silver halide film to a very short time. The effect of camera shake or image blur is reduced by shortening the time. Moreover, in a simple silver halide camera such as a film with a lens and an inexpensive digital still camera, camera shake or image blur is prevented mainly by using a xenon flash that emits light only for a very short time. In addition, some digital still cameras and portable video camera recorders have a camera shake correction function that mechanically drives lenses and image sensors to cancel camera shake.

  The technology related to camera shake correction mainly includes a camera shake amount measuring unit that detects the direction and amount of camera shake (camera shake information) and a camera shake information detected by the camera shake amount measuring unit so as not to include a shake component in a captured image. Or correction means for removing the blur component from the photographed image. Several methods are known for correcting camera shake.

For example, the following Patent Document 1 describes means for detecting the presence or absence of camera shake using a distance measuring sensor. In Patent Document 1, an optical sensor for measuring the distance to the subject is provided independently of the main image sensor, and the presence or absence of camera shake is detected using the output of the distance measuring sensor. However, the optical sensor used by the technology as a camera shake sensor is originally intended for distance measurement, and is not a two-dimensional image sensor capable of measuring the direction and amount of camera shake, but a one-dimensional line sensor. The line sensor can detect the presence or absence of camera shake, but cannot obtain information on the direction and amount of camera shake necessary for camera shake correction. For this reason, a camera system using the technology can realize a camera shake correction mechanism that eliminates camera shake by only detecting a camera shake warning function that detects the presence of camera shake and notifies the user of the occurrence of the camera shake. I can't.

Patent Document 2 and Patent Document 3 describe camera shake measurement means for detecting the amount of camera shake using information on a plurality of continuous images (frames) output from the main image sensor. Since the camera shake amount measuring means of this system uses the video signal output from the main image sensor, it is impossible to detect camera shake at a frequency higher than the frame frequency of the main image sensor. Usually, the frame frequency of the image sensor is about 30 Hz, and the vibration component generated when the portable device is held with one hand includes a frequency component up to about 100 Hz. For this reason, it is impossible to detect camera shake of a portable device using this method. The above-mentioned patent documents are not intended for portable cameras, but are intended for applications where high-frequency vibration components do not occur, such as fixed cameras or surveillance cameras.

Patent Document 4 describes a camera shake amount measuring unit that detects an amount of camera shake using an angular velocity sensor. With this method, it is possible to measure the amount of camera shake in the direction in which the angular velocity sensor is arranged, and it is also possible to deal with high frequency vibration frequency components. The vibration that is greatly involved in the blur of the captured image in normal shooting is the blur in the direction of rotating the optical axis of the imaging means. For this reason, if the optical axis of the imaging means is the z axis, the rotation around the x axis and the y axis By providing two angular velocity sensors that detect the surrounding rotation, it is possible to efficiently detect the amount of camera shake with respect to camera shake in the direction of rotating the optical axis. Since it is a mechanical device that uses the inertia of a rotating object, it is difficult to achieve both miniaturization and high accuracy.

In addition, the two angular velocity sensors can only detect the rotation of the optical axis of the camera, and more advanced camera shake measurement, for example, camera shake that causes the camera to move in a direction perpendicular to the optical axis. In this case, the number of angular velocity sensors increases, or an acceleration sensor that detects linear acceleration is separately provided. Furthermore, camera shake that the camera itself moves and vibrates can be detected, but effective detection cannot be performed for image blur when the subject moves.

JP 2003-344890 A JP 2004-015515 A JP 2003-209735 A JP 2002-207232 A

  An imaging apparatus according to the present invention includes an optical system that receives light from a subject, a first imaging element that receives light from the subject through the optical system and outputs an image of the subject, and the optical system. A second image sensor that receives light from the subject and outputs a plurality of images while the first image sensor outputs one image, and a plurality of images captured by the second image sensor. A blur amount extracting unit that calculates a blur amount by taking a correlation between two consecutive images, and a blur component removing unit that removes a blur component from the image output from the first image sensor using the calculated blur amount; Is provided.

  The present invention has been made in order to solve the above-described problems, and it is possible to accurately detect the amount and direction of a blur component such as translation as well as the rotation of the optical axis, and it is easy to reduce the size. An object is to provide an imaging device and a portable device equipped with the imaging device.

  An imaging apparatus according to the present invention includes an optical system that receives light from a subject, a first imaging element that receives light from the subject through the optical system and outputs an image of the subject, and the optical system. A second image sensor that receives light from the subject and outputs a plurality of images while the first image sensor outputs one image, and a plurality of images captured by the second image sensor. A blur amount extracting unit that calculates a blur amount by taking a correlation between two consecutive images, and a blur component removing unit that removes a blur component from the image output from the first image sensor using the calculated blur amount; Is provided.

  An imaging apparatus according to the present invention includes an optical system that receives light from a subject, a first imaging element that receives light from the subject through the optical system and outputs an image of the subject, and the optical system. A second image sensor that receives light from the subject and outputs a plurality of images while the first image sensor outputs one image, and a plurality of images captured by the second image sensor. A blur amount extracting unit that calculates a blur amount by taking a correlation between two consecutive images, and a blur component removing unit that removes a blur component from the image output from the first image sensor using the calculated blur amount; And an image pickup apparatus having a camera shake correction function that can be easily miniaturized with a simple configuration.

It is a figure which shows the structure of the imaging device which concerns on Embodiment 1 of this invention. It is a figure explaining the method of camera-shake amount extraction of this invention. 1 is an external view of a mobile phone equipped with an imaging device of the present invention. It is a figure which shows the structure of the imaging device which concerns on Embodiment 2 of this invention. It is a figure which shows the structure of the imaging device which concerns on Embodiment 3 of this invention.

Explanation of symbols

1 incident light from a subject, 2a main optical system, 2b sub optical system,
3a main image sensor, 3b sub image sensor, 4 analog processing circuit,
5 A / D converter, 6 digital processing circuit, 7 blur component removing means,
8 blur amount extraction means, 9 optical system control drive means, 10 half mirror,
11 mirror, 12 LCD side case, 13 rear LCD screen,
14 Key side housing

  FIG. 1 is a configuration diagram of a camera system equipped with an image pickup device including a shake amount measuring unit according to Embodiment 1 of the present invention. As shown in the figure, the image pickup device according to the first embodiment includes a main image pickup device 3a and a sub image pickup device 3b, and accordingly, includes a main optical system 2a and a sub optical system 2b including lenses. The sub optical system 2b is mounted substantially parallel to the main optical system 2a, and forms incident light 1 from the same subject received by the main optical system 2a on the photoelectric conversion surface of the sub image pickup element 3b. As will be described later, the sub imaging device 3b is for measuring the amount of camera shake, and does not generate information for directly forming an output image, so the imaging range is the imaging range of the main optical system 2a. There is no problem using an optical system that is set narrower than the main optical system 2a and has optical performance such as transmitted light amount and resolution.

The main image pickup device 3a according to the present invention converts light from a subject incident through the main optical system into an electric signal and generates an image signal. However, the image pickup device used in a general digital still camera is used. The number of pixels is about 300,000 to 8 million pixels, and the exposure time for photographing one image is about 1/1000 second to several seconds. Among them, the effect of camera shake becomes significant when the exposure time is longer than 1/100 second. The main image sensor 3a uses the same image sensor as a general digital still camera.

On the other hand, since the sub image pickup device 3b only needs to detect the amount of camera shake, for example, a black and white image pickup device having about 10,000 pixels is used, and the light receiving area per pixel compared to the main image pickup device 3a. The exposure time can be shortened by increasing the sensitivity and increasing the sensitivity. Therefore, more than 2000 images can be captured and output per second, and the sub image sensor 3b can generate a plurality of image signals while the main image sensor 3a exposes and images one image. .

When the light 1 from the subject is input to the main and sub optical systems 2a and 2b, the light 1 from the subject is imaged on the photoelectric conversion surfaces of the main image sensor 3a and the sub image sensor 3b, respectively. The main image sensor 3a converts the light 1 from the imaged subject into an electrical signal and generates a two-dimensional image signal. At this time, if there are controllable elements such as the exposure time, aperture value, and focal position in the main optical system 2a and the main image sensor 3a, the imaging condition control means 9 is in a state where the output image signal of the main image sensor 3a is optimal. They are automatically controlled so that

The image signal thus obtained is input to the analog signal processing means 4 and subjected to analog signal processing such as gamma processing. Further, the analog image signal is A / D converted by the A / D conversion means 5 to be converted into a digital signal. Converted. This digital image signal is input to the digital image processing means 6 and light amount adjustment and color adjustment are performed so that an appropriate image signal can be generated. For example, the YCbCr format suitable for generating a JPEG compressed image is used for the purpose. It is converted into the corresponding image output.

This image processed image signal is input to the blur component removing means 7. The blur component removing unit 7 receives a digital image signal subjected to image processing from the main image sensor 3a, and also inputs a blur amount vector extracted from an image captured by the sub image sensor 3b from the blur amount extracting unit 8. Is done. Based on the magnitude and direction of the input blur amount vector, the blur component removing unit 7 performs image restoration using, for example, a Wiener filter. The Wiener filter is a restoration operator that gives an image that minimizes the mean square error from the original image. Specifically, it is expressed by the following equation and is processing in the frequency space.

Here, F (u, v): corrected image, G (u, v): degraded image, H (u, v): spread function, α: constant. A spread function H (u, v) is defined based on the blur amount vector, and a blur-free image is generated and output from the above equation. For example, in the case of a digital still camera, the output image is converted into a standard image format such as JPEG and then stored in a storage medium such as a memory card.

  In addition to the Wiener filter, many algorithms have been proposed for recent image processing techniques in addition to the Wiener filter. However, it goes without saying that image restoration may be performed using them.

  FIG. 2 is a diagram for explaining the operation of the blur amount extracting means 8 and shows the output of the sub-image sensor and the blur amount output calculated by taking the correlation from the output. Since the main image sensor 3a has a large number of pixels, the sensitivity is lower than that of the sub image sensor, and the exposure time becomes longer because high-quality imaging is performed. For this reason, there is a high possibility that an image including blurring is output due to the camera or the subject moving during the exposure time. Therefore, as shown in FIG. 2, while the exposure to the main image sensor 3a is completed from the start of photographing, the sub image sensor 3b outputs an image a plurality of times, and the main image sensor is obtained by taking a correlation between the plurality of images. The amount of blur within the exposure time 3a is calculated. A plurality of pieces of image data output from the sub image sensor 3b are respectively input to the shake amount extraction means 8.

The blur amount extraction means 8 compares and correlates temporally continuous images, extracts how the subject image shown on the sub image sensor 3b has moved, and outputs it as a blur amount vector. Specifically, for each pixel of the output image A of the sub image sensor, an absolute value or a square value of a difference from all the pixels of the output image B that is the next image of the output image A is calculated. If the image is not moving due to movement of the subject, the above differences should all be zero. On the other hand, if the image of the subject is moving due to camera shake, etc., the difference calculation result of a certain pixel and the difference calculation for the surrounding pixels based on the absolute value of the difference calculated for each pixel as described above Combined with the result, how the image is shaken is calculated, and a shake amount vector is calculated like a shake amount output A in FIG.
Note that, for each pixel of the output image, instead of taking a difference from all the pixels of the next image, the blur amount may be calculated by taking the difference for each pixel column and each row. This reduces accuracy but requires less computation.

Note that, with regard to the sub-imaging device and the camera shake extraction unit according to the present invention, a semiconductor device having a similar configuration is mounted on an optical mouse that is widely used for operating personal computers. The image sensor in the optical mouse is a small-scale solid-state image sensor having about 10,000 pixels, and images and outputs 2000 frames or more per second. A very simple plastic lens with a focal length of several millimeters is attached to the sensor, and when the optical mouse is placed on a plane, an image of the plane is formed on the imaging surface of the sensor. In addition, an LED is provided for irradiating a plane as a subject to ensure illuminance for imaging.

The output image signal of the sensor is transmitted to an arithmetic circuit formed on the same semiconductor chip as the sensor itself, and fine irregularities and shadows on the plane imaged by the sensor are extracted, and the positions of these images between frames By calculating the relationship, the relative movement direction and movement amount of the mouse and the plane are calculated. The blur detection algorithm is described in Patent Documents 2 and 3 described above, for example. If such an optical mouse configuration is used, the sub-imaging device 3b and the camera shake amount extraction means 8 of the present invention can be formed on the same chip. Therefore, it operates stably without adjustment and is manufactured at low cost. can do.

  FIG. 3 shows an example of an external appearance when the imaging apparatus according to Embodiment 1 of the present invention is applied to a mobile phone. It is a figure seen from the back in the state where the half-fold type mobile phone is opened, and only the surfaces of the optical system for the main image sensor and the optical system for the sub image sensor can be seen on the back of the liquid crystal side housing. Reference numeral 12 denotes an LCD side housing, 13 denotes a rear liquid crystal screen, and 14 denotes a key side housing. While the main image sensor captures a single image, the sub-image sensor placed beside it captures a plurality of images, measures the blur amount from the plurality of images, and the image captured by the main image sensor The blur component contained in is estimated.

  In the above example, a separate sub optical system independent from the main optical system is provided. However, in recent mobile phones equipped with a video phone function, a main camera for still image shooting and a video phone are used. Since the two image sensors of the sub camera for photographing the user himself are sometimes mounted, the camera shake amount extraction may be performed using the sub camera when taking a still image with the main camera. Such a sub-camera may not be accurate when measuring the amount of camera shake because its optical axis is not necessarily in a substantially parallel relationship with the main camera. However, since the camera shake amount detecting means and the sub camera can be integrated into one image pickup means, it is possible to obtain a mobile phone that corrects blurring without increasing the size of the apparatus.

Embodiment 2. FIG.
In the first embodiment, the image pickup apparatus removes the blur component by signal processing based on the measured blur amount vector. However, in the second embodiment, the lens is based on the measured blur amount vector. And mechanically driving the sensor to correct the blur component. FIG. 4 is a diagram showing the configuration of the camera system according to the second embodiment. As shown in the figure, the amount of blur is measured by a plurality of output images from the sub-imaging element 3b as in the first embodiment. The extracted blur amount is input to the optical system driving means 9. In the optical system driving means 9, the lens and the main image pickup device 3a included in the main optical system 2a are moved in two directions X and Y perpendicular to each other in a plane perpendicular to the optical axis Z in accordance with the shake amount vector. The image blur is corrected by decentering the optical axis Z.

Embodiment 3 FIG.
In the first and second embodiments, the optical system dedicated to the sub-imaging device is provided. However, as shown in FIG. 5, the light passing through the main optical system 2a is divided by the half mirror 10, the mirror 11, and the like. It is also possible to share the main optical system and the sub optical system by dividing by means and supplying the sub image pickup element 3b. During optical zoom, the field of view changes according to the main optical system 2a.

Embodiment 4 FIG.
In the first to third embodiments, the sub-imaging unit is used only for detecting the amount of camera shake. However, the sub-imaging element may be used for obtaining the optical condition of the subject that is photographed by the main imaging unit. The optical conditions include, for example, the brightness or light quantity of the subject, the color temperature of the illumination that illuminates the subject, and the TTL distance measurement that measures the distance from the image output by the image sensor to the subject.

When used for TTL distance measurement, the focal position of the secondary optical system can be moved in the same manner as the focal position of the primary optical system, or the primary optical system can be supplied with light passing through the primary optical system, etc. The imaging state on the sub-imaging device needs to change due to the change in the focal position of the system.
When performing TTL distance measurement using the sub-image sensor, for example, after the user presses the shutter, the lens of the sub-optical system is moved so that the focus position is 20 cm, 30 cm, 50 cm, 1 m, or 3 m, and shooting is performed at each focus position. The lens of the main image sensor is moved to the focus position where an image with the highest contrast was obtained. Since the sub-image sensor has fewer pixels than the main image sensor, shooting at each focus position can be performed faster than TTL distance measurement with the main image sensor. Can be shortened.

Regarding the brightness of the subject, a black and white distribution is obtained from one image captured by the sub image sensor before the output image is captured by the main image sensor, and the distribution is not biased to either white or black. The pixel gain of the main image sensor is selected. Also in this case, the sub-imaging device has fewer pixels than the main imaging device, and photographing can be performed at high speed, so that it is possible to shorten the time until the exposure for photographing the output image is started.

Regarding the measurement of the color temperature, for example, a part with a color filter is provided in a part of the sub-image sensor, and the part is used as a sensor for measuring the color temperature. It is desirable to introduce more light into the color temperature measuring sensor. When performing color temperature measurement, for example, when a user shoots a yellow human figure over the entire shooting range, the camera may not be able to determine whether the light is actually yellow or the subject is yellow, and may perform incorrect color temperature correction . In high-performance digital still cameras, etc., it is programmed to distinguish whether the light is yellow or the subject is yellow as much as possible using the internal CPU, but it is installed in mobile phones that have limited CPU power available for the camera In a camera that has been used, erroneous determination is more likely to occur. Therefore, if the color temperature measurement is performed with the main image sensor and the color temperature measurement is performed with the sub image sensor, more information can be collected, so that erroneous determination can be reduced. Note that the sensor for measuring the color temperature may be provided independently without being provided in a part of the sub-imaging device as described above.

  The above-mentioned optical conditions can be used in combination with the detection of camera shake, or only by obtaining the optical conditions.

The imaging apparatus according to the present invention can be used by being mounted on a digital still camera or a mobile phone with a camera function.

Claims (9)

An optical system that receives light from the subject;
A first image sensor that receives light from the subject via the optical system and outputs an image of the subject;
A second image sensor that receives light from the subject via the optical system and outputs a plurality of images while the first image sensor outputs one image;
A blur amount extracting means for calculating a blur amount by correlating two consecutive images from a plurality of images captured by the second image sensor;
An imaging apparatus comprising: a blur component removing unit that removes a blur component from an image output from the first imaging device using the calculated blur amount. The optical system includes: a first optical system that makes light incident on the first image sensor; and a second optical system that makes light incident on the second image sensor. The imaging device described in 1. The imaging apparatus according to claim 2, wherein the second optical system has an optical axis substantially parallel to the optical axis of the first optical system. The optical system includes only the first optical system, has a splitting unit that splits the light from the first optical system, and causes the split light to enter the first and second imaging elements, respectively. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized. An optical system that receives light from the subject;
A first image sensor that receives light from the subject via the optical system and outputs an image of the subject;
A second image sensor that receives light from the subject via the optical system and outputs a plurality of images while the first image sensor outputs one image;
A blur amount extracting means for calculating a blur amount by correlating two consecutive images from a plurality of images captured by the second image sensor;
An imaging apparatus comprising: an optical system control driving unit that drives the optical system so as to remove blur based on the calculated blur amount. A sensor unit for measuring a color temperature is provided in a part of the second image sensor, and after the color temperature is measured by the sensor unit for color temperature measurement, the first image sensor starts to receive light. The imaging apparatus according to 1. The first and second optical systems have focus changing means for changing the focal position, and the second imaging element performs imaging at a plurality of focal positions, and the highest contrast among the plurality of captured images. The imaging apparatus according to claim 2, wherein light reception of the first imaging element is started by focusing the first optical system on a focal position where an image is obtained. The second image pickup device picks up an image, sets the gain of the first image pickup device based on the brightness of each pixel of the obtained image, and starts light reception of the first image pickup device. The imaging apparatus according to claim 1. The portable apparatus provided with the imaging device as described in any one of Claims 1-8.

Images